dsh-vision-router 3.0.4 → 3.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -9,142 +9,142 @@
9
9
  <p align="center">DeepSeek keeps thinking; the built-in free vision chain and fourteen deep tools do the seeing. When an image matters, enable the composer’s “👁 Vision” control and use image turns like ordinary tool-calling turns — grounded, measurable, repeatable.</p>
10
10
 
11
11
  <p align="center">
12
- <a href="https://awesome-dsh-plugin.com"><img src="https://awesome-dsh-plugin.com/badge.svg" alt="awesome · DSH plugin" /></a>
13
12
  <a href="https://github.com/zp-home/dsh-recommend"><img src="https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fzp-home%2Fdsh-recommend%2Fmain%2Fdata%2Fbadges%2Fysr666__dsh-vision-router.certified.json&amp;style=flat-square" alt="dsh-recommend 🏅 certified" /></a>
14
13
  <a href="https://www.dshbase.com/plugins/dsh-vision-router/"><img src="https://img.shields.io/badge/dshbase-install--tested-2EA44F?style=flat-square" alt="dshbase install-tested" /></a>
15
14
  <a href="https://github.com/SoberReport-AI/DeepGuard/blob/main/reports/dsh-vision-router/2.0.1/39c8f2b2d69aa398418fd6c8ab40b691a92a1a3d.json"><img src="https://img.shields.io/badge/DeepGuard-audit%20clean-2EA44F?style=flat-square" alt="DeepGuard audit clean" /></a>
16
- <a href="https://whyihaveyou.github.io/dsh-suite/"><img src="https://img.shields.io/badge/featured%20on-dsh--suite-4d6bfe?style=flat-square" alt="featured on dsh-suite" /></a>
17
15
  </p>
18
16
 
19
- <p align="center">
20
- <a href="https://github.com/ysr666/dsh-vision-router/releases/latest"><img src="https://img.shields.io/github/v/release/ysr666/dsh-vision-router?label=release&color=5B4CF0&style=flat-square" alt="Latest stable release" /></a>
21
- <a href="tests"><img src="https://img.shields.io/badge/verified-Node%2022%20%2B%2024-2EA44F?style=flat-square" alt="Verified: Node 22 + 24" /></a>
22
- <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-2EA44F?style=flat-square" alt="License: MIT" /></a>
23
- <a href="package.json"><img src="https://img.shields.io/badge/Node.js-%3E%3D22-339933?style=flat-square&amp;logo=nodedotjs&amp;logoColor=white" alt="Node.js >=22" /></a>
24
- <img src="https://img.shields.io/badge/runtime-no%20Python-8A2BE2?style=flat-square" alt="No Python" />
25
- </p>
26
-
27
- <p align="center">
28
- <sub>Ecosystem:</sub>
29
- <a href="https://dshplugin.app/plugins/dsh-vision-router">DSHPlugin.app</a> ·
30
- <a href="https://github.com/diegosouzapw/awesome-omni-dsh-plugins">Awesome Omni DSH Plugins</a> ·
31
- <a href="https://dshpluginhub.ai/plugins/dsh-vision-router">dshpluginhub.ai</a> ·
32
- <a href="https://www.dsh.plus/en/plugins/dsh-vision-router/">dsh.plus</a> ·
33
- <a href="https://dshplugins.ai/">dshplugins.ai</a> ·
34
- <a href="https://dshmarket.com/p/ysr666/dsh-vision-router/">dsh-market</a>
35
- </p>
36
-
37
- <p align="center">Repository: <a href="https://github.com/ysr666/dsh-vision-router">GitHub (primary · issues, PRs and releases)</a> · <a href="https://atomgit.com/ysr666/dsh-vision-router">AtomGit (China mirror)</a></p>
17
+ <p align="center"><sub>DeepGuard badge links to a historical v2.0.1 audit; it does not certify current releases.</sub></p>
38
18
 
39
19
  <p align="center">English · <a href="README.zh.md">中文</a></p>
40
20
 
41
- <p align="center">💬 <strong>QQ community group: 1105463028</strong></p>
42
-
43
21
  > [!CAUTION]
44
22
  > **Data leaves your machine by default.** When a cloud vision model is used, Vision Router sends the image (or a derived crop), the vision prompt and related request metadata to that provider. A fresh install has an anonymous OVHcloud vision fallback enabled, so "free" and "no key" do **not** mean offline. Do not use the cloud chain for confidential, regulated or classified material. See [Data flow and strict local-only use](#data-flow-and-strict-local-only-use).
45
23
 
46
24
  > [!WARNING]
47
- > 📌 **Announcement (v3.0.4)**
48
- >
49
- > **v3.0.4: DSH alpha.5 verification and release hardening.** Exact Host/Chromium
50
- > gates passed for DSH `0.2.1-alpha.5`; CI supply-chain actions were refreshed.
51
- > The public DSH support range stays unchanged. Future `0.2.1-rc.1` is not
52
- > yet independently verified. [What’s new →](docs/releases/v3.0.4.md)
25
+ > **v3.1.0 release notes:** simpler image-input setup and clear cloud-upload privacy controls, plus DSH compatibility, HTTP and dependency fixes. See [v3.1.0 release notes](docs/releases/v3.1.0.md). For the installable version, check [GitHub Releases](https://github.com/ysr666/dsh-vision-router/releases/latest) or npm for the installable version.
26
+
27
+ ## Start here
28
+
29
+ **1. Install in an existing DSH Web profile:**
30
+
31
+ ```sh
32
+ npx @deepseek-ai/dsh plugin --profile web add dsh-vision-router
33
+ ```
34
+
35
+ **2. Keep your usual chat model** and click **👁 Vision** beside the composer; check for **👁 Vision ✓**.
36
+
37
+ **3. Paste or upload an image** and ask the agent what to inspect. The built-in keyless cloud fallback is available out of the box.
38
+
39
+ Using Desktop? Change `--profile web` to `--profile desktop` and reload the app. For Oh-DSH and manual setups, follow the [complete installation guide](#quick-start).
40
+
41
+ [Full quick start](#quick-start) · [What it can do](#highlights) · [Data flow and local-only use](#data-flow-and-strict-local-only-use)
53
42
 
54
43
  <p align="center">
55
44
  <img src="assets/vision-demo.gif" width="640" alt="Demo: paste an image, the agent locates the send button with vision_ground / vision_crop / vision_pixel_diff and answers with coordinates" />
56
45
  </p>
57
46
 
47
+ ## Try it on real images
48
+
49
+ - **Understand a screenshot:** “What does this error dialog mean? Read the exact message.”
50
+ - **Find a UI element:** “Where is the Save button in this screenshot? Give me its position.”
51
+ - **Compare two designs:** “Compare my page to this reference. What should I fix next?” Use `vision_pixel_diff` to measure changes after each iteration.
52
+ - **Read and inspect a long image:** “Extract the relevant text, then crop the unclear part to double-check it.”
53
+
54
+ These are ordinary chat requests: your normal model reasons through the task and calls vision tools as needed. [See the tool list](#tools) or [configure your own/local vision backend](#configuration).
55
+
58
56
  ## Contents
59
57
 
60
- - [Data flow and strict local-only use](#data-flow-and-strict-local-only-use)
61
- - [Why this exists](#why-this-exists)
62
- - [How it compares](#how-it-compares)
63
- - [Design lineage](#design-lineage)
64
- - [Acknowledgements](#acknowledgements)
65
- - [Quick start](#quick-start)
66
- - [Free vision key channels](#free-vision-key-channels)
58
+ - [Try it on real images](#try-it-on-real-images)
67
59
  - [Highlights](#highlights)
68
60
  - [How it works](#how-it-works)
69
61
  - [Tools](#tools)
62
+ - [Quick start](#quick-start)
63
+ - [Data flow and strict local-only use](#data-flow-and-strict-local-only-use)
64
+ - [Configure your own vision models](#configure-your-own-vision-models)
65
+ - [Why this exists](#why-this-exists)
66
+ - [How it compares](#how-it-compares)
70
67
  - [Configuration](#configuration)
71
68
  - [Install and lifecycle](#install-and-lifecycle)
72
69
  - [Troubleshooting](#troubleshooting)
70
+ - [Design lineage](#design-lineage)
71
+ - [Acknowledgements](#acknowledgements)
72
+ - [Release notes and community](#release-notes-and-community)
73
73
 
74
- ## Data flow and strict local-only use
75
-
76
- Vision Router does not operate fully offline in its default configuration. The exact boundary depends on the operation and the selected models:
77
-
78
- | Operation | Where data goes |
79
- |---|---|
80
- | Local pixel tools such as crop, pixel diff, palette, SVG trace, cutout, materialize and HTML screenshot | Processed on the DSH machine; these operations do not call a vision model. |
81
- | `vision_ocr` with Tesseract | Processed on the DSH machine. With the default `auto` engine, an unavailable or empty local OCR result may fall back to a vision model. |
82
- | Image-turn routing and vision-model tools such as describe, detect, ground or vision OCR | The image or derived crop, prompt and relevant context are sent to the selected vision provider. |
83
- | Built-in free fallback | Sent anonymously (no API key) to OVHcloud AI Endpoints at `oai.endpoints.kepler.ai.cloud.ovh.net`; the service still receives the request payload and network metadata such as the source IP. |
84
- | User-configured cloud model / HTTP provider | Sent to that provider's configured endpoint under its own retention and privacy terms. |
85
- | Local Ollama / LM Studio | Image pixels are sent to the configured local endpoint. The resulting text still returns to the current chat model, which may itself be remote. |
74
+ ## Highlights
86
75
 
87
- To stop **Vision Router itself** from sending image data to remote vision endpoints, enable **Settings → Vision Router → General → Local-only vision**. This is a runtime policy, not a destructive rewrite: saved cloud rows and `freeFallback` stay configured but cannot execute while the switch is on. Only loopback visual endpoints (`localhost`, `127.0.0.0/8`, `::1`) are eligible, including Ollama, LM Studio, and custom local HTTP backends. For a **strict local-only workflow**, also use a local chat model because vision result text still returns to the current chat model. Test the final network boundary in your own environment; Vision Router cannot make a remote chat model, proxy or Host integration local.
76
+ - **Capability-aware Auto routing.** Keep configured order for deterministic control, or explicitly enable Auto to prioritize already-configured models using measured capability evidence. Auto never infers capability from model names, and enabling Auto alone does not start benchmarks.
77
+ - **Verifiable model profiling.** Exact Test Vision sends one request to one exact model; Quick and Full benchmark OCR / general / structured / document / grounding capabilities. Background profiling is separately authorized and yields to real foreground vision work.
78
+ - **Original pixels, real answers.** The vision chain reads the image at original resolution (auto-downscaled only to protect latency/quota); the agent's question travels with the image, so answers are about *your* question, not a generic description.
79
+ - **Automatic failover with classified errors.** Region blocks, ToS filtering, 402 quota, 429 rate limits, context overflow, network failures — the chain walks providers one by one and only reports after all of them failed, with actionable advice. A 429 immediately advances to the next backend and opens a Retry-After-aware cooldown instead of sleeping inside the request.
80
+ - **Image memory.** Vision answers are cached by attachment content hash; later text turns substitute the recorded description (marked as untrusted evidence), so DeepSeek genuinely remembers earlier images without re-spending vision calls.
81
+ - **A verifiable pixel loop.** Reference → `vision_html_screenshot` → `vision_pixel_diff` (ratio + worst 8×8-grid regions) → fix → repeat until the mismatch converges. UI restoration becomes measurable instead of eyeballed.
82
+ - **Stable tool schema.** All fourteen deep tools are registered from session start by default, avoiding a mid-conversation tool-list expansion that can invalidate long-context KV/prefix caches. `progressiveTools: true` remains an advanced boot-time opt-in; only then does `vision_activate` mount the tools on demand. See [`docs/progressive-tools-cache.md`](docs/progressive-tools-cache.md).
83
+ - **Selective proxy.** Only the configured vision provider hosts go through your local proxy; DeepSeek stays direct.
88
84
 
89
- ## Why this exists
85
+ ### Pixel loop in practice
90
86
 
91
- Most DSH vision plugins bridge images to DeepSeek as *text descriptions* — lossy, one-shot, and blind to pixels. This plugin keeps the **Host-canonical image pixels on the vision model's side** and DeepSeek on the reasoning side, and makes looking at an image an **ordinary tool call**:
87
+ [![Reference design and final agent rebuild, verified with vision_pixel_diff at 2.54% final difference.](https://raw.githubusercontent.com/ysr666/dsh-vision-router/main/assets/pixel-loop.png)](https://raw.githubusercontent.com/ysr666/dsh-vision-router/main/assets/pixel-loop.png)
92
88
 
93
- - **One command install.** The package ships its own composition patch (`dsh.bundle.patch`): `dsh plugin add` wires the row, the admission wrapper and the attachment limits automatically — zero manual file edits. Taking over the official DeepSeek route is an optional setting (stealth mode, off by default).
94
- - **Free by default.** Vision tools end with a five-model OVHcloud anonymous fallback: no account, no key, 2 requests/minute per IP per model, roughly 10 RPM in theory across independent buckets. User-provided vision models run first.
95
- - **No Python.** The whole pipeline — downscale, grounding, crop, pixel diff, palette, OCR, SVG trace, cutout, HTML screenshot — runs on sharp / potrace / tesseract / system Chrome.
96
- - **Continuous multi-step image work.** An image turn is a text turn that calls tools: `vision_ground` → `vision_crop` → `vision_describe` → `vision_pixel_diff` → fix → screenshot again. The agent keeps iterating until the work is done.
97
- - **DeepSeek stays the brain.** Text turns are untouched in model, cost and context. The vision model is only the eyes, called on demand; answers are cached by image content.
98
- - **Transparent to the user.** Uploaded images keep rendering as images in the conversation UI; the rewrite that points the model at the vision tools happens only inside the model call, never in the session log.
89
+ <p align="center"><sub>Click the image to open the full-resolution original.</sub></p>
99
90
 
100
- ## How it compares
91
+ The agent rebuilt the UI from the reference image, then verified the final result with `vision_pixel_diff`: **2.54% final diff** (32,939 / 1,296,000 differing pixels, threshold 16/channel).
101
92
 
102
- **One-line take**: most dsh vision plugins turn images into *text descriptions* for DeepSeek
103
- (description bridge — lossy); this plugin hands the image turn *straight to a vision model*
104
- (routing bridge — pixel-level), with a built-in keyless free fallback.
93
+ ## How it works
105
94
 
106
- > [!NOTE]
107
- > On DSH 0.1.2-alpha.1+, attachments remain Host-owned. Vision Router consumes the Host-persisted canonical image: clean single-frame 8-bit sRGB/sRGBA images inside the configured normalization limits can pass through byte-identically, while images that need orientation, color-space, metadata, animation, or size normalization may be re-encoded. Pixel tools therefore promise the Host-canonical raster, not preservation of the uploader's original encoded bytes.
95
+ <p align="center">
96
+ <img src="assets/how-it-works.svg" width="100%" alt="How DSH Vision Router keeps DeepSeek as the brain and vision tools as the eyes." />
97
+ </p>
108
98
 
109
- | | Manual model switching | MCP vision bridge | dsh-vision-router |
110
- |---|---|---|---|
111
- | Image pixels | ✅ available (when switched) | ❌ text description only | ✅ Host-canonical raster, on the image turn |
112
- | Automatic | ❌ | ✅ | ✅ |
113
- | Daily model untouched | ❌ (whole session swapped) | ✅ | ✅ |
114
- | Provider failure recovery | ❌ | ❌ | ✅ fallback chains |
115
- | Reusable structured queries | — | partial | ✅ JSON mode + caching |
116
- | Free out-of-the-box | ❌ | ❌ | ✅ built-in keyless endpoint |
117
- | Fits dsh composition | — | external server | ✅ one plugin row |
99
+ The vision model is **only the eyes**; DeepSeek is **always the brain**. An image turn is never hijacked by a one-shot vision answer — the agent drives the tools itself and can keep operating on the image across as many steps as the task needs.
118
100
 
119
- **Difference from existing dsh community projects** (all excellent, each with its own focus; descriptions reflect their READMEs as of 2026-08):
101
+ ## Tools
120
102
 
121
- | Project | Approach | What this plugin adds |
122
- |---|---|---|
123
- | [dsh-vision-sidecar](https://github.com/121103qwq/dsh-vision-sidecar) | Pre-describes images with an external VLM; the description joins the session as a message to DeepSeek; LLM7.io anonymous endpoint by default (OVHcloud listed as a no-key alternative) | Description bridge; this plugin adds image routing, with `vision_describe` covering descriptions on demand |
124
- | [dsh-vision-proxy](https://github.com/Flyvhidbwo/dsh-vision-proxy) | Wraps a provider route and transcribes images into text in the request stream | Transcription bridge; this plugin wraps no provider — it rewrites routing through `agent/request` waterfalls |
125
- | [dsh-vision-provider](https://github.com/libinyam/dsh-vision-provider) | Registers `DeepSeek + Vision` combined routes: images are described by the chosen vision model before reaching DeepSeek | Two-model bridge idea; this plugin adds automatic routing, fallback chains and tools on top |
126
- | [modlens](https://github.com/liustack/modlens) | The first dsh vision plugin; reuses local Claude Code/Codex/OpenCode/Pi logins as vision engines | Engine-reuse idea; this plugin ships its own provider chain and depends on no other local CLI |
127
- | [dsh-vision-toolkit](https://github.com/Anionex/dsh-vision-toolkit) | Ten intent-aware visual tools (Q&A/OCR/pixel verification/UI restoration), called explicitly on demand | Broader tool set; this plugin adds whole-turn auto-routing and a keyless free fallback |
128
- | [dsh-tool-vision](https://github.com/Scorp1o117/dsh-tool-vision) | An `inspect_image` tool plus an `agent/pre-step` waterfall bridge (pasted images become tool hints before entering the log) | Similar waterfall bridge; this plugin adds turn routing, fallback chains, caching and the free endpoint |
103
+ Default `progressiveTools: false`: all fourteen deep tools stay registered from plugin startup, so text and image turns can call them immediately. If you explicitly set `progressiveTools: true` in the profile/composition `cordis.patch.yml`, progressive mode is restored: only `vision_activate` is exposed initially, the full tool set mounts on first use, and the `vision-tools` skill is registered. This is a boot-time switch; restart DSH after changing it. Built on sharp / potrace / tesseract / system Chrome — no Python:
129
104
 
130
- ## Design lineage
105
+ <p align="center">
106
+ <img src="assets/vision-tools.svg" width="100%" alt="Eleven image-processing tools available in DSH Vision Router." />
107
+ </p>
131
108
 
132
- The deep-vision tool layer and UI-restoration workflow in this project were informed by [Anionex/agent-vision-toolkit](https://github.com/Anionex/agent-vision-toolkit) and its native DSH implementation [Anionex/dsh-vision-toolkit](https://github.com/Anionex/dsh-vision-toolkit). In particular, this project drew on their intent-driven tool selection, progressive tool exposure, pixel-diff verification loop, and parts of the visual-tool decomposition and naming, including long-screenshot OCR, foreground extraction, and HTML screenshot tooling.
109
+ The diagram covers the eleven image-processing tools. `vision_present` (durable image delivery) and `vision_bootstrap` (the optional 1+x structured first pass) bring the default deep-tool set to fourteen. Enabling the privacy-gated `vision_screenshot` at boot adds an optional fifteenth tool.
133
110
 
134
- All code in `dsh-vision-router` is independently implemented. On top of those referenced design ideas, this project independently developed its turn-level/tools-first vision routing, DSH admission/wrapper integration, multi-backend provider and failure fallback chains, built-in free vision chain, attachment/image-memory handling, caching, and related runtime resilience mechanisms.
111
+ | Tool | What it does | Artifact |
112
+ |---|---|---|
113
+ | `vision_bootstrap` | Optional 1+x structured first visual pass; establishes task-independent evidence before at least one follow-up vision call | — |
114
+ | `vision_describe` | Image Q&A / multi-image compare / structured-evidence JSON mode (summary + layout regions + entity inventory + verbatim transcription) | — |
115
+ | `vision_materialize` | Copy an authorized attachment into a stable content-addressed workspace handoff file and return absolute + short workspace-relative paths for local OCR/parser fallbacks; no vision/network call | image copy |
116
+ | `vision_ground` | Locate a target → **original-pixel box x1/y1/x2/y2** | annotated PNG (optional) |
117
+ | `vision_detect` | Numbered inventory of every element of a kind (buttons/inputs/links…) with original-pixel boxes | annotated PNG with numbered boxes |
118
+ | `vision_crop` | Crop and zoom into a pixel box | PNG |
119
+ | `vision_present` | Publish a generated or edited local image as a durable chat attachment so the user can see it | image attachment |
120
+ | `vision_pixel_diff` | Per-pixel comparison: diff ratio + worst 8×8-grid regions | red heatmap PNG + JSON report |
121
+ | `vision_colors` | Dominant colors (hex + share) | — |
122
+ | `vision_ocr` | Text transcription: configurable default engine (`auto` / local Tesseract / vision model); an explicit per-call engine still wins | — |
123
+ | `vision_trace` | SVG vectorization (potrace posterization; icons/logos) | SVG |
124
+ | `vision_extract_foreground` | Cutout via border flood fill (uniform backgrounds) | transparent PNG |
125
+ | `vision_html_screenshot` | Screenshot a local HTML file (headless system Chrome); `fullPage: true` captures the whole page and reports `pageHeight` | PNG |
126
+ | `vision_screenshot` | **Disabled by default; explicit opt-in required.** Capture the Windows virtual screen, macOS main display, or Linux root display. Windows uses PowerShell CopyFromScreen, macOS uses `screencapture`, and Linux requires ImageMagick `import` or `scrot`; optional `identify=true` tries enabled local recognition backends in order and returns the description with the path | PNG / + description |
127
+ | `vision_long_screenshot_ocr` | Long-screenshot transcription: overlapping chunks, tesseract first / vision model fallback, stitched Markdown | chunk PNGs + Markdown + manifest |
135
128
 
136
- We appreciate Anionex's prior work and the broader DSH community. Clear attribution and independent iteration can coexist; both help keep the DSH ecosystem open, collaborative, and healthy.
129
+ Formats are sniffed from magic bytes, so extensionless content-addressed attachment files work everywhere (no `.png` renaming needed).
137
130
 
138
- ## Acknowledgements
131
+ **Common workflows**
139
132
 
140
- This project borrows ideas from all of the above — especially the keyless free-endpoint
141
- exploration (LLM7.io and OVHcloud anonymous tiers) by
142
- [dsh-vision-sidecar](https://github.com/121103qwq/dsh-vision-sidecar). Thanks to the authors of
143
- [dsh-vision-proxy](https://github.com/Flyvhidbwo/dsh-vision-proxy),
144
- [dsh-vision-provider](https://github.com/libinyam/dsh-vision-provider),
145
- [modlens](https://github.com/liustack/modlens),
146
- [dsh-vision-toolkit](https://github.com/Anionex/dsh-vision-toolkit), and
147
- [dsh-tool-vision](https://github.com/Scorp1o117/dsh-tool-vision).
133
+ ```text
134
+ vision_ground image="ref.png" target="the send button"
135
+ vision_detect image="page.png" target="input fields"
136
+ vision_crop image="ref.png" region="1067,841,1108,881"
137
+ vision_present path="rebuilt.png"
138
+ vision_describe paths=["ref.png","impl.png"] question="list the differences" json=true
139
+ vision_pixel_diff original="ref.png" rebuilt="screenshot.png"
140
+ vision_ocr image="screenshot.png"
141
+ vision_colors image="ref.png" top=8
142
+ vision_trace image="icon.png" steps=4
143
+ vision_extract_foreground image="logo.png"
144
+ vision_html_screenshot source="page.html" width=1200 height=720
145
+ vision_html_screenshot source="page.html" width=1200 height=720 fullPage=true
146
+ vision_long_screenshot_ocr image="chat-log.png" chunkHeight=1200 overlap=120
147
+ ```
148
148
 
149
149
  ## Quick start
150
150
 
@@ -217,101 +217,69 @@ The built-in anonymous OVH vision fallback is already configured, so normal imag
217
217
  <img src="assets/dsh-conversation-image-qa-result.png" width="49%" alt="The agent's structured answer describing the image content." />
218
218
  </p>
219
219
 
220
- ## Free vision key channels
221
-
222
- The built-in OVH fallback is anonymous by design, and OVH caps anonymous use at **2 requests/minute per IP per model**. If that feels tight, every channel below offers **free vision models with much higher quotas** — all of them are free to register, and none charges for the free tier. Free policies rotate often; treat this table as an August 2026 snapshot and double-check each provider's console before relying on it.
223
-
224
- | Channel | Free vision model(s) | Free quota | CN direct? | Where to get the key |
225
- |---|---|---|---|---|
226
- | OVHcloud AI Endpoints (access key) | `Qwen2.5-VL-72B-Instruct` — the same endpoint the built-in fallback uses | **400 req/min** per project per model (vs 2 anonymous) | ✅ | OVH account → Public Cloud project (attach a payment method; free models are not charged) → AI Endpoints access key |
227
- | Zhipu (bigmodel.cn) | `glm-4.6v-flash` · `glm-4.1v-thinking-flash` · `glm-4v-flash` — three permanently free models; chaining them triples capacity | uncapped tokens | ✅ | open.bigmodel.cn → API keys |
228
- | DashScope (Aliyun) | `qwen3-vl-flash` (limited-time free) and the Qwen-VL series | new users: 1M tokens per model series / 90 days | ✅ | bailian.console.aliyun.com |
229
- | Intern AI (Shanghai AI Lab) | `internvl-latest` · `internvl3.5-latest` | 30 RPM, **90M tokens/month** | ✅ | chat.intern-ai.org.cn |
230
- | Groq | `meta-llama/llama-4-scout-17b-16e-instruct` (native multimodal, up to 5 images) | 30 RPM / 14,400 req/day, no card | ❌ proxy | console.groq.com |
231
- | Google AI Studio | `gemini-2.5-flash` · `gemini-2.5-flash-lite` | 10–30 RPM / 500–1,500 req/day | ❌ proxy | aistudio.google.com |
232
- | NVIDIA NIM | `meta/llama-3.2-11b-vision-instruct` · `nvidia/nemotron-nano-12b-v2-vl` | 40 RPM, no card | ⚠️ | build.nvidia.com |
233
- | OpenCode Zen | `mimo-v2.5-free` (vision + code) | 30 RPM / 500 req/day | ⚠️ | opencode.ai/zen |
234
- | OpenRouter | `google/gemma-4-26b-a4b-it:free` · `google/gemma-4-31b-it:free` | 50 req/day on unpaid accounts | ❌ proxy | openrouter.ai |
220
+ ## Data flow and strict local-only use
235
221
 
236
- Any of these channels can join the vision chain as an `httpProviders` entry (key in the matching environment variable or `~/.dsh/.credentials.yaml`), and the chain tries your entries before the anonymous fallback.
222
+ Vision Router does not operate fully offline in its default configuration. The exact boundary depends on the operation and the selected models:
237
223
 
238
- > [!NOTE]
239
- > Free-tier policies change without notice — Cerebras retired its free tier in July 2026 (now a one-time $5 credit), SambaNova's free tier is down to 20 requests/day, and Hugging Face's is $0.10/month. Third-party “`:free` relay” aggregators are deliberately not listed: they rotate quickly, lack SLAs, and some resell quota in ways that violate upstream terms.
224
+ | Operation | Where data goes |
225
+ |---|---|
226
+ | Local pixel tools such as crop, pixel diff, palette, SVG trace, cutout, materialize and HTML screenshot | Processed on the DSH machine; these operations do not call a vision model. |
227
+ | `vision_ocr` with Tesseract | Processed on the DSH machine. With the default `auto` engine, an unavailable or empty local OCR result may fall back to a vision model. |
228
+ | Image-turn routing and vision-model tools such as describe, detect, ground or vision OCR | The image or derived crop, prompt and relevant context are sent to the selected vision provider. |
229
+ | Built-in free fallback | Sent anonymously (no API key) to OVHcloud AI Endpoints at `oai.endpoints.kepler.ai.cloud.ovh.net`; the service still receives the request payload and network metadata such as the source IP. |
230
+ | User-configured cloud model / HTTP provider | Sent to that provider's configured endpoint under its own retention and privacy terms. |
231
+ | Local Ollama / LM Studio | Image pixels are sent to the configured local endpoint. The resulting text still returns to the current chat model, which may itself be remote. |
240
232
 
241
- ## Highlights
233
+ To stop **Vision Router itself** from sending image data to remote vision endpoints, enable **Settings → Vision Router → General → Local-only vision**. This is a runtime policy, not a destructive rewrite: saved cloud rows and `freeFallback` stay configured but cannot execute while the switch is on. Only loopback visual endpoints (`localhost`, `127.0.0.0/8`, `::1`) are eligible, including Ollama, LM Studio, and custom local HTTP backends. For a **strict local-only workflow**, also use a local chat model because vision result text still returns to the current chat model. Test the final network boundary in your own environment; Vision Router cannot make a remote chat model, proxy or Host integration local.
242
234
 
243
- - **Capability-aware Auto routing.** Keep configured order for deterministic control, or explicitly enable Auto to prioritize already-configured models using measured capability evidence. Auto never infers capability from model names, and enabling Auto alone does not start benchmarks.
244
- - **Verifiable model profiling.** Exact Test Vision sends one request to one exact model; Quick and Full benchmark OCR / general / structured / document / grounding capabilities. Background profiling is separately authorized and yields to real foreground vision work.
245
- - **Original pixels, real answers.** The vision chain reads the image at original resolution (auto-downscaled only to protect latency/quota); the agent's question travels with the image, so answers are about *your* question, not a generic description.
246
- - **Automatic failover with classified errors.** Region blocks, ToS filtering, 402 quota, 429 rate limits, context overflow, network failures — the chain walks providers one by one and only reports after all of them failed, with actionable advice. A 429 immediately advances to the next backend and opens a Retry-After-aware cooldown instead of sleeping inside the request.
247
- - **Image memory.** Vision answers are cached by attachment content hash; later text turns substitute the recorded description (marked as untrusted evidence), so DeepSeek genuinely remembers earlier images without re-spending vision calls.
248
- - **A verifiable pixel loop.** Reference → `vision_html_screenshot` → `vision_pixel_diff` (ratio + worst 8×8-grid regions) → fix → repeat until the mismatch converges. UI restoration becomes measurable instead of eyeballed.
249
- - **Stable tool schema.** All fourteen deep tools are registered from session start by default, avoiding a mid-conversation tool-list expansion that can invalidate long-context KV/prefix caches. `progressiveTools: true` remains an advanced boot-time opt-in; only then does `vision_activate` mount the tools on demand. See [`docs/progressive-tools-cache.md`](docs/progressive-tools-cache.md).
250
- - **Selective proxy.** Only the configured vision provider hosts go through your local proxy; DeepSeek stays direct.
235
+ ## Configure your own vision models
251
236
 
252
- ### Pixel loop in practice
237
+ **DSH owns providers, credentials and the global model catalog; Vision Router uses configured Host models for visual orchestration, execution, testing and diagnostics.** No plugin-specific commercial vendor YAML is required. Vision Router can privately supplement its own backend picker with live `/models` evidence for **already configured** Host providers and endpoint-scoped image-capability hints; these never register a new provider or mutate DSH's global model directory. Cached/saved-only IDs do not themselves authorize a direct model bridge. See [architecture provenance](docs/architecture/model-catalog-ownership.md).
253
238
 
254
- [![Reference design and final agent rebuild, verified with vision_pixel_diff at 2.54% final difference.](https://raw.githubusercontent.com/ysr666/dsh-vision-router/main/assets/pixel-loop.png)](https://raw.githubusercontent.com/ysr666/dsh-vision-router/main/assets/pixel-loop.png)
239
+ Manage provider credentials and models in **DSH Settings → Models**, then pick those models from the same Host catalog in **Settings → Vision Router**. Use **Test Vision** to verify that the exact model actually reads image content. If Host metadata incorrectly marks a real vision model text-only, prefer DSH **Model input types → Image**; on older Hosts, declare `input: [text, image]` on a **verified image-capable** `llm-pi-ai` model and reload configuration. Never fabricate image support for text-only models.
255
240
 
256
- <p align="center"><sub>Click the image to open the full-resolution original.</sub></p>
241
+ The built-in anonymous OVH fallback is a separate Vision Router `httpProviders` runtime and not a DSH provider preset. Strictly local workflows also require a local chat model and the Local-only vision setting. Vendor price, quota and model IDs change; this project does not keep an ongoing marketing or free-tier list. The [August 2026 survey](docs/free-models.zh-CN.md) remains available solely as historical research.
257
242
 
258
- The agent rebuilt the UI from the reference image, then verified the final result with `vision_pixel_diff`: **2.54% final diff** (32,939 / 1,296,000 differing pixels, threshold 16/channel).
243
+ ## Why this exists
259
244
 
260
- ## How it works
245
+ Most DSH vision plugins bridge images to DeepSeek as *text descriptions* — lossy, one-shot, and blind to pixels. This plugin keeps the **Host-canonical image pixels on the vision model's side** and DeepSeek on the reasoning side, and makes looking at an image an **ordinary tool call**:
261
246
 
262
- <p align="center">
263
- <img src="assets/how-it-works.svg" width="100%" alt="How DSH Vision Router keeps DeepSeek as the brain and vision tools as the eyes." />
264
- </p>
247
+ - **One command install.** The package ships its own composition patch (`dsh.bundle.patch`): `dsh plugin add` wires the row, the admission wrapper and the attachment limits automatically — zero manual file edits. Taking over the official DeepSeek route is an optional setting (stealth mode, off by default).
248
+ - **Free by default.** Vision tools end with a four-model OVHcloud anonymous fallback: no account, no key, 2 requests/minute per IP per model, roughly 8 RPM in theory across independent buckets. User-provided vision models run first.
249
+ - **No Python.** The whole pipeline — downscale, grounding, crop, pixel diff, palette, OCR, SVG trace, cutout, HTML screenshot — runs on sharp / potrace / tesseract / system Chrome.
250
+ - **Continuous multi-step image work.** An image turn is a text turn that calls tools: `vision_ground` → `vision_crop` → `vision_describe` → `vision_pixel_diff` → fix → screenshot again. The agent keeps iterating until the work is done.
251
+ - **DeepSeek stays the brain.** Text turns are untouched in model, cost and context. The vision model is only the eyes, called on demand; answers are cached by image content.
252
+ - **Transparent to the user.** Uploaded images keep rendering as images in the conversation UI; the rewrite that points the model at the vision tools happens only inside the model call, never in the session log.
265
253
 
266
- The vision model is **only the eyes**; DeepSeek is **always the brain**. An image turn is never hijacked by a one-shot vision answer — the agent drives the tools itself and can keep operating on the image across as many steps as the task needs.
254
+ ## How it compares
267
255
 
268
- ## Tools
256
+ **One-line take**: most dsh vision plugins turn images into *text descriptions* for DeepSeek
257
+ (description bridge — lossy); this plugin hands the image turn *straight to a vision model*
258
+ (routing bridge — pixel-level), with a built-in keyless free fallback.
269
259
 
270
- Default `progressiveTools: false`: all fourteen deep tools stay registered from plugin startup, so text and image turns can call them immediately. If you explicitly set `progressiveTools: true` in the profile/composition `cordis.patch.yml`, progressive mode is restored: only `vision_activate` is exposed initially, the full tool set mounts on first use, and the `vision-tools` skill is registered. This is a boot-time switch; restart DSH after changing it. Built on sharp / potrace / tesseract / system Chrome — no Python:
260
+ > [!NOTE]
261
+ > On DSH 0.1.2-alpha.1+, attachments remain Host-owned. Vision Router consumes the Host-persisted canonical image: clean single-frame 8-bit sRGB/sRGBA images inside the configured normalization limits can pass through byte-identically, while images that need orientation, color-space, metadata, animation, or size normalization may be re-encoded. Pixel tools therefore promise the Host-canonical raster, not preservation of the uploader's original encoded bytes.
271
262
 
272
- <p align="center">
273
- <img src="assets/vision-tools.svg" width="100%" alt="Eleven image-processing tools available in DSH Vision Router." />
274
- </p>
263
+ | | Manual model switching | MCP vision bridge | dsh-vision-router |
264
+ |---|---|---|---|
265
+ | Image pixels | ✅ available (when switched) | ❌ text description only | ✅ Host-canonical raster, on the image turn |
266
+ | Automatic | ❌ | ✅ | ✅ |
267
+ | Daily model untouched | ❌ (whole session swapped) | ✅ | ✅ |
268
+ | Provider failure recovery | ❌ | ❌ | ✅ fallback chains |
269
+ | Reusable structured queries | — | partial | ✅ JSON mode + caching |
270
+ | Free out-of-the-box | ❌ | ❌ | ✅ built-in keyless endpoint |
271
+ | Fits dsh composition | — | external server | ✅ one plugin row |
275
272
 
276
- The diagram covers the eleven image-processing tools. `vision_present` (durable image delivery) and `vision_bootstrap` (the optional 1+x structured first pass) bring the default deep-tool set to fourteen. Enabling the privacy-gated `vision_screenshot` at boot adds an optional fifteenth tool.
273
+ **Difference from existing dsh community projects** (all excellent, each with its own focus; descriptions reflect their READMEs as of 2026-08):
277
274
 
278
- | Tool | What it does | Artifact |
275
+ | Project | Approach | What this plugin adds |
279
276
  |---|---|---|
280
- | `vision_bootstrap` | Optional 1+x structured first visual pass; establishes task-independent evidence before at least one follow-up vision call | — |
281
- | `vision_describe` | Image Q&A / multi-image compare / structured-evidence JSON mode (summary + layout regions + entity inventory + verbatim transcription) | — |
282
- | `vision_materialize` | Copy an authorized attachment into a stable content-addressed workspace handoff file and return absolute + short workspace-relative paths for local OCR/parser fallbacks; no vision/network call | image copy |
283
- | `vision_ground` | Locate a target → **original-pixel box x1/y1/x2/y2** | annotated PNG (optional) |
284
- | `vision_detect` | Numbered inventory of every element of a kind (buttons/inputs/links…) with original-pixel boxes | annotated PNG with numbered boxes |
285
- | `vision_crop` | Crop and zoom into a pixel box | PNG |
286
- | `vision_present` | Publish a generated or edited local image as a durable chat attachment so the user can see it | image attachment |
287
- | `vision_pixel_diff` | Per-pixel comparison: diff ratio + worst 8×8-grid regions | red heatmap PNG + JSON report |
288
- | `vision_colors` | Dominant colors (hex + share) | — |
289
- | `vision_ocr` | Text transcription: configurable default engine (`auto` / local Tesseract / vision model); an explicit per-call engine still wins | — |
290
- | `vision_trace` | SVG vectorization (potrace posterization; icons/logos) | SVG |
291
- | `vision_extract_foreground` | Cutout via border flood fill (uniform backgrounds) | transparent PNG |
292
- | `vision_html_screenshot` | Screenshot a local HTML file (headless system Chrome); `fullPage: true` captures the whole page and reports `pageHeight` | PNG |
293
- | `vision_screenshot` | **Disabled by default; explicit opt-in required.** Capture the Windows virtual screen, macOS main display, or Linux root display. Windows uses PowerShell CopyFromScreen, macOS uses `screencapture`, and Linux requires ImageMagick `import` or `scrot`; optional `identify=true` tries enabled local recognition backends in order and returns the description with the path | PNG / + description |
294
- | `vision_long_screenshot_ocr` | Long-screenshot transcription: overlapping chunks, tesseract first / vision model fallback, stitched Markdown | chunk PNGs + Markdown + manifest |
295
-
296
- Formats are sniffed from magic bytes, so extensionless content-addressed attachment files work everywhere (no `.png` renaming needed).
297
-
298
- **Common workflows**
299
-
300
- ```text
301
- vision_ground image="ref.png" target="the send button"
302
- vision_detect image="page.png" target="input fields"
303
- vision_crop image="ref.png" region="1067,841,1108,881"
304
- vision_present path="rebuilt.png"
305
- vision_describe paths=["ref.png","impl.png"] question="list the differences" json=true
306
- vision_pixel_diff original="ref.png" rebuilt="screenshot.png"
307
- vision_ocr image="screenshot.png"
308
- vision_colors image="ref.png" top=8
309
- vision_trace image="icon.png" steps=4
310
- vision_extract_foreground image="logo.png"
311
- vision_html_screenshot source="page.html" width=1200 height=720
312
- vision_html_screenshot source="page.html" width=1200 height=720 fullPage=true
313
- vision_long_screenshot_ocr image="chat-log.png" chunkHeight=1200 overlap=120
314
- ```
277
+ | [dsh-vision-sidecar](https://github.com/121103qwq/dsh-vision-sidecar) | Pre-describes images with an external VLM; the description joins the session as a message to DeepSeek; LLM7.io anonymous endpoint by default (OVHcloud listed as a no-key alternative) | Description bridge; this plugin adds image routing, with `vision_describe` covering descriptions on demand |
278
+ | [dsh-vision-proxy](https://github.com/Flyvhidbwo/dsh-vision-proxy) | Wraps a provider route and transcribes images into text in the request stream | Transcription bridge; this plugin wraps no provider — it rewrites routing through `agent/request` waterfalls |
279
+ | [dsh-vision-provider](https://github.com/libinyam/dsh-vision-provider) | Registers `DeepSeek + Vision` combined routes: images are described by the chosen vision model before reaching DeepSeek | Two-model bridge idea; this plugin adds automatic routing, fallback chains and tools on top |
280
+ | [modlens](https://github.com/liustack/modlens) | The first dsh vision plugin; reuses local Claude Code/Codex/OpenCode/Pi logins as vision engines | Engine-reuse idea; this plugin ships its own provider chain and depends on no other local CLI |
281
+ | [dsh-vision-toolkit](https://github.com/Anionex/dsh-vision-toolkit) | Ten intent-aware visual tools (Q&A/OCR/pixel verification/UI restoration), called explicitly on demand | Broader tool set; this plugin adds whole-turn auto-routing and a keyless free fallback |
282
+ | [dsh-tool-vision](https://github.com/Scorp1o117/dsh-tool-vision) | An `inspect_image` tool plus an `agent/pre-step` waterfall bridge (pasted images become tool hints before entering the log) | Similar waterfall bridge; this plugin adds turn routing, fallback chains, caching and the free endpoint |
315
283
 
316
284
  ## Provider fallback chain
317
285
 
@@ -321,7 +289,7 @@ The vision tools try backends in order and surface an error only after all of th
321
289
  2. **Local Ollama (optional, off by default)**: `localOllama.enabled` adds keyless, offline recognition through your local Ollama (for example qwen2.5vl);
322
290
  3. **Local LM Studio (optional, off by default)**: `localLmStudio.enabled` follows Ollama and requires the real model identifier shown in LM Studio Developer or returned by `/v1/models`;
323
291
  4. **Advanced custom HTTP vision endpoints**: legacy/advanced `httpProviders`, when present, run after the local backends;
324
- 5. **Built-in anonymous OVH fallback**: always last and never exposed in a model picker. The current quality-first chain is `Qwen3.5-397B-A17B` → `Qwen2.5-VL-72B-Instruct` → `Qwen3.6-27B` → `Mistral-Small-3.2-24B-Instruct-2506` → `Qwen3.5-9B`. OVH anonymous limits are **2 requests/minute per IP per model**. The five models have independent buckets, so spreading requests across them is about **10 RPM in theory**, subject to OVH's actual rate limiting. No signup or API key is required. Want more headroom? See [Free vision key channels](#free-vision-key-channels) — a free OVH access key lifts this same endpoint to 400 requests/minute.
292
+ 5. **Built-in anonymous OVH fallback**: always last and never exposed in a model picker. The current quality-first chain is `Qwen3.5-397B-A17B` → `Qwen2.5-VL-72B-Instruct` → `Qwen3.6-27B` → `Qwen3.5-9B`. OVH anonymous limits are **2 requests/minute per IP per model**. The four models have independent buckets, so spreading requests across them is about **8 RPM in theory**, subject to OVH's actual rate limiting. No signup or API key is required. OVH has scheduled the previously bundled Mistral Small 3.2 model for decommission on November 4, 2026; it is no longer part of the automatic fallback, but user-managed rows are not changed. For more capacity, configure your own model in DSH; see [Configure your own vision models](#configure-your-own-vision-models).
325
293
 
326
294
  > [!IMPORTANT]
327
295
  > This “vision chain” is the **eyes** used by Vision Router: each settings row selects one user vision model, while the lower-right chat picker selects the **brain/conversation model**. The two are deliberately separate. Text-only DeepSeek/opencode models are filtered out of the vision-backend dropdown, and the internal `Vision HTTP` transport route is no longer exposed to users.
@@ -620,6 +588,43 @@ Manual fallback: in VS Code, use “Save with Encoding” → `UTF-8` (without B
620
588
  - Secrets never travel: `apiKeyEnv` names a DSH credential reference; the value is resolved per call and never logged.
621
589
  - The settings write path goes through the settings service (schema-validated, revision-checked) — a stale or invalid save is rejected, not partially applied.
622
590
 
591
+ ## Design lineage
592
+
593
+ The deep-vision tool layer and UI-restoration workflow in this project were informed by [Anionex/agent-vision-toolkit](https://github.com/Anionex/agent-vision-toolkit) and its native DSH implementation [Anionex/dsh-vision-toolkit](https://github.com/Anionex/dsh-vision-toolkit). In particular, this project drew on their intent-driven tool selection, progressive tool exposure, pixel-diff verification loop, and parts of the visual-tool decomposition and naming, including long-screenshot OCR, foreground extraction, and HTML screenshot tooling.
594
+
595
+ All code in `dsh-vision-router` is independently implemented. On top of those referenced design ideas, this project independently developed its turn-level/tools-first vision routing, DSH admission/wrapper integration, multi-backend provider and failure fallback chains, built-in free vision chain, attachment/image-memory handling, caching, and related runtime resilience mechanisms.
596
+
597
+ We appreciate Anionex's prior work and the broader DSH community. Clear attribution and independent iteration can coexist; both help keep the DSH ecosystem open, collaborative, and healthy.
598
+
599
+ ## Acknowledgements
600
+
601
+ This project borrows ideas from all of the above — especially the keyless free-endpoint
602
+ exploration (LLM7.io and OVHcloud anonymous tiers) by
603
+ [dsh-vision-sidecar](https://github.com/121103qwq/dsh-vision-sidecar). Thanks to the authors of
604
+ [dsh-vision-proxy](https://github.com/Flyvhidbwo/dsh-vision-proxy),
605
+ [dsh-vision-provider](https://github.com/libinyam/dsh-vision-provider),
606
+ [modlens](https://github.com/liustack/modlens),
607
+ [dsh-vision-toolkit](https://github.com/Anionex/dsh-vision-toolkit), and
608
+ [dsh-tool-vision](https://github.com/Scorp1o117/dsh-tool-vision).
609
+
610
+ ## Release notes and community
611
+
612
+ Release details: [v3.1.0 release notes](docs/releases/v3.1.0.md). Check GitHub Releases or npm for the published version.
613
+
614
+ <p align="center">
615
+ <sub>Ecosystem:</sub>
616
+ <a href="https://dshplugin.app/plugins/dsh-vision-router">DSHPlugin.app</a> ·
617
+ <a href="https://github.com/diegosouzapw/awesome-omni-dsh-plugins">Awesome Omni DSH Plugins</a> ·
618
+ <a href="https://dshpluginhub.ai/plugins/dsh-vision-router">dshpluginhub.ai</a> ·
619
+ <a href="https://www.dsh.plus/en/plugins/dsh-vision-router/">dsh.plus</a> ·
620
+ <a href="https://dshplugins.ai/">dshplugins.ai</a> ·
621
+ <a href="https://dshmarket.com/p/ysr666/dsh-vision-router/">dsh-market</a>
622
+ </p>
623
+
624
+ <p align="center">Repository: <a href="https://github.com/ysr666/dsh-vision-router">GitHub (primary · issues, PRs and releases)</a> · <a href="https://atomgit.com/ysr666/dsh-vision-router">AtomGit (China mirror)</a></p>
625
+
626
+ <p align="center">💬 <strong>QQ community group: 1105463028</strong></p>
627
+
623
628
  ## License
624
629
 
625
630
  [MIT](LICENSE)