@anionex/dsh-vision-toolkit 0.1.9 → 0.1.11
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.i18n.yaml +2 -2
- package/README.md +135 -76
- package/README.zh.md +135 -70
- package/assets/dsh-view-example.png +0 -0
- package/lib/client.js +22 -8
- package/lib/client.js.map +1 -1
- package/lib/config.js +22 -13
- package/lib/config.js.map +1 -1
- package/lib/defaults.js +6 -0
- package/lib/defaults.js.map +1 -0
- package/lib/image-input-variants.js +204 -23
- package/lib/image-input-variants.js.map +1 -1
- package/lib/runtime.js +8 -2
- package/lib/runtime.js.map +1 -1
- package/lib/types/client/index.d.ts +1 -1
- package/lib/types/client/index.d.ts.map +1 -1
- package/lib/types/config.d.ts +4 -1
- package/lib/types/config.d.ts.map +1 -1
- package/lib/types/defaults.d.ts +6 -0
- package/lib/types/defaults.d.ts.map +1 -0
- package/lib/types/image-input-variants.d.ts +2 -2
- package/lib/types/image-input-variants.d.ts.map +1 -1
- package/lib/types/runtime.d.ts +1 -1
- package/lib/types/runtime.d.ts.map +1 -1
- package/lib/types/web.d.ts.map +1 -1
- package/lib/web.js +7 -1
- package/lib/web.js.map +1 -1
- package/package.json +1 -1
- package/src/client/index.tsx +23 -8
- package/src/config.ts +33 -14
- package/src/defaults.ts +5 -0
- package/src/image-input-variants.ts +226 -25
- package/src/runtime.ts +8 -3
- package/src/web.ts +7 -0
package/README.i18n.yaml
CHANGED
|
@@ -2,5 +2,5 @@
|
|
|
2
2
|
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
|
3
3
|
# after editing either side, bring the other along and re-record with:
|
|
4
4
|
# pnpm run verify-translation-pairing --write dsh-vision-dark-theme/README.md
|
|
5
|
-
README.md:
|
|
6
|
-
README.zh.md:
|
|
5
|
+
README.md: 0c81049d6e43d5c09056e7733349b45ea29fc22e
|
|
6
|
+
README.zh.md: 223b1d7f98b26eac72ab1ee0a76a83aed8b81673
|
package/README.md
CHANGED
|
@@ -1,38 +1,66 @@
|
|
|
1
|
-
|
|
1
|
+
<p align="center">
|
|
2
|
+
<img src="assets/hero.png" alt="DSH Vision Toolkit — native visual engineering for text-only DeepSeek Harness agents" />
|
|
3
|
+
</p>
|
|
4
|
+
|
|
5
|
+
<h1 align="center">DSH Vision Toolkit</h1>
|
|
6
|
+
|
|
7
|
+
<p align="center">
|
|
8
|
+
English | <a href="https://github.com/Anionex/dsh-vision-toolkit/blob/main/README.zh.md">中文</a>
|
|
9
|
+
</p>
|
|
10
|
+
|
|
11
|
+
<p align="center">
|
|
12
|
+
<a href="https://dshfind.com/en/plugins/Anionex/dsh-vision-toolkit"><img src="https://img.shields.io/badge/recommended%20by-dshfind-FFD700?style=flat-square" alt="Recommended by dshfind" /></a>
|
|
13
|
+
<a href="https://dshfind.com/en/plugins/Anionex/dsh-vision-toolkit"><img src="https://img.shields.io/badge/dshfind%20score-94%20%7C%20highest--rated%20plugin-5B4CF0?style=flat-square" alt="dshfind score: 94 — highest-rated plugin" /></a>
|
|
14
|
+
<a href="https://x.com/anion_ex"><img src="https://img.shields.io/badge/-@anion__ex-000000?style=flat-square&logo=x&logoColor=white" alt="X: @anion_ex" /></a>
|
|
15
|
+
<a href="https://github.com/Anionex/dsh-vision-toolkit/releases/tag/v0.1.11"><img src="https://img.shields.io/badge/release-v0.1.11-5B4CF0?style=flat-square" alt="Release v0.1.11" /></a>
|
|
16
|
+
<a href="tests"><img src="https://img.shields.io/badge/verified-233%20tests-2EA44F?style=flat-square" alt="Verified: 233 tests" /></a>
|
|
17
|
+
</p>
|
|
18
|
+
|
|
19
|
+
<p align="center">
|
|
20
|
+
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-0B7285?style=flat-square" alt="License: MIT" /></a>
|
|
21
|
+
<a href="package.json"><img src="https://img.shields.io/badge/Node.js-%5E22.19%20%7C%20%3E%3D24-339933?style=flat-square&logo=nodedotjs&logoColor=white" alt="Node.js ^22.19 or >=24" /></a>
|
|
22
|
+
<a href="runtime/requirements.lock"><img src="https://img.shields.io/badge/Python-3.11%2B-3776AB?style=flat-square&logo=python&logoColor=white" alt="Python 3.11+" /></a>
|
|
23
|
+
<a href="cordis.patch.yml"><img src="https://img.shields.io/badge/DSH-Web%20%2B%20Headless-5B4CF0?style=flat-square" alt="DSH Web and Headless profiles" /></a>
|
|
24
|
+
</p>
|
|
2
25
|
|
|
3
|
-
|
|
26
|
+
## Give your DSH agent eyes
|
|
4
27
|
|
|
5
|
-
|
|
6
|
-
[](https://dshfind.com/en/plugins/Anionex/dsh-vision-toolkit)
|
|
7
|
-
[](https://x.com/anion_ex)
|
|
8
|
-
[](https://github.com/Anionex/dsh-vision-toolkit/releases/tag/v0.1.9)
|
|
9
|
-
[](tests)
|
|
10
|
-
[](LICENSE)
|
|
11
|
-
[](package.json)
|
|
12
|
-
[](runtime/requirements.lock)
|
|
13
|
-
[](cordis.patch.yml)
|
|
28
|
+
Drop in a screenshot and let a text-only DeepSeek Harness agent inspect it, read it, locate elements, extract assets, rebuild interfaces, and measure whether the result matches.
|
|
14
29
|
|
|
15
|
-
|
|
30
|
+
DSH Vision Toolkit packages [`agent-vision-toolkit`](https://github.com/Anionex/agent-vision-toolkit) as a native DSH plugin. You get focused image Q&A, OCR, original-pixel coordinates, UI restoration, pixel comparison, downloadable results, and a Web Settings panel without assembling scripts by hand.
|
|
16
31
|
|
|
17
|
-
|
|
32
|
+
```sh
|
|
33
|
+
dsh plugin --profile web add @anionex/dsh-vision-toolkit
|
|
34
|
+
```
|
|
18
35
|
|
|
19
|
-
|
|
36
|
+
The npm package includes the visual toolkit snapshot and uses a managed runtime by default. **Normal installation does not require a source checkout or an `agentVisionToolkitPath`.**
|
|
20
37
|
|
|
21
38
|
**Upstream toolkit:** [Anionex/agent-vision-toolkit](https://github.com/Anionex/agent-vision-toolkit) · **Project website:** [agent-vision.anionex.me](https://agent-vision.anionex.me)
|
|
22
39
|
|
|
23
|
-
|
|
40
|
+
## What you can do
|
|
24
41
|
|
|
25
|
-
|
|
42
|
+
| Goal | What the agent can deliver |
|
|
43
|
+
|---|---|
|
|
44
|
+
| Understand a screenshot | Focused answers, visual descriptions, multi-image comparison, and OCR |
|
|
45
|
+
| Find an interface element | Original-image pixel coordinates with an optional labeled preview |
|
|
46
|
+
| Rebuild a page from a reference | Screenshot rendering, region-by-region diagnosis, and measurable iteration |
|
|
47
|
+
| Extract a usable asset | Cropped images, transparent foregrounds, dominant colors, or editable SVG |
|
|
48
|
+
| Read a long screenshot | Auditable chunks, Markdown output, manifests, and resumable OCR runs |
|
|
49
|
+
| Verify a visual result | A difference percentage, ranked mismatch regions, heatmap, and JSON report |
|
|
50
|
+
|
|
51
|
+
You can use remote vision only where it adds value. Cropping, tracing, pixel comparison, color analysis, foreground extraction, and HTML screenshots run locally.
|
|
26
52
|
|
|
27
|
-
|
|
53
|
+
## See it in action
|
|
28
54
|
|
|
29
|
-
|
|
55
|
+
The first example is a live DSH Web view. The next two examples come from the same `agent-vision-toolkit` lineage packaged with this plugin, and the last shows the workflow inside a live DeepSeek Harness Web session. See the [asset provenance record](assets/upstream/README.md) for source details.
|
|
30
56
|
|
|
31
|
-
|
|
57
|
+
### DSH view example
|
|
32
58
|
|
|
33
|
-
|
|
59
|
+
<p align="center">
|
|
60
|
+
<img src="assets/dsh-view-example.png" width="80%" alt="DSH Web session view in which a text-only DeepSeek-V4-Flash (Vision Toolkit) model answers a question about a pasted banner image." />
|
|
61
|
+
</p>
|
|
34
62
|
|
|
35
|
-
|
|
63
|
+
*A live DSH Web view: the user pastes a brand-banner screenshot, and the text-only model answers what the image contains through the `DeepSeek-V4-Flash (Vision Toolkit)` image-input variant.*
|
|
36
64
|
|
|
37
65
|
### Infographic restoration: screenshot to editable HTML/CSS
|
|
38
66
|
|
|
@@ -61,47 +89,60 @@ The first two panels are official upstream reference runs from the same pinned `
|
|
|
61
89
|
|
|
62
90
|
*Left: intent-aware image Q&A in DSH Web. Right: a DSH Web screenshot-debugging turn that lists the concrete UI differences and continues toward `vision_pixel_diff`. The upstream workflow source is the same [`agent-vision-toolkit` reference](https://github.com/Anionex/agent-vision-toolkit/blob/c27d1a300962b553c0884993c575cd3e819465ce/README.md#real-world-effects).*
|
|
63
91
|
|
|
64
|
-
DSH Vision Toolkit
|
|
92
|
+
DSH Vision Toolkit brings this workflow into DSH, where the result can become a file, a coordinate, a measured comparison, or the next step in the same session.
|
|
65
93
|
|
|
66
|
-
##
|
|
94
|
+
## From a rough match to pixel-perfect
|
|
67
95
|
|
|
68
|
-
The
|
|
96
|
+
The included UI-restoration workflow starts with an intentionally inaccurate HTML implementation. Vision Toolkit measures a `6.04%` difference, points to the worst regions, and helps drive the next iteration. The final render reaches an exact `0%` difference at `1200 × 720`.
|
|
69
97
|
|
|
70
98
|
<p>
|
|
71
99
|
<img src="examples/ui-restoration/assets/initial.png" width="49%" alt="Initial UI restoration candidate before Vision Toolkit iteration, with measurable layout and styling differences from the reference." />
|
|
72
100
|
<img src="examples/ui-restoration/assets/implementation.png" width="49%" alt="Final UI restoration output reproduced by the checked-in workflow with zero pixel difference from the reference." />
|
|
73
101
|
</p>
|
|
74
102
|
|
|
75
|
-
|
|
|
103
|
+
| Start | Result |
|
|
76
104
|
|---|---|
|
|
77
|
-
|
|
|
78
|
-
|
|
|
79
|
-
|
|
|
80
|
-
| Visual acceptance | Reproducible HTML screenshot → pixel diff example with a final `0%` difference |
|
|
105
|
+
| Reference image | A working HTML implementation you can open and edit |
|
|
106
|
+
| First comparison | `6.04%` difference across the visible problem regions |
|
|
107
|
+
| Final comparison | `0%` difference at `1200 × 720` |
|
|
81
108
|
|
|
82
|
-
##
|
|
109
|
+
## Why it feels different
|
|
83
110
|
|
|
84
|
-
- **
|
|
85
|
-
- **
|
|
86
|
-
- **
|
|
87
|
-
- **
|
|
88
|
-
- **
|
|
89
|
-
- **Use the same bundle in Web and Headless profiles:** Web adds cards, previews, Settings, and health actions; Headless receives the same tool semantics and complete structured results.
|
|
111
|
+
- **Ask for the thing you need.** “Where is the submit button?” and “Why does this screenshot differ from the reference?” lead to focused visual work instead of a generic caption.
|
|
112
|
+
- **Get evidence you can use.** The agent returns coordinates, OCR, measurements, JSON, and files you can open or pass to the next step.
|
|
113
|
+
- **Keep the workflow in DSH.** Credentials, Settings, Artifacts, Web cards, and Headless results live alongside the rest of your session.
|
|
114
|
+
- **Use local tools when you can.** Crop, trace, pixel comparison, color analysis, foreground extraction, and HTML screenshots do not consume a vision API request.
|
|
115
|
+
- **Repeat the loop.** Reference image → implementation → screenshot → pixel diff gives UI work a measurable finish line.
|
|
90
116
|
|
|
91
|
-
##
|
|
117
|
+
## Start in three steps
|
|
92
118
|
|
|
93
|
-
|
|
119
|
+
Use DeepSeek Harness `0.1.0-rc.6` or a compatible later `0.1.x` release. The plugin prepares its managed runtime on first use.
|
|
94
120
|
|
|
95
121
|
```sh
|
|
96
122
|
dsh plugin --profile web add @anionex/dsh-vision-toolkit
|
|
97
123
|
dsh plugin --profile headless add @anionex/dsh-vision-toolkit
|
|
98
|
-
dsh --profile web --dump-config | grep vision-toolkit
|
|
99
|
-
dsh --profile headless --dump-config | grep vision-toolkit
|
|
100
124
|
```
|
|
101
125
|
|
|
102
|
-
|
|
126
|
+
1. Restart your Web profile and open **Settings → Vision Toolkit**.
|
|
127
|
+
2. New installations use the built-in free Gemma 4 provider, so you can run **Test API connection** and **Test vision model** without an API key. To use another provider, edit the endpoint/model/protocol and provide its DSH Credential.
|
|
128
|
+
3. In a conversation, paste an image or put it in the workspace, invoke `/vision-tools`, and ask for a concrete visual task.
|
|
129
|
+
|
|
130
|
+
If you use an older DSH launcher, the profile may need `nodeLinker: hoisted` and `autoInstallPeers: false` before installation. Current launchers repair these settings for you.
|
|
131
|
+
|
|
132
|
+
Local crop, trace, pixel, color, foreground, and HTML operations do not require a visual API credential.
|
|
133
|
+
|
|
134
|
+
## Community Group
|
|
103
135
|
|
|
104
|
-
|
|
136
|
+
Join the `agent-vision-toolkit` community group to exchange usage tips, share feedback, and suggest improvements.
|
|
137
|
+
|
|
138
|
+
<p align="center">
|
|
139
|
+
<img src="assets/community-group-qr.png" alt="QR code for the agent-vision-toolkit community group" width="260">
|
|
140
|
+
</p>
|
|
141
|
+
|
|
142
|
+
> **No local path is required.** Keep the default `runtime.mode: managed` for the normal npm installation. The optional `runtime.agentVisionToolkitPath` setting is only for developers or controlled deployments that deliberately use an external pinned checkout.
|
|
143
|
+
|
|
144
|
+
<details>
|
|
145
|
+
<summary><strong>Technical architecture</strong></summary>
|
|
105
146
|
|
|
106
147
|
## How it works
|
|
107
148
|
|
|
@@ -124,6 +165,8 @@ flowchart LR
|
|
|
124
165
|
|
|
125
166
|
Tool definitions call one runtime; the runtime validates paths, limits, credentials, cancellation, and deadlines before dispatching to the pinned upstream snapshot or configured vision provider endpoint. Web presentation consumes the same structured results and Artifact descriptors, so it does not change Headless behavior. Health, connection testing, and version inspection stay in Settings rather than model tool schemas.
|
|
126
167
|
|
|
168
|
+
</details>
|
|
169
|
+
|
|
127
170
|
## Tools
|
|
128
171
|
|
|
129
172
|
| Tool | Execution | Structured result | Artifact delivery |
|
|
@@ -141,6 +184,9 @@ Tool definitions call one runtime; the runtime validates paths, limits, credenti
|
|
|
141
184
|
|
|
142
185
|
The plugin does not reimplement visual algorithms. Its DSH-owned layer validates paths and limits, resolves credentials, calls the pinned upstream scripts with argv vectors, parses their exact output contracts, classifies failures, describes files, and projects results to the model and Web client.
|
|
143
186
|
|
|
187
|
+
<details>
|
|
188
|
+
<summary><strong>Advanced model behavior</strong></summary>
|
|
189
|
+
|
|
144
190
|
## Progressive model exposure
|
|
145
191
|
|
|
146
192
|
Runtime readiness is profile-wide, but the ten visual execution schemas are Agent-scoped. Before an Agent loads `vision-tools`, the plugin contributes only the small `vision_toolkit_activate` bootstrap; the visual tools are absent from that Agent's request schema. A successful call to the standard `skill` tool with `name="vision-tools"` mounts all ten tools automatically for the next model step and hides the bootstrap. A direct `/vision-tools` invocation injects the Skill instructions; if the visual tools are still absent, those instructions require one `vision_toolkit_activate` call. Activation affects only that Agent, restores when the Session contains durable evidence matching the bundled Skill version, and lasts until the Agent or plugin is disposed.
|
|
@@ -149,18 +195,20 @@ Health checks, connection testing, and plugin/upstream version inspection are ad
|
|
|
149
195
|
|
|
150
196
|
## Image-input variants for text-only models
|
|
151
197
|
|
|
152
|
-
Text-only model routes get sibling model-selector entries named `<model> (Vision Toolkit)` under a matching provider group.
|
|
198
|
+
Text-only model routes get sibling model-selector entries named `<model> (Vision Toolkit)` under a matching provider group. DSH cannot pass a pasted attachment's local path through its native image block, so the default paste flow copies each image into the session workspace and inserts its absolute path into the model-visible message. The DSH model can then call `vision_glance` (or another visual tool) with that path, using the same focus-hinted bridge and `[vision model description]` channel markers as `agent-vision-toolkit`. The session log contains the durable path reference and the UI keeps the paste record.
|
|
199
|
+
|
|
200
|
+
A variant is still registered automatically for every model the host positively declares text-only (for example the DeepSeek chat family), but automatic switching is opt-in. With the default `autoSwitch: false`, a text-only session uses the paste-to-path flow so the DSH model receives a usable absolute path. If `autoSwitch: true` is explicitly enabled, the browser switches to `<model> (Vision Toolkit)` and the server-side image-input variant rewrites native image blocks into descriptions. The host's verdict uses the exact model route the browser read from the live model catalog, with the selector label as fallback; unconfirmed or image-capable routes keep the native flow.
|
|
153
201
|
|
|
154
|
-
|
|
202
|
+
Description conversion needs the configured vision provider and its credential when the opt-in image-input variant is used; when the runtime is not ready or a read fails, the wire block degrades to the upstream-compatible `[vision unavailable: ...]` note instead of failing the turn. The bridge does not treat injected context files as the current user intent, and it uses the latest assistant paragraph when a tool-fetched image is being described. Disable variants with `imageInputVariants.enabled: false`, restrict the wrapped routes with `imageInputVariants.providers`, or opt into native attachment switching with `imageInputVariants.autoSwitch: true`.
|
|
155
203
|
|
|
156
|
-
|
|
204
|
+
</details>
|
|
157
205
|
|
|
158
206
|
## Requirements
|
|
159
207
|
|
|
160
208
|
- DeepSeek Harness with a Web or Headless profile and `pnpm` available to `dsh plugin`.
|
|
161
209
|
- Python 3.11 or newer. Managed mode creates an isolated environment, so users do not install the upstream CLI or Python packages manually.
|
|
162
210
|
- Network access on the first managed-runtime activation unless the exact packages in `runtime/requirements.lock` are already available in the configured package cache.
|
|
163
|
-
-
|
|
211
|
+
- The built-in free Gemma 4 provider is ready for `vision_glance`, `vision_ground`, `vision_detect`, and non-split-only long-screenshot OCR. A DSH Credential is required only when a custom OpenAI-compatible or Anthropic endpoint is configured. Local tools remain usable without either provider.
|
|
164
212
|
- Chrome, Chromium, or Edge only for `vision_html_screenshot`; all other tools remain available when no supported browser is installed.
|
|
165
213
|
- PNG, JPEG, GIF, or WebP inputs inside the session workspace or an explicitly configured `allowedDirs` root.
|
|
166
214
|
|
|
@@ -201,7 +249,7 @@ dsh plugin --profile web remove @dsh-external/dsh-vision-toolkit
|
|
|
201
249
|
dsh plugin --profile web add @anionex/dsh-vision-toolkit
|
|
202
250
|
```
|
|
203
251
|
|
|
204
|
-
After restarting, Settings → Vision should report plugin version **0.1.
|
|
252
|
+
After restarting, Settings → Vision should report plugin version **0.1.11**. The built-in free provider is selected automatically; custom providers still use the configured DSH Credential.
|
|
205
253
|
|
|
206
254
|
For a registry installation, update the dependency through the profile package manager:
|
|
207
255
|
|
|
@@ -229,16 +277,16 @@ The bundle defaults to the managed runtime. A profile patch can override the pro
|
|
|
229
277
|
- id: vision-toolkit
|
|
230
278
|
config:
|
|
231
279
|
provider:
|
|
232
|
-
baseUrl: https://
|
|
233
|
-
credential:
|
|
234
|
-
model:
|
|
280
|
+
baseUrl: https://vision.anionex.me/v1
|
|
281
|
+
credential: ANIONEX_FREE_VISION
|
|
282
|
+
model: gemma-4-26b-a4b-it
|
|
235
283
|
protocol: openai
|
|
236
284
|
anthropicThinking: omit
|
|
237
285
|
userAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
|
|
238
286
|
language: zh
|
|
239
287
|
timeoutMs: 60000
|
|
240
|
-
maxImageBytes:
|
|
241
|
-
maxImagePixels:
|
|
288
|
+
maxImageBytes: 4194304
|
|
289
|
+
maxImagePixels: 20000000
|
|
242
290
|
concurrency: 4
|
|
243
291
|
runtime:
|
|
244
292
|
mode: managed
|
|
@@ -246,23 +294,23 @@ The bundle defaults to the managed runtime. A profile patch can override the pro
|
|
|
246
294
|
imageInputVariants:
|
|
247
295
|
enabled: true
|
|
248
296
|
providers: []
|
|
249
|
-
autoSwitch:
|
|
297
|
+
autoSwitch: false
|
|
250
298
|
```
|
|
251
299
|
|
|
252
300
|
### Configuration fields
|
|
253
301
|
|
|
254
302
|
| Field | Default | Contract |
|
|
255
303
|
|---|---|---|
|
|
256
|
-
| `provider.baseUrl` | `https://
|
|
257
|
-
| `provider.credential` | `
|
|
258
|
-
| `provider.model` | `
|
|
304
|
+
| `provider.baseUrl` | `https://vision.anionex.me/v1` | Built-in free OpenAI-compatible endpoint; custom providers may use another base URL, normalized without trailing slashes |
|
|
305
|
+
| `provider.credential` | `ANIONEX_FREE_VISION` | Read-only built-in reference for the free service; custom providers use a DSH Credential reference, never a secret value |
|
|
306
|
+
| `provider.model` | `gemma-4-26b-a4b-it` | Multimodal model name sent to remote tools |
|
|
259
307
|
| `provider.protocol` | `openai` | `openai` sends Chat Completions requests; `anthropic` sends native Messages requests |
|
|
260
308
|
| `provider.anthropicThinking` | `omit` | Anthropic thinking field. `omit` sends no thinking field and has the broadest compatibility. Use `disabled` or `adaptive` only when the selected model documents that mode; restore `omit` first if the provider returns HTTP 400. |
|
|
261
309
|
| `provider.userAgent` | browser-compatible default | User-Agent sent by vision requests and explicit connection tests; override it for provider or proxy compatibility |
|
|
262
310
|
| `language` | `zh` | Vision output language: `zh` or `en` |
|
|
263
311
|
| `timeoutMs` | `60000` | Whole-operation deadline, 1000-600000 ms; each tool may request a narrower override |
|
|
264
|
-
| `maxImageBytes` | `
|
|
265
|
-
| `maxImagePixels` | `
|
|
312
|
+
| `maxImageBytes` | `4194304` | Encoded-byte limit per input image; the built-in free service accepts up to 4 MiB |
|
|
313
|
+
| `maxImagePixels` | `20000000` | Decoded-pixel limit per input image; the built-in free service accepts up to 20,000,000 pixels |
|
|
266
314
|
| `concurrency` | `4` | In-flight operations per session, 1-16 |
|
|
267
315
|
| `runtime.mode` | `managed` | `managed` uses the packaged snapshot; `external` accepts only the exact pin |
|
|
268
316
|
| `runtime.agentVisionToolkitPath` | unset | Required in `external` mode; exported exact snapshot or clean pinned Git checkout |
|
|
@@ -270,19 +318,34 @@ The bundle defaults to the managed runtime. A profile patch can override the pro
|
|
|
270
318
|
| `allowedDirs` | `[]` | Additional realpath-resolved input roots; the session workspace is always allowed |
|
|
271
319
|
| `imageInputVariants.enabled` | `true` | Register image-input variant entries for text-only model routes in the model selector |
|
|
272
320
|
| `imageInputVariants.providers` | `[]` | Restrict wrapped upstream routes by provider id; empty wraps every eligible route |
|
|
273
|
-
| `imageInputVariants.autoSwitch` | `
|
|
321
|
+
| `imageInputVariants.autoSwitch` | `false` | Opt into automatically switching a text-only session to its image-input variant on paste; the default `false` keeps the DSH-compatible paste-to-path flow |
|
|
274
322
|
|
|
275
323
|
### Credentials
|
|
276
324
|
|
|
277
|
-
The
|
|
325
|
+
The built-in free provider uses the fixed `ANIONEX_FREE_VISION` reference and does not accept or store a user API key. If you change the endpoint, model, or protocol to a custom provider, the write-only **API key** field unlocks; saving a non-empty value writes it under the advanced **Credential name** reference. Headless deployments can pre-provision that custom reference in `$DSH_HOME/.credentials.yaml`.
|
|
278
326
|
|
|
279
327
|
Settings store only the reference, never the value. The browser does not receive a stored value, and a successful save clears the field instead of echoing it. Remote operations resolve the reference once per call and inject the value only into that subprocess environment. The plugin excludes user `.env` files, checkout `.env` files, `PYTHONPATH`, `PYTHONHOME`, `VIRTUAL_ENV`, and user site-packages so ambient Python or upstream configuration cannot override the selected DSH provider. Logs, errors, tool results, Artifact metadata, and Settings responses never contain the secret.
|
|
280
328
|
|
|
281
|
-
###
|
|
329
|
+
### Built-in free service limits
|
|
330
|
+
|
|
331
|
+
The public service is shared and intended as a zero-configuration default, not an unlimited private endpoint. Limits are enforced by the proxy and returned as OpenAI-style errors with a reason code and readable message; rate-limit responses also include `Retry-After` and request-quota headers.
|
|
332
|
+
|
|
333
|
+
| Limit | Current value |
|
|
334
|
+
|---|---:|
|
|
335
|
+
| Per client | 100 requests per UTC day |
|
|
336
|
+
| Global service | 400 requests per UTC day |
|
|
337
|
+
| Burst | 20 requests per 60 seconds |
|
|
338
|
+
| Image bytes | 4 MiB per image |
|
|
339
|
+
| Decoded pixels | 20,000,000 per image |
|
|
340
|
+
| Output | 512 tokens maximum |
|
|
341
|
+
|
|
342
|
+
### Managed runtime and optional external runtime
|
|
282
343
|
|
|
283
344
|
Managed mode verifies `vendor/agent-vision-toolkit/UPSTREAM_MANIFEST.json`, prefers `uv`, falls back to `venv` plus pip, installs exact versions from `runtime/requirements.lock`, coordinates concurrent preparation with a heartbeat lock, and publishes a staged environment only after all probes pass.
|
|
284
345
|
|
|
285
|
-
|
|
346
|
+
Most users should stop at managed mode. It is included in the npm package and prepares the pinned Python environment for you.
|
|
347
|
+
|
|
348
|
+
The optional external mode is for plugin development or controlled deployments that already maintain the exact upstream checkout:
|
|
286
349
|
|
|
287
350
|
```yaml
|
|
288
351
|
- id: vision-toolkit
|
|
@@ -299,6 +362,8 @@ The path must be an exported copy matching the packaged manifest or the root of
|
|
|
299
362
|
|
|
300
363
|
The Web profile registers a Vision Toolkit Settings section for the provider URL, Credential reference, model, OpenAI/Anthropic protocol, Anthropic thinking mode, User-Agent, language, timeout, byte/pixel limits, concurrency, runtime mode, Python override, external source path, and allowed directories. It also shows plugin/upstream versions, the active runtime generation, non-secret Credential configured/source/writable facts, runtime paths, health results, and Artifact-route availability.
|
|
301
364
|
|
|
365
|
+
The **Plugin updates** card checks the profile's configured npm registry for a newer `@anionex/dsh-vision-toolkit` release. **Update and restart** installs that exact confirmed version into the current DSH profile, verifies the installed package, starts an independent restart helper, and gracefully restarts DSH Web; the open page waits for the replacement process and reloads after the new plugin version is serving. The action is same-origin, fixed to this package, serialized, and unavailable for `link:`, `file:`, workspace, git, URL, transitive, ambiguous, read-only, or missing-`pnpm` installations so local development sources are never overwritten. A restart can interrupt work that is currently running, so the UI requires an explicit confirmation.
|
|
366
|
+
|
|
302
367
|
`Save and apply` validates the complete value, prepares the candidate Python/upstream runtime, commits the Settings revision, and only then atomically switches generations. A rejected candidate leaves the previous generation serving and is reported separately from a genuinely unavailable runtime. `Reload` always restores the authoritative saved value, even when its revision did not change, so a rejected browser draft is discarded. If initial startup cannot prepare a runtime, the Settings route remains available so a valid configuration can make the first generation operational. A stale browser revision receives a conflict instead of overwriting a newer save; reload before retrying. A read-only Settings provider allows inspection and health checks but disables saves.
|
|
303
368
|
|
|
304
369
|
`Run health check` performs local checks only. `Test API connection` is an explicit action that sends the configured Credential to `GET /models`; OpenAI uses Bearer authentication, while Anthropic uses `x-api-key` and `anthropic-version`. That lightweight probe uploads no image and creates no completion. `Test vision model` separately sends the bundled `assets/vision-model-test.png` through the same multimodal runtime path as `vision_glance`; it creates one real completion and is the authoritative check that the selected endpoint, credential, model, protocol, and upstream account can process images. The Vision model health card displays a dedicated `Verified`, `Not tested`, or `Test failed` tag, so an HTTP 200 response from `/models` is not presented as a successful image test. Plugin load and ordinary Settings reads never make either request.
|
|
@@ -341,11 +406,6 @@ npm run example:ui-restoration:write
|
|
|
341
406
|
|
|
342
407
|
The committed evidence records an initial `6.04%` difference across six non-zero worst regions and a final `0%` difference with no non-zero worst region. Check mode reproduces the tool path and verifies the committed assets; write mode intentionally refreshes the evidence.
|
|
343
408
|
|
|
344
|
-
## Communication group
|
|
345
|
-
|
|
346
|
-
<img width="254" height="328" alt="image" src="https://github.com/user-attachments/assets/63c25c69-c3ba-4c47-8dee-98d60fe3954d" />
|
|
347
|
-
|
|
348
|
-
|
|
349
409
|
## Troubleshooting
|
|
350
410
|
|
|
351
411
|
| Symptom | Resolution |
|
|
@@ -382,7 +442,14 @@ Update the upstream snapshot only through `pnpm run upstream:sync -- <checkout>`
|
|
|
382
442
|
|
|
383
443
|
## Project status and scope
|
|
384
444
|
|
|
385
|
-
Version `0.1.
|
|
445
|
+
Version `0.1.11` is the current public npm release. The product focuses on screenshot understanding, visual grounding, OCR, asset extraction, UI restoration, and pixel-level verification in DSH Web and Headless profiles. Web upload, drag-and-drop, camera/video/audio/document ingestion, interactive box editing, automatic GUI clicking, service clusters, model routing, model voting, and cross-session vision caches remain outside the current product.
|
|
446
|
+
|
|
447
|
+
<details>
|
|
448
|
+
<summary><strong>Maintainer scope note</strong></summary>
|
|
449
|
+
|
|
450
|
+
The stable `ctx.visionToolkit` service and capability-discovery API remain unpublished until an independent plugin becomes a real consumer. This keeps the public integration surface tied to a tested use case rather than an unvalidated ecosystem contract.
|
|
451
|
+
|
|
452
|
+
</details>
|
|
386
453
|
|
|
387
454
|
## Community and About
|
|
388
455
|
|
|
@@ -401,11 +468,3 @@ If you would like to follow my future work, [follow me on X](https://x.com/anion
|
|
|
401
468
|
## License
|
|
402
469
|
|
|
403
470
|
The plugin is MIT-licensed. The packaged `agent-vision-toolkit` snapshot retains its upstream MIT license in `vendor/agent-vision-toolkit/LICENSE` and remains the sole implementation of its visual algorithms.
|
|
404
|
-
|
|
405
|
-
## Join the Community Group
|
|
406
|
-
|
|
407
|
-
You are welcome to join the `agent-vision-toolkit` community group to exchange usage tips, share feedback, and suggest improvements. Scan the QR code below to join.
|
|
408
|
-
|
|
409
|
-
<p align="center">
|
|
410
|
-
<img src="assets/community-group-qr.png" alt="QR code for the agent-vision-toolkit community group" width="320">
|
|
411
|
-
</p>
|