@anionex/dsh-vision-toolkit 0.1.10 → 0.1.12
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.i18n.yaml +2 -2
- package/README.md +107 -56
- package/README.zh.md +107 -55
- package/assets/dsh-view-example.png +0 -0
- package/assets/hero.png +0 -0
- package/lib/client.js +1 -1
- package/lib/client.js.map +1 -1
- package/lib/config.js +8 -1
- package/lib/config.js.map +1 -1
- package/lib/defaults.js +1 -1
- package/lib/defaults.js.map +1 -1
- package/lib/image-input-variants.js +65 -23
- package/lib/image-input-variants.js.map +1 -1
- package/lib/paste-images.js +6 -1
- package/lib/paste-images.js.map +1 -1
- package/lib/types/config.d.ts +2 -2
- package/lib/types/config.d.ts.map +1 -1
- package/lib/types/defaults.d.ts +1 -1
- package/lib/types/defaults.d.ts.map +1 -1
- package/lib/types/image-input-variants.d.ts +6 -4
- package/lib/types/image-input-variants.d.ts.map +1 -1
- package/lib/types/paste-images.d.ts +10 -0
- package/lib/types/paste-images.d.ts.map +1 -1
- package/package.json +1 -1
- package/src/client/index.tsx +1 -1
- package/src/config.ts +11 -3
- package/src/defaults.ts +1 -2
- package/src/image-input-variants.ts +81 -23
- package/src/paste-images.ts +7 -2
package/README.i18n.yaml
CHANGED
|
@@ -2,5 +2,5 @@
|
|
|
2
2
|
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
|
3
3
|
# after editing either side, bring the other along and re-record with:
|
|
4
4
|
# pnpm run verify-translation-pairing --write dsh-vision-dark-theme/README.md
|
|
5
|
-
README.md:
|
|
6
|
-
README.zh.md:
|
|
5
|
+
README.md: 0c81049d6e43d5c09056e7733349b45ea29fc22e
|
|
6
|
+
README.zh.md: 223b1d7f98b26eac72ab1ee0a76a83aed8b81673
|
package/README.md
CHANGED
|
@@ -1,38 +1,66 @@
|
|
|
1
|
-
|
|
1
|
+
<p align="center">
|
|
2
|
+
<img src="assets/hero.png" alt="DSH Vision Toolkit — native visual engineering for text-only DeepSeek Harness agents" />
|
|
3
|
+
</p>
|
|
4
|
+
|
|
5
|
+
<h1 align="center">DSH Vision Toolkit</h1>
|
|
6
|
+
|
|
7
|
+
<p align="center">
|
|
8
|
+
English | <a href="https://github.com/Anionex/dsh-vision-toolkit/blob/main/README.zh.md">中文</a>
|
|
9
|
+
</p>
|
|
10
|
+
|
|
11
|
+
<p align="center">
|
|
12
|
+
<a href="https://dshfind.com/en/plugins/Anionex/dsh-vision-toolkit"><img src="https://img.shields.io/badge/recommended%20by-dshfind-FFD700?style=flat-square" alt="Recommended by dshfind" /></a>
|
|
13
|
+
<a href="https://dshfind.com/en/plugins/Anionex/dsh-vision-toolkit"><img src="https://img.shields.io/badge/dshfind%20score-94%20%7C%20highest--rated%20plugin-5B4CF0?style=flat-square" alt="dshfind score: 94 — highest-rated plugin" /></a>
|
|
14
|
+
<a href="https://x.com/anion_ex"><img src="https://img.shields.io/badge/-@anion__ex-000000?style=flat-square&logo=x&logoColor=white" alt="X: @anion_ex" /></a>
|
|
15
|
+
<a href="https://github.com/Anionex/dsh-vision-toolkit/releases/tag/v0.1.12"><img src="https://img.shields.io/badge/release-v0.1.12-5B4CF0?style=flat-square" alt="Release v0.1.12" /></a>
|
|
16
|
+
<a href="tests"><img src="https://img.shields.io/badge/verified-233%20tests-2EA44F?style=flat-square" alt="Verified: 233 tests" /></a>
|
|
17
|
+
</p>
|
|
18
|
+
|
|
19
|
+
<p align="center">
|
|
20
|
+
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-0B7285?style=flat-square" alt="License: MIT" /></a>
|
|
21
|
+
<a href="package.json"><img src="https://img.shields.io/badge/Node.js-%5E22.19%20%7C%20%3E%3D24-339933?style=flat-square&logo=nodedotjs&logoColor=white" alt="Node.js ^22.19 or >=24" /></a>
|
|
22
|
+
<a href="runtime/requirements.lock"><img src="https://img.shields.io/badge/Python-3.11%2B-3776AB?style=flat-square&logo=python&logoColor=white" alt="Python 3.11+" /></a>
|
|
23
|
+
<a href="cordis.patch.yml"><img src="https://img.shields.io/badge/DSH-Web%20%2B%20Headless-5B4CF0?style=flat-square" alt="DSH Web and Headless profiles" /></a>
|
|
24
|
+
</p>
|
|
2
25
|
|
|
3
|
-
|
|
26
|
+
## Give your DSH agent eyes
|
|
4
27
|
|
|
5
|
-
|
|
6
|
-
[](https://dshfind.com/en/plugins/Anionex/dsh-vision-toolkit)
|
|
7
|
-
[](https://x.com/anion_ex)
|
|
8
|
-
[](https://github.com/Anionex/dsh-vision-toolkit/releases/tag/v0.1.10)
|
|
9
|
-
[](tests)
|
|
10
|
-
[](LICENSE)
|
|
11
|
-
[](package.json)
|
|
12
|
-
[](runtime/requirements.lock)
|
|
13
|
-
[](cordis.patch.yml)
|
|
28
|
+
Drop in a screenshot and let a text-only DeepSeek Harness agent inspect it, read it, locate elements, extract assets, rebuild interfaces, and measure whether the result matches.
|
|
14
29
|
|
|
15
|
-
|
|
30
|
+
DSH Vision Toolkit packages [`agent-vision-toolkit`](https://github.com/Anionex/agent-vision-toolkit) as a native DSH plugin. You get focused image Q&A, OCR, original-pixel coordinates, UI restoration, pixel comparison, downloadable results, and a Web Settings panel without assembling scripts by hand.
|
|
16
31
|
|
|
17
|
-
|
|
32
|
+
```sh
|
|
33
|
+
dsh plugin --profile web add @anionex/dsh-vision-toolkit
|
|
34
|
+
```
|
|
18
35
|
|
|
19
|
-
|
|
36
|
+
The npm package includes the visual toolkit snapshot and uses a managed runtime by default. **Normal installation does not require a source checkout or an `agentVisionToolkitPath`.**
|
|
20
37
|
|
|
21
38
|
**Upstream toolkit:** [Anionex/agent-vision-toolkit](https://github.com/Anionex/agent-vision-toolkit) · **Project website:** [agent-vision.anionex.me](https://agent-vision.anionex.me)
|
|
22
39
|
|
|
23
|
-
|
|
40
|
+
## What you can do
|
|
41
|
+
|
|
42
|
+
| Goal | What the agent can deliver |
|
|
43
|
+
|---|---|
|
|
44
|
+
| Understand a screenshot | Focused answers, visual descriptions, multi-image comparison, and OCR |
|
|
45
|
+
| Find an interface element | Original-image pixel coordinates with an optional labeled preview |
|
|
46
|
+
| Rebuild a page from a reference | Screenshot rendering, region-by-region diagnosis, and measurable iteration |
|
|
47
|
+
| Extract a usable asset | Cropped images, transparent foregrounds, dominant colors, or editable SVG |
|
|
48
|
+
| Read a long screenshot | Auditable chunks, Markdown output, manifests, and resumable OCR runs |
|
|
49
|
+
| Verify a visual result | A difference percentage, ranked mismatch regions, heatmap, and JSON report |
|
|
24
50
|
|
|
25
|
-
|
|
51
|
+
You can use remote vision only where it adds value. Cropping, tracing, pixel comparison, color analysis, foreground extraction, and HTML screenshots run locally.
|
|
26
52
|
|
|
27
|
-
|
|
53
|
+
## See it in action
|
|
28
54
|
|
|
29
|
-
|
|
55
|
+
The first example is a live DSH Web view. The next two examples come from the same `agent-vision-toolkit` lineage packaged with this plugin, and the last shows the workflow inside a live DeepSeek Harness Web session. See the [asset provenance record](assets/upstream/README.md) for source details.
|
|
30
56
|
|
|
31
|
-
|
|
57
|
+
### DSH view example
|
|
32
58
|
|
|
33
|
-
|
|
59
|
+
<p align="center">
|
|
60
|
+
<img src="assets/dsh-view-example.png" width="80%" alt="DSH Web session view in which a text-only DeepSeek-V4-Flash (Vision Toolkit) model answers a question about a pasted banner image." />
|
|
61
|
+
</p>
|
|
34
62
|
|
|
35
|
-
|
|
63
|
+
*A live DSH Web view: the user pastes a brand-banner screenshot, and the text-only model answers what the image contains through the `DeepSeek-V4-Flash (Vision Toolkit)` image-input variant.*
|
|
36
64
|
|
|
37
65
|
### Infographic restoration: screenshot to editable HTML/CSS
|
|
38
66
|
|
|
@@ -61,47 +89,47 @@ The first two panels are official upstream reference runs from the same pinned `
|
|
|
61
89
|
|
|
62
90
|
*Left: intent-aware image Q&A in DSH Web. Right: a DSH Web screenshot-debugging turn that lists the concrete UI differences and continues toward `vision_pixel_diff`. The upstream workflow source is the same [`agent-vision-toolkit` reference](https://github.com/Anionex/agent-vision-toolkit/blob/c27d1a300962b553c0884993c575cd3e819465ce/README.md#real-world-effects).*
|
|
63
91
|
|
|
64
|
-
DSH Vision Toolkit
|
|
92
|
+
DSH Vision Toolkit brings this workflow into DSH, where the result can become a file, a coordinate, a measured comparison, or the next step in the same session.
|
|
65
93
|
|
|
66
|
-
##
|
|
94
|
+
## From a rough match to pixel-perfect
|
|
67
95
|
|
|
68
|
-
The
|
|
96
|
+
The included UI-restoration workflow starts with an intentionally inaccurate HTML implementation. Vision Toolkit measures a `6.04%` difference, points to the worst regions, and helps drive the next iteration. The final render reaches an exact `0%` difference at `1200 × 720`.
|
|
69
97
|
|
|
70
98
|
<p>
|
|
71
99
|
<img src="examples/ui-restoration/assets/initial.png" width="49%" alt="Initial UI restoration candidate before Vision Toolkit iteration, with measurable layout and styling differences from the reference." />
|
|
72
100
|
<img src="examples/ui-restoration/assets/implementation.png" width="49%" alt="Final UI restoration output reproduced by the checked-in workflow with zero pixel difference from the reference." />
|
|
73
101
|
</p>
|
|
74
102
|
|
|
75
|
-
|
|
|
103
|
+
| Start | Result |
|
|
76
104
|
|---|---|
|
|
77
|
-
|
|
|
78
|
-
|
|
|
79
|
-
|
|
|
80
|
-
| Visual acceptance | Reproducible HTML screenshot → pixel diff example with a final `0%` difference |
|
|
105
|
+
| Reference image | A working HTML implementation you can open and edit |
|
|
106
|
+
| First comparison | `6.04%` difference across the visible problem regions |
|
|
107
|
+
| Final comparison | `0%` difference at `1200 × 720` |
|
|
81
108
|
|
|
82
|
-
##
|
|
109
|
+
## Why it feels different
|
|
83
110
|
|
|
84
|
-
- **
|
|
85
|
-
- **
|
|
86
|
-
- **
|
|
87
|
-
- **
|
|
88
|
-
- **
|
|
89
|
-
- **Use the same bundle in Web and Headless profiles:** Web adds cards, previews, Settings, and health actions; Headless receives the same tool semantics and complete structured results.
|
|
111
|
+
- **Ask for the thing you need.** “Where is the submit button?” and “Why does this screenshot differ from the reference?” lead to focused visual work instead of a generic caption.
|
|
112
|
+
- **Get evidence you can use.** The agent returns coordinates, OCR, measurements, JSON, and files you can open or pass to the next step.
|
|
113
|
+
- **Keep the workflow in DSH.** Credentials, Settings, Artifacts, Web cards, and Headless results live alongside the rest of your session.
|
|
114
|
+
- **Use local tools when you can.** Crop, trace, pixel comparison, color analysis, foreground extraction, and HTML screenshots do not consume a vision API request.
|
|
115
|
+
- **Repeat the loop.** Reference image → implementation → screenshot → pixel diff gives UI work a measurable finish line.
|
|
90
116
|
|
|
91
|
-
##
|
|
117
|
+
## Start in three steps
|
|
92
118
|
|
|
93
|
-
|
|
119
|
+
Use DeepSeek Harness `0.1.0-rc.6` or a compatible later `0.1.x` release. The plugin prepares its managed runtime on first use.
|
|
94
120
|
|
|
95
121
|
```sh
|
|
96
122
|
dsh plugin --profile web add @anionex/dsh-vision-toolkit
|
|
97
123
|
dsh plugin --profile headless add @anionex/dsh-vision-toolkit
|
|
98
|
-
dsh --profile web --dump-config | grep vision-toolkit
|
|
99
|
-
dsh --profile headless --dump-config | grep vision-toolkit
|
|
100
124
|
```
|
|
101
125
|
|
|
102
|
-
|
|
126
|
+
1. Restart your Web profile and open **Settings → Vision Toolkit**.
|
|
127
|
+
2. New installations use the built-in free Gemma 4 provider, so you can run **Test API connection** and **Test vision model** without an API key. To use another provider, edit the endpoint/model/protocol and provide its DSH Credential.
|
|
128
|
+
3. In a conversation, paste an image or put it in the workspace, invoke `/vision-tools`, and ask for a concrete visual task.
|
|
129
|
+
|
|
130
|
+
If you use an older DSH launcher, the profile may need `nodeLinker: hoisted` and `autoInstallPeers: false` before installation. Current launchers repair these settings for you.
|
|
103
131
|
|
|
104
|
-
|
|
132
|
+
Local crop, trace, pixel, color, foreground, and HTML operations do not require a visual API credential.
|
|
105
133
|
|
|
106
134
|
## Community Group
|
|
107
135
|
|
|
@@ -111,6 +139,11 @@ Join the `agent-vision-toolkit` community group to exchange usage tips, share fe
|
|
|
111
139
|
<img src="assets/community-group-qr.png" alt="QR code for the agent-vision-toolkit community group" width="260">
|
|
112
140
|
</p>
|
|
113
141
|
|
|
142
|
+
> **No local path is required.** Keep the default `runtime.mode: managed` for the normal npm installation. The optional `runtime.agentVisionToolkitPath` setting is only for developers or controlled deployments that deliberately use an external pinned checkout.
|
|
143
|
+
|
|
144
|
+
<details>
|
|
145
|
+
<summary><strong>Technical architecture</strong></summary>
|
|
146
|
+
|
|
114
147
|
## How it works
|
|
115
148
|
|
|
116
149
|
```mermaid
|
|
@@ -132,6 +165,8 @@ flowchart LR
|
|
|
132
165
|
|
|
133
166
|
Tool definitions call one runtime; the runtime validates paths, limits, credentials, cancellation, and deadlines before dispatching to the pinned upstream snapshot or configured vision provider endpoint. Web presentation consumes the same structured results and Artifact descriptors, so it does not change Headless behavior. Health, connection testing, and version inspection stay in Settings rather than model tool schemas.
|
|
134
167
|
|
|
168
|
+
</details>
|
|
169
|
+
|
|
135
170
|
## Tools
|
|
136
171
|
|
|
137
172
|
| Tool | Execution | Structured result | Artifact delivery |
|
|
@@ -149,6 +184,9 @@ Tool definitions call one runtime; the runtime validates paths, limits, credenti
|
|
|
149
184
|
|
|
150
185
|
The plugin does not reimplement visual algorithms. Its DSH-owned layer validates paths and limits, resolves credentials, calls the pinned upstream scripts with argv vectors, parses their exact output contracts, classifies failures, describes files, and projects results to the model and Web client.
|
|
151
186
|
|
|
187
|
+
<details>
|
|
188
|
+
<summary><strong>Advanced model behavior</strong></summary>
|
|
189
|
+
|
|
152
190
|
## Progressive model exposure
|
|
153
191
|
|
|
154
192
|
Runtime readiness is profile-wide, but the ten visual execution schemas are Agent-scoped. Before an Agent loads `vision-tools`, the plugin contributes only the small `vision_toolkit_activate` bootstrap; the visual tools are absent from that Agent's request schema. A successful call to the standard `skill` tool with `name="vision-tools"` mounts all ten tools automatically for the next model step and hides the bootstrap. A direct `/vision-tools` invocation injects the Skill instructions; if the visual tools are still absent, those instructions require one `vision_toolkit_activate` call. Activation affects only that Agent, restores when the Session contains durable evidence matching the bundled Skill version, and lasts until the Agent or plugin is disposed.
|
|
@@ -157,18 +195,20 @@ Health checks, connection testing, and plugin/upstream version inspection are ad
|
|
|
157
195
|
|
|
158
196
|
## Image-input variants for text-only models
|
|
159
197
|
|
|
160
|
-
Text-only model routes get sibling model-selector entries named `<model> (Vision Toolkit)` under a matching provider group.
|
|
198
|
+
Text-only model routes get sibling model-selector entries named `<model> (Vision Toolkit)` under a matching provider group. DSH cannot pass a pasted attachment's local path through its native image block, so every bridge path materializes the image inside the session workspace and exposes its absolute path to the model. The model can then call `vision_glance` (or another visual tool) with that path. When the server-side image-input variant is active, the same model-visible block also contains the focus-hinted `[vision model description]` evidence aligned with `agent-vision-toolkit`; the path remains available for a second, more targeted visual call. The session log contains the durable path reference and the UI keeps the paste record.
|
|
161
199
|
|
|
162
|
-
A variant is registered automatically for every model the host positively declares text-only (for example the DeepSeek chat family).
|
|
200
|
+
A variant is registered automatically for every model the host positively declares text-only (for example the DeepSeek chat family). With the default `autoSwitch: true`, the browser switches to `<model> (Vision Toolkit)` and the server-side bridge rewrites each native image block into **both** the workspace path and the focus-hinted description; the path is not hidden from the model. Setting `autoSwitch: false` keeps the older path-only takeover instead. The host's verdict uses the exact model route the browser read from the live model catalog, with the selector label as fallback; unconfirmed or image-capable routes keep their native flow.
|
|
163
201
|
|
|
164
|
-
Description conversion needs the configured vision provider and its credential; when the runtime is not ready or a read fails, the wire block
|
|
202
|
+
Description conversion needs the configured vision provider and its credential when the opt-in image-input variant is used; when the runtime is not ready or a read fails, the wire block keeps the workspace path and adds the upstream-compatible `[vision unavailable: ...]` note instead of failing the turn. The bridge does not treat injected context files as the current user intent, and it uses the latest assistant paragraph when a tool-fetched image is being described. Disable variants with `imageInputVariants.enabled: false`, restrict the wrapped routes with `imageInputVariants.providers`, or opt into native attachment switching with `imageInputVariants.autoSwitch: true`.
|
|
203
|
+
|
|
204
|
+
</details>
|
|
165
205
|
|
|
166
206
|
## Requirements
|
|
167
207
|
|
|
168
208
|
- DeepSeek Harness with a Web or Headless profile and `pnpm` available to `dsh plugin`.
|
|
169
209
|
- Python 3.11 or newer. Managed mode creates an isolated environment, so users do not install the upstream CLI or Python packages manually.
|
|
170
210
|
- Network access on the first managed-runtime activation unless the exact packages in `runtime/requirements.lock` are already available in the configured package cache.
|
|
171
|
-
- The built-in free
|
|
211
|
+
- The built-in free Gemma 4 provider is ready for `vision_glance`, `vision_ground`, `vision_detect`, and non-split-only long-screenshot OCR. A DSH Credential is required only when a custom OpenAI-compatible or Anthropic endpoint is configured. Local tools remain usable without either provider.
|
|
172
212
|
- Chrome, Chromium, or Edge only for `vision_html_screenshot`; all other tools remain available when no supported browser is installed.
|
|
173
213
|
- PNG, JPEG, GIF, or WebP inputs inside the session workspace or an explicitly configured `allowedDirs` root.
|
|
174
214
|
|
|
@@ -209,7 +249,7 @@ dsh plugin --profile web remove @dsh-external/dsh-vision-toolkit
|
|
|
209
249
|
dsh plugin --profile web add @anionex/dsh-vision-toolkit
|
|
210
250
|
```
|
|
211
251
|
|
|
212
|
-
After restarting, Settings → Vision should report plugin version **0.1.
|
|
252
|
+
After restarting, Settings → Vision should report plugin version **0.1.12**. The built-in free provider is selected automatically; custom providers still use the configured DSH Credential.
|
|
213
253
|
|
|
214
254
|
For a registry installation, update the dependency through the profile package manager:
|
|
215
255
|
|
|
@@ -239,7 +279,7 @@ The bundle defaults to the managed runtime. A profile patch can override the pro
|
|
|
239
279
|
provider:
|
|
240
280
|
baseUrl: https://vision.anionex.me/v1
|
|
241
281
|
credential: ANIONEX_FREE_VISION
|
|
242
|
-
model:
|
|
282
|
+
model: gemma-4-26b-a4b-it
|
|
243
283
|
protocol: openai
|
|
244
284
|
anthropicThinking: omit
|
|
245
285
|
userAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
|
|
@@ -263,7 +303,7 @@ The bundle defaults to the managed runtime. A profile patch can override the pro
|
|
|
263
303
|
|---|---|---|
|
|
264
304
|
| `provider.baseUrl` | `https://vision.anionex.me/v1` | Built-in free OpenAI-compatible endpoint; custom providers may use another base URL, normalized without trailing slashes |
|
|
265
305
|
| `provider.credential` | `ANIONEX_FREE_VISION` | Read-only built-in reference for the free service; custom providers use a DSH Credential reference, never a secret value |
|
|
266
|
-
| `provider.model` | `
|
|
306
|
+
| `provider.model` | `gemma-4-26b-a4b-it` | Multimodal model name sent to remote tools |
|
|
267
307
|
| `provider.protocol` | `openai` | `openai` sends Chat Completions requests; `anthropic` sends native Messages requests |
|
|
268
308
|
| `provider.anthropicThinking` | `omit` | Anthropic thinking field. `omit` sends no thinking field and has the broadest compatibility. Use `disabled` or `adaptive` only when the selected model documents that mode; restore `omit` first if the provider returns HTTP 400. |
|
|
269
309
|
| `provider.userAgent` | browser-compatible default | User-Agent sent by vision requests and explicit connection tests; override it for provider or proxy compatibility |
|
|
@@ -278,7 +318,7 @@ The bundle defaults to the managed runtime. A profile patch can override the pro
|
|
|
278
318
|
| `allowedDirs` | `[]` | Additional realpath-resolved input roots; the session workspace is always allowed |
|
|
279
319
|
| `imageInputVariants.enabled` | `true` | Register image-input variant entries for text-only model routes in the model selector |
|
|
280
320
|
| `imageInputVariants.providers` | `[]` | Restrict wrapped upstream routes by provider id; empty wraps every eligible route |
|
|
281
|
-
| `imageInputVariants.autoSwitch` | `true` | Automatically switch a text-only session to its image-input variant
|
|
321
|
+
| `imageInputVariants.autoSwitch` | `true` | Automatically switch a text-only session to its image-input variant; the model receives both the workspace path and focused description. `false` keeps the path-only takeover |
|
|
282
322
|
|
|
283
323
|
### Credentials
|
|
284
324
|
|
|
@@ -292,18 +332,20 @@ The public service is shared and intended as a zero-configuration default, not a
|
|
|
292
332
|
|
|
293
333
|
| Limit | Current value |
|
|
294
334
|
|---|---:|
|
|
295
|
-
| Per client |
|
|
296
|
-
| Global service |
|
|
297
|
-
| Burst |
|
|
335
|
+
| Per client | 100 requests per UTC day |
|
|
336
|
+
| Global service | 400 requests per UTC day |
|
|
337
|
+
| Burst | 20 requests per 60 seconds |
|
|
298
338
|
| Image bytes | 4 MiB per image |
|
|
299
339
|
| Decoded pixels | 20,000,000 per image |
|
|
300
340
|
| Output | 512 tokens maximum |
|
|
301
341
|
|
|
302
|
-
### Managed and external
|
|
342
|
+
### Managed runtime and optional external runtime
|
|
303
343
|
|
|
304
344
|
Managed mode verifies `vendor/agent-vision-toolkit/UPSTREAM_MANIFEST.json`, prefers `uv`, falls back to `venv` plus pip, installs exact versions from `runtime/requirements.lock`, coordinates concurrent preparation with a heartbeat lock, and publishes a staged environment only after all probes pass.
|
|
305
345
|
|
|
306
|
-
|
|
346
|
+
Most users should stop at managed mode. It is included in the npm package and prepares the pinned Python environment for you.
|
|
347
|
+
|
|
348
|
+
The optional external mode is for plugin development or controlled deployments that already maintain the exact upstream checkout:
|
|
307
349
|
|
|
308
350
|
```yaml
|
|
309
351
|
- id: vision-toolkit
|
|
@@ -320,6 +362,8 @@ The path must be an exported copy matching the packaged manifest or the root of
|
|
|
320
362
|
|
|
321
363
|
The Web profile registers a Vision Toolkit Settings section for the provider URL, Credential reference, model, OpenAI/Anthropic protocol, Anthropic thinking mode, User-Agent, language, timeout, byte/pixel limits, concurrency, runtime mode, Python override, external source path, and allowed directories. It also shows plugin/upstream versions, the active runtime generation, non-secret Credential configured/source/writable facts, runtime paths, health results, and Artifact-route availability.
|
|
322
364
|
|
|
365
|
+
The **Plugin updates** card checks the profile's configured npm registry for a newer `@anionex/dsh-vision-toolkit` release. **Update and restart** installs that exact confirmed version into the current DSH profile, verifies the installed package, starts an independent restart helper, and gracefully restarts DSH Web; the open page waits for the replacement process and reloads after the new plugin version is serving. The action is same-origin, fixed to this package, serialized, and unavailable for `link:`, `file:`, workspace, git, URL, transitive, ambiguous, read-only, or missing-`pnpm` installations so local development sources are never overwritten. A restart can interrupt work that is currently running, so the UI requires an explicit confirmation.
|
|
366
|
+
|
|
323
367
|
`Save and apply` validates the complete value, prepares the candidate Python/upstream runtime, commits the Settings revision, and only then atomically switches generations. A rejected candidate leaves the previous generation serving and is reported separately from a genuinely unavailable runtime. `Reload` always restores the authoritative saved value, even when its revision did not change, so a rejected browser draft is discarded. If initial startup cannot prepare a runtime, the Settings route remains available so a valid configuration can make the first generation operational. A stale browser revision receives a conflict instead of overwriting a newer save; reload before retrying. A read-only Settings provider allows inspection and health checks but disables saves.
|
|
324
368
|
|
|
325
369
|
`Run health check` performs local checks only. `Test API connection` is an explicit action that sends the configured Credential to `GET /models`; OpenAI uses Bearer authentication, while Anthropic uses `x-api-key` and `anthropic-version`. That lightweight probe uploads no image and creates no completion. `Test vision model` separately sends the bundled `assets/vision-model-test.png` through the same multimodal runtime path as `vision_glance`; it creates one real completion and is the authoritative check that the selected endpoint, credential, model, protocol, and upstream account can process images. The Vision model health card displays a dedicated `Verified`, `Not tested`, or `Test failed` tag, so an HTTP 200 response from `/models` is not presented as a successful image test. Plugin load and ordinary Settings reads never make either request.
|
|
@@ -398,7 +442,14 @@ Update the upstream snapshot only through `pnpm run upstream:sync -- <checkout>`
|
|
|
398
442
|
|
|
399
443
|
## Project status and scope
|
|
400
444
|
|
|
401
|
-
Version `0.1.
|
|
445
|
+
Version `0.1.12` is the current public npm release. The product focuses on screenshot understanding, visual grounding, OCR, asset extraction, UI restoration, and pixel-level verification in DSH Web and Headless profiles. Web upload, drag-and-drop, camera/video/audio/document ingestion, interactive box editing, automatic GUI clicking, service clusters, model routing, model voting, and cross-session vision caches remain outside the current product.
|
|
446
|
+
|
|
447
|
+
<details>
|
|
448
|
+
<summary><strong>Maintainer scope note</strong></summary>
|
|
449
|
+
|
|
450
|
+
The stable `ctx.visionToolkit` service and capability-discovery API remain unpublished until an independent plugin becomes a real consumer. This keeps the public integration surface tied to a tested use case rather than an unvalidated ecosystem contract.
|
|
451
|
+
|
|
452
|
+
</details>
|
|
402
453
|
|
|
403
454
|
## Community and About
|
|
404
455
|
|