@anionex/dsh-vision-toolkit 0.1.10 → 0.1.12

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.i18n.yaml CHANGED
@@ -2,5 +2,5 @@
2
2
  # side as of the last confirmed-consistent state. Both languages carry equal authority;
3
3
  # after editing either side, bring the other along and re-record with:
4
4
  # pnpm run verify-translation-pairing --write dsh-vision-dark-theme/README.md
5
- README.md: 2a05d467a5c0cff82900d6074220392553f5d972
6
- README.zh.md: 07cff26ce48a7f0df6791ac9723debaeb723d5f4
5
+ README.md: 0c81049d6e43d5c09056e7733349b45ea29fc22e
6
+ README.zh.md: 223b1d7f98b26eac72ab1ee0a76a83aed8b81673
package/README.md CHANGED
@@ -1,38 +1,66 @@
1
- ![DSH Vision Toolkit — native visual engineering for text-only DeepSeek Harness agents](assets/hero.png)
1
+ <p align="center">
2
+ <img src="assets/hero.png" alt="DSH Vision Toolkit — native visual engineering for text-only DeepSeek Harness agents" />
3
+ </p>
4
+
5
+ <h1 align="center">DSH Vision Toolkit</h1>
6
+
7
+ <p align="center">
8
+ English | <a href="https://github.com/Anionex/dsh-vision-toolkit/blob/main/README.zh.md">中文</a>
9
+ </p>
10
+
11
+ <p align="center">
12
+ <a href="https://dshfind.com/en/plugins/Anionex/dsh-vision-toolkit"><img src="https://img.shields.io/badge/recommended%20by-dshfind-FFD700?style=flat-square" alt="Recommended by dshfind" /></a>
13
+ <a href="https://dshfind.com/en/plugins/Anionex/dsh-vision-toolkit"><img src="https://img.shields.io/badge/dshfind%20score-94%20%7C%20highest--rated%20plugin-5B4CF0?style=flat-square" alt="dshfind score: 94 — highest-rated plugin" /></a>
14
+ <a href="https://x.com/anion_ex"><img src="https://img.shields.io/badge/-@anion__ex-000000?style=flat-square&amp;logo=x&amp;logoColor=white" alt="X: @anion_ex" /></a>
15
+ <a href="https://github.com/Anionex/dsh-vision-toolkit/releases/tag/v0.1.12"><img src="https://img.shields.io/badge/release-v0.1.12-5B4CF0?style=flat-square" alt="Release v0.1.12" /></a>
16
+ <a href="tests"><img src="https://img.shields.io/badge/verified-233%20tests-2EA44F?style=flat-square" alt="Verified: 233 tests" /></a>
17
+ </p>
18
+
19
+ <p align="center">
20
+ <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-0B7285?style=flat-square" alt="License: MIT" /></a>
21
+ <a href="package.json"><img src="https://img.shields.io/badge/Node.js-%5E22.19%20%7C%20%3E%3D24-339933?style=flat-square&amp;logo=nodedotjs&amp;logoColor=white" alt="Node.js ^22.19 or >=24" /></a>
22
+ <a href="runtime/requirements.lock"><img src="https://img.shields.io/badge/Python-3.11%2B-3776AB?style=flat-square&amp;logo=python&amp;logoColor=white" alt="Python 3.11+" /></a>
23
+ <a href="cordis.patch.yml"><img src="https://img.shields.io/badge/DSH-Web%20%2B%20Headless-5B4CF0?style=flat-square" alt="DSH Web and Headless profiles" /></a>
24
+ </p>
2
25
 
3
- # DSH Vision Toolkit
26
+ ## Give your DSH agent eyes
4
27
 
5
- [![Recommended by dshfind](https://img.shields.io/badge/recommended%20by-dshfind-FFD700?style=flat-square)](https://dshfind.com/en/plugins/Anionex/dsh-vision-toolkit)
6
- [![dshfind score: 94 — highest-rated plugin](https://img.shields.io/badge/dshfind%20score-94%20%7C%20highest--rated%20plugin-5B4CF0?style=flat-square)](https://dshfind.com/en/plugins/Anionex/dsh-vision-toolkit)
7
- [![X (Twitter)](https://img.shields.io/badge/-@anion__ex-000000?style=flat-square&logo=x&logoColor=white)](https://x.com/anion_ex)
8
- [![Release v0.1.10](https://img.shields.io/badge/release-v0.1.10-5B4CF0?style=flat-square)](https://github.com/Anionex/dsh-vision-toolkit/releases/tag/v0.1.10)
9
- [![Verified: 233 tests](https://img.shields.io/badge/verified-233%20tests-2EA44F?style=flat-square)](tests)
10
- [![License: MIT](https://img.shields.io/badge/license-MIT-0B7285?style=flat-square)](LICENSE)
11
- [![Node.js](https://img.shields.io/badge/Node.js-%5E22.19%20%7C%20%3E%3D24-339933?style=flat-square&logo=nodedotjs&logoColor=white)](package.json)
12
- [![Python](https://img.shields.io/badge/Python-3.11%2B-3776AB?style=flat-square&logo=python&logoColor=white)](runtime/requirements.lock)
13
- [![DSH profiles](https://img.shields.io/badge/DSH-Web%20%2B%20Headless-5B4CF0?style=flat-square)](cordis.patch.yml)
28
+ Drop in a screenshot and let a text-only DeepSeek Harness agent inspect it, read it, locate elements, extract assets, rebuild interfaces, and measure whether the result matches.
14
29
 
15
- **Install:** `dsh plugin --profile web add @anionex/dsh-vision-toolkit`
30
+ DSH Vision Toolkit packages [`agent-vision-toolkit`](https://github.com/Anionex/agent-vision-toolkit) as a native DSH plugin. You get focused image Q&A, OCR, original-pixel coordinates, UI restoration, pixel comparison, downloadable results, and a Web Settings panel without assembling scripts by hand.
16
31
 
17
- **DSH Vision Toolkit brings [`agent-vision-toolkit`](https://github.com/Anionex/agent-vision-toolkit) into DeepSeek Harness as a native Profile Bundle.**
32
+ ```sh
33
+ dsh plugin --profile web add @anionex/dsh-vision-toolkit
34
+ ```
18
35
 
19
- Give text-only DSH agents eyes—and keep vision in the harness—with intent-aware image Q&A, OCR, original-pixel grounding, UI restoration, pixel verification, managed Artifacts, and Web Settings. Ten independent tools replace shell glue with structured schemas and Agent-scoped progressive exposure.
36
+ The npm package includes the visual toolkit snapshot and uses a managed runtime by default. **Normal installation does not require a source checkout or an `agentVisionToolkitPath`.**
20
37
 
21
38
  **Upstream toolkit:** [Anionex/agent-vision-toolkit](https://github.com/Anionex/agent-vision-toolkit) · **Project website:** [agent-vision.anionex.me](https://agent-vision.anionex.me)
22
39
 
23
- English | [中文](README.zh.md)
40
+ ## What you can do
41
+
42
+ | Goal | What the agent can deliver |
43
+ |---|---|
44
+ | Understand a screenshot | Focused answers, visual descriptions, multi-image comparison, and OCR |
45
+ | Find an interface element | Original-image pixel coordinates with an optional labeled preview |
46
+ | Rebuild a page from a reference | Screenshot rendering, region-by-region diagnosis, and measurable iteration |
47
+ | Extract a usable asset | Cropped images, transparent foregrounds, dominant colors, or editable SVG |
48
+ | Read a long screenshot | Auditable chunks, Markdown output, manifests, and resumable OCR runs |
49
+ | Verify a visual result | A difference percentage, ranked mismatch regions, heatmap, and JSON report |
24
50
 
25
- ## Why this exists
51
+ You can use remote vision only where it adds value. Cropping, tracing, pixel comparison, color analysis, foreground extraction, and HTML screenshots run locally.
26
52
 
27
- `agent-vision-toolkit` treats vision as an Agent-callable capability rather than a property of the base model. Its method carries the reason for looking into the visual request, moves from the whole image to targeted regions, and verifies coordinates, colors, geometry, and differences with focused tools instead of accepting a generic description as evidence.
53
+ ## See it in action
28
54
 
29
- DSH Vision Toolkit preserves that method while replacing CLI installation and Bash argument construction with native schemas, DSH Credentials, lifecycle-managed runtime preparation, structured Session-log results, previewable Artifacts, dedicated Web cards, and Settings. The Agent loads one versioned Skill and receives the ten visual schemas only when the current task needs them.
55
+ The first example is a live DSH Web view. The next two examples come from the same `agent-vision-toolkit` lineage packaged with this plugin, and the last shows the workflow inside a live DeepSeek Harness Web session. See the [asset provenance record](assets/upstream/README.md) for source details.
30
56
 
31
- The package delivers the committed P0 and P1 product scope. P2's stable `ctx.visionToolkit` service remains deliberately unpublished until an independent plugin becomes a real consumer; the internal runtime does not pretend that an unvalidated ecosystem API is stable.
57
+ ### DSH view example
32
58
 
33
- ## Proven use cases from agent-vision-toolkit
59
+ <p align="center">
60
+ <img src="assets/dsh-view-example.png" width="80%" alt="DSH Web session view in which a text-only DeepSeek-V4-Flash (Vision Toolkit) model answers a question about a pasted banner image." />
61
+ </p>
34
62
 
35
- The first two panels are official upstream reference runs from the same pinned `agent-vision-toolkit` lineage packaged by this bundle. The image Q&A and screenshot-guided debugging panel is a live DeepSeek Harness Web session, showing the same workflows through DSH. See the [asset provenance record](assets/upstream/README.md) for the upstream source images.
63
+ *A live DSH Web view: the user pastes a brand-banner screenshot, and the text-only model answers what the image contains through the `DeepSeek-V4-Flash (Vision Toolkit)` image-input variant.*
36
64
 
37
65
  ### Infographic restoration: screenshot to editable HTML/CSS
38
66
 
@@ -61,47 +89,47 @@ The first two panels are official upstream reference runs from the same pinned `
61
89
 
62
90
  *Left: intent-aware image Q&A in DSH Web. Right: a DSH Web screenshot-debugging turn that lists the concrete UI differences and continues toward `vision_pixel_diff`. The upstream workflow source is the same [`agent-vision-toolkit` reference](https://github.com/Anionex/agent-vision-toolkit/blob/c27d1a300962b553c0884993c575cd3e819465ce/README.md#real-world-effects).*
63
91
 
64
- DSH Vision Toolkit adds native tool schemas, versioned lifecycle, Credentials, structured Session results, Artifacts, Web presentation, Settings, and progressive exposure around these upstream capabilities. The next section is the reproducible proof executed and checked into this DSH repository.
92
+ DSH Vision Toolkit brings this workflow into DSH, where the result can become a file, a coordinate, a measured comparison, or the next step in the same session.
65
93
 
66
- ## DSH-native proof: reference-to-pixel verification
94
+ ## From a rough match to pixel-perfect
67
95
 
68
- The checked-in UI-restoration workflow renders an intentionally inaccurate HTML implementation, measures a `6.04%` pixel difference across six non-zero regions, iterates, and reaches an exact `0%` difference against the reference at `1200 × 720`.
96
+ The included UI-restoration workflow starts with an intentionally inaccurate HTML implementation. Vision Toolkit measures a `6.04%` difference, points to the worst regions, and helps drive the next iteration. The final render reaches an exact `0%` difference at `1200 × 720`.
69
97
 
70
98
  <p>
71
99
  <img src="examples/ui-restoration/assets/initial.png" width="49%" alt="Initial UI restoration candidate before Vision Toolkit iteration, with measurable layout and styling differences from the reference." />
72
100
  <img src="examples/ui-restoration/assets/implementation.png" width="49%" alt="Final UI restoration output reproduced by the checked-in workflow with zero pixel difference from the reference." />
73
101
  </p>
74
102
 
75
- | Verified surface | Evidence |
103
+ | Start | Result |
76
104
  |---|---|
77
- | Product scope | 10 independent visual tools, matching `vision-tools` Skill, Artifacts, dedicated Web cards, and live Settings |
78
- | Automated coverage | 17 Vitest files / 136 passing tests, plus a dependency-free portable package check |
79
- | Real profiles | Clean temporary Web and Headless installation, activation, disable, re-enable, and uninstall |
80
- | Visual acceptance | Reproducible HTML screenshot → pixel diff example with a final `0%` difference |
105
+ | Reference image | A working HTML implementation you can open and edit |
106
+ | First comparison | `6.04%` difference across the visible problem regions |
107
+ | Final comparison | `0%` difference at `1200 × 720` |
81
108
 
82
- ## Highlights
109
+ ## Why it feels different
83
110
 
84
- - **See images without bloating every prompt:** only `vision_toolkit_activate` is initially visible; loading `vision-tools` mounts ten independent schemas for that Agent and keeps version/health administration out of model context.
85
- - **Act on coordinates instead of parsing prose:** grounding and detection return original-image pixel boxes, while every model-visible result remains structured text or JSON.
86
- - **Deliver files, not temporary output:** crop, trace, OCR, pixel diff, foreground extraction, and HTML rendering produce described Artifacts that the Web client can preview, download, or open locally.
87
- - **Keep runtime and credentials controlled:** DSH Credentials hold API keys, managed mode prepares an exact isolated Python environment, and a failed Settings candidate cannot replace the serving generation.
88
- - **Close the visual verification loop:** local HTML rendering and pixel-diff ranking support reference → implementation → screenshot → measured iteration without a model-native image channel.
89
- - **Use the same bundle in Web and Headless profiles:** Web adds cards, previews, Settings, and health actions; Headless receives the same tool semantics and complete structured results.
111
+ - **Ask for the thing you need.** “Where is the submit button?” and “Why does this screenshot differ from the reference?” lead to focused visual work instead of a generic caption.
112
+ - **Get evidence you can use.** The agent returns coordinates, OCR, measurements, JSON, and files you can open or pass to the next step.
113
+ - **Keep the workflow in DSH.** Credentials, Settings, Artifacts, Web cards, and Headless results live alongside the rest of your session.
114
+ - **Use local tools when you can.** Crop, trace, pixel comparison, color analysis, foreground extraction, and HTML screenshots do not consume a vision API request.
115
+ - **Repeat the loop.** Reference image → implementation → screenshot → pixel diff gives UI work a measurable finish line.
90
116
 
91
- ## Quick start
117
+ ## Start in three steps
92
118
 
93
- Prerequisites: DeepSeek Harness `0.1.0-rc.6` or a compatible later `0.1.x` release, Python 3.11+, and `pnpm` available to `dsh plugin`. Install the published bundle from npm, add it to the profiles you use, and confirm the bundle row:
119
+ Use DeepSeek Harness `0.1.0-rc.6` or a compatible later `0.1.x` release. The plugin prepares its managed runtime on first use.
94
120
 
95
121
  ```sh
96
122
  dsh plugin --profile web add @anionex/dsh-vision-toolkit
97
123
  dsh plugin --profile headless add @anionex/dsh-vision-toolkit
98
- dsh --profile web --dump-config | grep vision-toolkit
99
- dsh --profile headless --dump-config | grep vision-toolkit
100
124
  ```
101
125
 
102
- Legacy profiles must use `nodeLinker: hoisted` and `autoInstallPeers: false` in their `pnpm-workspace.yaml`. An updated DSH launcher repairs these owned settings before `dsh plugin` runs; when using an older launcher, set them before installation so pnpm does not assemble a second Harness dependency graph inside the profile.
126
+ 1. Restart your Web profile and open **Settings → Vision Toolkit**.
127
+ 2. New installations use the built-in free Gemma 4 provider, so you can run **Test API connection** and **Test vision model** without an API key. To use another provider, edit the endpoint/model/protocol and provide its DSH Credential.
128
+ 3. In a conversation, paste an image or put it in the workspace, invoke `/vision-tools`, and ask for a concrete visual task.
129
+
130
+ If you use an older DSH launcher, the profile may need `nodeLinker: hoisted` and `autoInstallPeers: false` before installation. Current launchers repair these settings for you.
103
131
 
104
- Restart a running Web profile, open **Settings → Vision Toolkit**, and run **Test API connection** followed by **Test vision model**. New installations use the built-in free Moondream provider automatically, so no API key or DSH Credential is required. To use another provider, edit the endpoint/model/protocol and provide its DSH Credential. In a conversation, make an image available as a workspace path, invoke `/vision-tools`, and ask the Agent to call a specific `vision_*` tool. Local crop, trace, pixel, color, foreground, and HTML operations do not require a visual API credential.
132
+ Local crop, trace, pixel, color, foreground, and HTML operations do not require a visual API credential.
105
133
 
106
134
  ## Community Group
107
135
 
@@ -111,6 +139,11 @@ Join the `agent-vision-toolkit` community group to exchange usage tips, share fe
111
139
  <img src="assets/community-group-qr.png" alt="QR code for the agent-vision-toolkit community group" width="260">
112
140
  </p>
113
141
 
142
+ > **No local path is required.** Keep the default `runtime.mode: managed` for the normal npm installation. The optional `runtime.agentVisionToolkitPath` setting is only for developers or controlled deployments that deliberately use an external pinned checkout.
143
+
144
+ <details>
145
+ <summary><strong>Technical architecture</strong></summary>
146
+
114
147
  ## How it works
115
148
 
116
149
  ```mermaid
@@ -132,6 +165,8 @@ flowchart LR
132
165
 
133
166
  Tool definitions call one runtime; the runtime validates paths, limits, credentials, cancellation, and deadlines before dispatching to the pinned upstream snapshot or configured vision provider endpoint. Web presentation consumes the same structured results and Artifact descriptors, so it does not change Headless behavior. Health, connection testing, and version inspection stay in Settings rather than model tool schemas.
134
167
 
168
+ </details>
169
+
135
170
  ## Tools
136
171
 
137
172
  | Tool | Execution | Structured result | Artifact delivery |
@@ -149,6 +184,9 @@ Tool definitions call one runtime; the runtime validates paths, limits, credenti
149
184
 
150
185
  The plugin does not reimplement visual algorithms. Its DSH-owned layer validates paths and limits, resolves credentials, calls the pinned upstream scripts with argv vectors, parses their exact output contracts, classifies failures, describes files, and projects results to the model and Web client.
151
186
 
187
+ <details>
188
+ <summary><strong>Advanced model behavior</strong></summary>
189
+
152
190
  ## Progressive model exposure
153
191
 
154
192
  Runtime readiness is profile-wide, but the ten visual execution schemas are Agent-scoped. Before an Agent loads `vision-tools`, the plugin contributes only the small `vision_toolkit_activate` bootstrap; the visual tools are absent from that Agent's request schema. A successful call to the standard `skill` tool with `name="vision-tools"` mounts all ten tools automatically for the next model step and hides the bootstrap. A direct `/vision-tools` invocation injects the Skill instructions; if the visual tools are still absent, those instructions require one `vision_toolkit_activate` call. Activation affects only that Agent, restores when the Session contains durable evidence matching the bundled Skill version, and lasts until the Agent or plugin is disposed.
@@ -157,18 +195,20 @@ Health checks, connection testing, and plugin/upstream version inspection are ad
157
195
 
158
196
  ## Image-input variants for text-only models
159
197
 
160
- Text-only model routes get sibling model-selector entries named `<model> (Vision Toolkit)` under a matching provider group. A variant declares image input, so pasted images keep the native attachment flow — composer thumbnail, durable session image, and history rendering — and the plugin rewrites every image block into a Vision Toolkit description only on the wire to the model, before the request reaches the upstream route. The vision prompt is focus-hinted with the latest user or assistant intent, using the same role, instruction, image-text policy, and `[vision model description]` channel markers as `agent-vision-toolkit`; the model receives task-relevant evidence instead of a broad generic description. The session log is untouched; replay and the UI keep the real image.
198
+ Text-only model routes get sibling model-selector entries named `<model> (Vision Toolkit)` under a matching provider group. DSH cannot pass a pasted attachment's local path through its native image block, so every bridge path materializes the image inside the session workspace and exposes its absolute path to the model. The model can then call `vision_glance` (or another visual tool) with that path. When the server-side image-input variant is active, the same model-visible block also contains the focus-hinted `[vision model description]` evidence aligned with `agent-vision-toolkit`; the path remains available for a second, more targeted visual call. The session log contains the durable path reference and the UI keeps the paste record.
161
199
 
162
- A variant is registered automatically for every model the host positively declares text-only (for example the DeepSeek chat family). Paste handling is automatic: when the current model is confirmed text-only and its variant exists, the browser integration switches the session to the variant by itself (a short notice names the new model) and the paste then keeps the native flow; no manual model change is needed. The host's verdict uses the exact model route the browser read from the live model catalog, with the selector label as fallback; unconfirmed or image-capable routes always keep the native flow, and a text-only model without a variant (for example when variants are disabled) keeps the paste-to-path takeover, which copies the image into the session workspace and inserts its path as text.
200
+ A variant is registered automatically for every model the host positively declares text-only (for example the DeepSeek chat family). With the default `autoSwitch: true`, the browser switches to `<model> (Vision Toolkit)` and the server-side bridge rewrites each native image block into **both** the workspace path and the focus-hinted description; the path is not hidden from the model. Setting `autoSwitch: false` keeps the older path-only takeover instead. The host's verdict uses the exact model route the browser read from the live model catalog, with the selector label as fallback; unconfirmed or image-capable routes keep their native flow.
163
201
 
164
- Description conversion needs the configured vision provider and its credential; when the runtime is not ready or a read fails, the wire block degrades to the upstream-compatible `[vision unavailable: ...]` note instead of failing the turn. The bridge does not treat injected context files as the current user intent, and it uses the latest assistant paragraph when a tool-fetched image is being described. Disable variants with `imageInputVariants.enabled: false`, restrict the wrapped routes with `imageInputVariants.providers`, or keep the paste-to-path behavior for text-only models with `imageInputVariants.autoSwitch: false`.
202
+ Description conversion needs the configured vision provider and its credential when the opt-in image-input variant is used; when the runtime is not ready or a read fails, the wire block keeps the workspace path and adds the upstream-compatible `[vision unavailable: ...]` note instead of failing the turn. The bridge does not treat injected context files as the current user intent, and it uses the latest assistant paragraph when a tool-fetched image is being described. Disable variants with `imageInputVariants.enabled: false`, restrict the wrapped routes with `imageInputVariants.providers`, or opt into native attachment switching with `imageInputVariants.autoSwitch: true`.
203
+
204
+ </details>
165
205
 
166
206
  ## Requirements
167
207
 
168
208
  - DeepSeek Harness with a Web or Headless profile and `pnpm` available to `dsh plugin`.
169
209
  - Python 3.11 or newer. Managed mode creates an isolated environment, so users do not install the upstream CLI or Python packages manually.
170
210
  - Network access on the first managed-runtime activation unless the exact packages in `runtime/requirements.lock` are already available in the configured package cache.
171
- - The built-in free Moondream provider is ready for `vision_glance`, `vision_ground`, `vision_detect`, and non-split-only long-screenshot OCR. A DSH Credential is required only when a custom OpenAI-compatible or Anthropic endpoint is configured. Local tools remain usable without either provider.
211
+ - The built-in free Gemma 4 provider is ready for `vision_glance`, `vision_ground`, `vision_detect`, and non-split-only long-screenshot OCR. A DSH Credential is required only when a custom OpenAI-compatible or Anthropic endpoint is configured. Local tools remain usable without either provider.
172
212
  - Chrome, Chromium, or Edge only for `vision_html_screenshot`; all other tools remain available when no supported browser is installed.
173
213
  - PNG, JPEG, GIF, or WebP inputs inside the session workspace or an explicitly configured `allowedDirs` root.
174
214
 
@@ -209,7 +249,7 @@ dsh plugin --profile web remove @dsh-external/dsh-vision-toolkit
209
249
  dsh plugin --profile web add @anionex/dsh-vision-toolkit
210
250
  ```
211
251
 
212
- After restarting, Settings → Vision should report plugin version **0.1.10**. The built-in free provider is selected automatically; custom providers still use the configured DSH Credential.
252
+ After restarting, Settings → Vision should report plugin version **0.1.12**. The built-in free provider is selected automatically; custom providers still use the configured DSH Credential.
213
253
 
214
254
  For a registry installation, update the dependency through the profile package manager:
215
255
 
@@ -239,7 +279,7 @@ The bundle defaults to the managed runtime. A profile patch can override the pro
239
279
  provider:
240
280
  baseUrl: https://vision.anionex.me/v1
241
281
  credential: ANIONEX_FREE_VISION
242
- model: moondream-3.1
282
+ model: gemma-4-26b-a4b-it
243
283
  protocol: openai
244
284
  anthropicThinking: omit
245
285
  userAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
@@ -263,7 +303,7 @@ The bundle defaults to the managed runtime. A profile patch can override the pro
263
303
  |---|---|---|
264
304
  | `provider.baseUrl` | `https://vision.anionex.me/v1` | Built-in free OpenAI-compatible endpoint; custom providers may use another base URL, normalized without trailing slashes |
265
305
  | `provider.credential` | `ANIONEX_FREE_VISION` | Read-only built-in reference for the free service; custom providers use a DSH Credential reference, never a secret value |
266
- | `provider.model` | `moondream-3.1` | Multimodal model name sent to remote tools |
306
+ | `provider.model` | `gemma-4-26b-a4b-it` | Multimodal model name sent to remote tools |
267
307
  | `provider.protocol` | `openai` | `openai` sends Chat Completions requests; `anthropic` sends native Messages requests |
268
308
  | `provider.anthropicThinking` | `omit` | Anthropic thinking field. `omit` sends no thinking field and has the broadest compatibility. Use `disabled` or `adaptive` only when the selected model documents that mode; restore `omit` first if the provider returns HTTP 400. |
269
309
  | `provider.userAgent` | browser-compatible default | User-Agent sent by vision requests and explicit connection tests; override it for provider or proxy compatibility |
@@ -278,7 +318,7 @@ The bundle defaults to the managed runtime. A profile patch can override the pro
278
318
  | `allowedDirs` | `[]` | Additional realpath-resolved input roots; the session workspace is always allowed |
279
319
  | `imageInputVariants.enabled` | `true` | Register image-input variant entries for text-only model routes in the model selector |
280
320
  | `imageInputVariants.providers` | `[]` | Restrict wrapped upstream routes by provider id; empty wraps every eligible route |
281
- | `imageInputVariants.autoSwitch` | `true` | Automatically switch a text-only session to its image-input variant on paste, so the image keeps the native flow; `false` keeps the paste-to-path takeover for text-only models |
321
+ | `imageInputVariants.autoSwitch` | `true` | Automatically switch a text-only session to its image-input variant; the model receives both the workspace path and focused description. `false` keeps the path-only takeover |
282
322
 
283
323
  ### Credentials
284
324
 
@@ -292,18 +332,20 @@ The public service is shared and intended as a zero-configuration default, not a
292
332
 
293
333
  | Limit | Current value |
294
334
  |---|---:|
295
- | Per client | 30 requests per UTC day |
296
- | Global service | 120 requests per UTC day |
297
- | Burst | 6 requests per 60 seconds |
335
+ | Per client | 100 requests per UTC day |
336
+ | Global service | 400 requests per UTC day |
337
+ | Burst | 20 requests per 60 seconds |
298
338
  | Image bytes | 4 MiB per image |
299
339
  | Decoded pixels | 20,000,000 per image |
300
340
  | Output | 512 tokens maximum |
301
341
 
302
- ### Managed and external runtimes
342
+ ### Managed runtime and optional external runtime
303
343
 
304
344
  Managed mode verifies `vendor/agent-vision-toolkit/UPSTREAM_MANIFEST.json`, prefers `uv`, falls back to `venv` plus pip, installs exact versions from `runtime/requirements.lock`, coordinates concurrent preparation with a heartbeat lock, and publishes a staged environment only after all probes pass.
305
345
 
306
- External mode is intended for development or controlled deployments:
346
+ Most users should stop at managed mode. It is included in the npm package and prepares the pinned Python environment for you.
347
+
348
+ The optional external mode is for plugin development or controlled deployments that already maintain the exact upstream checkout:
307
349
 
308
350
  ```yaml
309
351
  - id: vision-toolkit
@@ -320,6 +362,8 @@ The path must be an exported copy matching the packaged manifest or the root of
320
362
 
321
363
  The Web profile registers a Vision Toolkit Settings section for the provider URL, Credential reference, model, OpenAI/Anthropic protocol, Anthropic thinking mode, User-Agent, language, timeout, byte/pixel limits, concurrency, runtime mode, Python override, external source path, and allowed directories. It also shows plugin/upstream versions, the active runtime generation, non-secret Credential configured/source/writable facts, runtime paths, health results, and Artifact-route availability.
322
364
 
365
+ The **Plugin updates** card checks the profile's configured npm registry for a newer `@anionex/dsh-vision-toolkit` release. **Update and restart** installs that exact confirmed version into the current DSH profile, verifies the installed package, starts an independent restart helper, and gracefully restarts DSH Web; the open page waits for the replacement process and reloads after the new plugin version is serving. The action is same-origin, fixed to this package, serialized, and unavailable for `link:`, `file:`, workspace, git, URL, transitive, ambiguous, read-only, or missing-`pnpm` installations so local development sources are never overwritten. A restart can interrupt work that is currently running, so the UI requires an explicit confirmation.
366
+
323
367
  `Save and apply` validates the complete value, prepares the candidate Python/upstream runtime, commits the Settings revision, and only then atomically switches generations. A rejected candidate leaves the previous generation serving and is reported separately from a genuinely unavailable runtime. `Reload` always restores the authoritative saved value, even when its revision did not change, so a rejected browser draft is discarded. If initial startup cannot prepare a runtime, the Settings route remains available so a valid configuration can make the first generation operational. A stale browser revision receives a conflict instead of overwriting a newer save; reload before retrying. A read-only Settings provider allows inspection and health checks but disables saves.
324
368
 
325
369
  `Run health check` performs local checks only. `Test API connection` is an explicit action that sends the configured Credential to `GET /models`; OpenAI uses Bearer authentication, while Anthropic uses `x-api-key` and `anthropic-version`. That lightweight probe uploads no image and creates no completion. `Test vision model` separately sends the bundled `assets/vision-model-test.png` through the same multimodal runtime path as `vision_glance`; it creates one real completion and is the authoritative check that the selected endpoint, credential, model, protocol, and upstream account can process images. The Vision model health card displays a dedicated `Verified`, `Not tested`, or `Test failed` tag, so an HTTP 200 response from `/models` is not presented as a successful image test. Plugin load and ordinary Settings reads never make either request.
@@ -398,7 +442,14 @@ Update the upstream snapshot only through `pnpm run upstream:sync -- <checkout>`
398
442
 
399
443
  ## Project status and scope
400
444
 
401
- Version `0.1.10` is the current public npm release. P0 and P1 are product commitments in this package. P2 is a design threshold: no stable `ctx.visionToolkit` service, capability-discovery API, or provider ecosystem is published until at least one independent plugin consumes the internal capability shape. Web upload, drag-and-drop, camera/video/audio/document ingestion, interactive box editing, automatic GUI clicking, service clusters, model routing, model voting, and cross-session vision caches remain outside the current product.
445
+ Version `0.1.12` is the current public npm release. The product focuses on screenshot understanding, visual grounding, OCR, asset extraction, UI restoration, and pixel-level verification in DSH Web and Headless profiles. Web upload, drag-and-drop, camera/video/audio/document ingestion, interactive box editing, automatic GUI clicking, service clusters, model routing, model voting, and cross-session vision caches remain outside the current product.
446
+
447
+ <details>
448
+ <summary><strong>Maintainer scope note</strong></summary>
449
+
450
+ The stable `ctx.visionToolkit` service and capability-discovery API remain unpublished until an independent plugin becomes a real consumer. This keeps the public integration surface tied to a tested use case rather than an unvalidated ecosystem contract.
451
+
452
+ </details>
402
453
 
403
454
  ## Community and About
404
455