@anionex/dsh-vision-toolkit 0.1.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.i18n.yaml +6 -0
- package/README.md +383 -0
- package/README.zh.md +383 -0
- package/assets/dsh-conversation-artifact.png +0 -0
- package/assets/dsh-conversation-image-qa-top.png +0 -0
- package/assets/dsh-conversation-image-qa.png +0 -0
- package/assets/dsh-conversation-pixel-diff.png +0 -0
- package/assets/dsh-conversation-screenshot-debugging-top.png +0 -0
- package/assets/dsh-conversation-screenshot-debugging.png +0 -0
- package/assets/dsh-conversation-tool-call.png +0 -0
- package/assets/dsh-conversation-vision-trace.png +0 -0
- package/assets/hero.png +0 -0
- package/assets/social-preview.png +0 -0
- package/assets/upstream/README.md +16 -0
- package/assets/upstream/image-qa.webp +0 -0
- package/assets/upstream/infographic-reference.webp +0 -0
- package/assets/upstream/infographic-result.webp +0 -0
- package/assets/upstream/screenshot-debugging.webp +0 -0
- package/assets/upstream/ui-result.webp +0 -0
- package/assets/upstream/ui-sketch.webp +0 -0
- package/assets/vision-settings.png +0 -0
- package/cordis.patch.yml +6 -0
- package/docs/assets/vision-settings.png +0 -0
- package/docs/requirements-traceability/README.i18n.yaml +6 -0
- package/docs/requirements-traceability/README.md +75 -0
- package/docs/requirements-traceability/README.zh.md +75 -0
- package/examples/ui-restoration/README.i18n.yaml +6 -0
- package/examples/ui-restoration/README.md +70 -0
- package/examples/ui-restoration/README.zh.md +70 -0
- package/examples/ui-restoration/assets/final-heatmap.png +0 -0
- package/examples/ui-restoration/assets/final-report.json +83 -0
- package/examples/ui-restoration/assets/implementation.png +0 -0
- package/examples/ui-restoration/assets/initial-heatmap.png +0 -0
- package/examples/ui-restoration/assets/initial-report.json +83 -0
- package/examples/ui-restoration/assets/initial.png +0 -0
- package/examples/ui-restoration/assets/metrics.json +12 -0
- package/examples/ui-restoration/assets/reference.png +0 -0
- package/examples/ui-restoration/implementation.html +94 -0
- package/examples/ui-restoration/initial.html +57 -0
- package/lib/artifact-access.js +369 -0
- package/lib/artifact-access.js.map +1 -0
- package/lib/artifacts.js +56 -0
- package/lib/artifacts.js.map +1 -0
- package/lib/client.js +952 -0
- package/lib/client.js.map +1 -0
- package/lib/config.js +117 -0
- package/lib/config.js.map +1 -0
- package/lib/errors.js +56 -0
- package/lib/errors.js.map +1 -0
- package/lib/exposure.js +213 -0
- package/lib/exposure.js.map +1 -0
- package/lib/index.js +97 -0
- package/lib/index.js.map +1 -0
- package/lib/paste-images.js +199 -0
- package/lib/paste-images.js.map +1 -0
- package/lib/paths.js +325 -0
- package/lib/paths.js.map +1 -0
- package/lib/runtime-install.js +601 -0
- package/lib/runtime-install.js.map +1 -0
- package/lib/runtime-manager.js +126 -0
- package/lib/runtime-manager.js.map +1 -0
- package/lib/runtime.js +1344 -0
- package/lib/runtime.js.map +1 -0
- package/lib/skill.js +139 -0
- package/lib/skill.js.map +1 -0
- package/lib/tools.js +528 -0
- package/lib/tools.js.map +1 -0
- package/lib/types/artifact-access.d.ts +61 -0
- package/lib/types/artifact-access.d.ts.map +1 -0
- package/lib/types/artifacts.d.ts +42 -0
- package/lib/types/artifacts.d.ts.map +1 -0
- package/lib/types/client/index.d.ts +179 -0
- package/lib/types/client/index.d.ts.map +1 -0
- package/lib/types/client/paste-images.d.ts +57 -0
- package/lib/types/client/paste-images.d.ts.map +1 -0
- package/lib/types/config.d.ts +73 -0
- package/lib/types/config.d.ts.map +1 -0
- package/lib/types/errors.d.ts +35 -0
- package/lib/types/errors.d.ts.map +1 -0
- package/lib/types/exposure.d.ts +40 -0
- package/lib/types/exposure.d.ts.map +1 -0
- package/lib/types/index.d.ts +18 -0
- package/lib/types/index.d.ts.map +1 -0
- package/lib/types/paste-images.d.ts +21 -0
- package/lib/types/paste-images.d.ts.map +1 -0
- package/lib/types/paths.d.ts +107 -0
- package/lib/types/paths.d.ts.map +1 -0
- package/lib/types/runtime-install.d.ts +49 -0
- package/lib/types/runtime-install.d.ts.map +1 -0
- package/lib/types/runtime-manager.d.ts +60 -0
- package/lib/types/runtime-manager.d.ts.map +1 -0
- package/lib/types/runtime.d.ts +389 -0
- package/lib/types/runtime.d.ts.map +1 -0
- package/lib/types/skill.d.ts +15 -0
- package/lib/types/skill.d.ts.map +1 -0
- package/lib/types/tools.d.ts +22 -0
- package/lib/types/tools.d.ts.map +1 -0
- package/lib/types/upstream.d.ts +207 -0
- package/lib/types/upstream.d.ts.map +1 -0
- package/lib/types/version.d.ts +15 -0
- package/lib/types/version.d.ts.map +1 -0
- package/lib/types/web-request.d.ts +4 -0
- package/lib/types/web-request.d.ts.map +1 -0
- package/lib/types/web.d.ts +74 -0
- package/lib/types/web.d.ts.map +1 -0
- package/lib/upstream.js +675 -0
- package/lib/upstream.js.map +1 -0
- package/lib/version.js +18 -0
- package/lib/version.js.map +1 -0
- package/lib/web-request.js +20 -0
- package/lib/web-request.js.map +1 -0
- package/lib/web.js +244 -0
- package/lib/web.js.map +1 -0
- package/package.json +139 -0
- package/runtime/requirements.lock +3 -0
- package/src/artifact-access.ts +386 -0
- package/src/artifacts.ts +85 -0
- package/src/client/index.tsx +866 -0
- package/src/client/paste-images.tsx +426 -0
- package/src/config.ts +177 -0
- package/src/errors.ts +62 -0
- package/src/exposure.ts +227 -0
- package/src/index.ts +122 -0
- package/src/paste-images.ts +234 -0
- package/src/paths.ts +348 -0
- package/src/runtime-install.ts +723 -0
- package/src/runtime-manager.ts +166 -0
- package/src/runtime.ts +1783 -0
- package/src/skill.ts +143 -0
- package/src/tools.ts +668 -0
- package/src/upstream.ts +861 -0
- package/src/version.ts +37 -0
- package/src/web-request.ts +17 -0
- package/src/web.ts +329 -0
- package/vendor/agent-vision-toolkit/CHANGELOG.md +16 -0
- package/vendor/agent-vision-toolkit/LICENSE +21 -0
- package/vendor/agent-vision-toolkit/README.md +399 -0
- package/vendor/agent-vision-toolkit/UPSTREAM_MANIFEST.json +89 -0
- package/vendor/agent-vision-toolkit/bin/crop +90 -0
- package/vendor/agent-vision-toolkit/bin/detect +13 -0
- package/vendor/agent-vision-toolkit/bin/glance +93 -0
- package/vendor/agent-vision-toolkit/bin/ground +13 -0
- package/vendor/agent-vision-toolkit/bin/trace +129 -0
- package/vendor/agent-vision-toolkit/detect.py +56 -0
- package/vendor/agent-vision-toolkit/ground.py +216 -0
- package/vendor/agent-vision-toolkit/skills/vision-tools/scripts/dominant_colors.py +224 -0
- package/vendor/agent-vision-toolkit/skills/vision-tools/scripts/extract_fg.py +278 -0
- package/vendor/agent-vision-toolkit/skills/vision-tools/scripts/html_shot.py +108 -0
- package/vendor/agent-vision-toolkit/skills/vision-tools/scripts/long_screenshot_ocr.py +1245 -0
- package/vendor/agent-vision-toolkit/skills/vision-tools/scripts/pixel_diff.py +88 -0
- package/vendor/agent-vision-toolkit/vision_client.py +156 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Anionex
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.i18n.yaml
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
1
|
+
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
|
2
|
+
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
|
3
|
+
# after editing either side, bring the other along and re-record with:
|
|
4
|
+
# pnpm run verify-translation-pairing --write README.md
|
|
5
|
+
README.md: 1b6de29f6b3b5e7373014301e2c33600f07e30ee
|
|
6
|
+
README.zh.md: c97d447ebfa8b776820069270b79d3db09c9daa5
|
package/README.md
ADDED
|
@@ -0,0 +1,383 @@
|
|
|
1
|
+

|
|
2
|
+
|
|
3
|
+
# DSH Vision Toolkit
|
|
4
|
+
|
|
5
|
+
[](https://x.com/anion_ex)
|
|
6
|
+
[](https://github.com/Anionex/dsh-vision-toolkit/releases/tag/v0.1.4)
|
|
7
|
+
[](tests)
|
|
8
|
+
[](LICENSE)
|
|
9
|
+
[](package.json)
|
|
10
|
+
[](runtime/requirements.lock)
|
|
11
|
+
[](cordis.patch.yml)
|
|
12
|
+
|
|
13
|
+
**Install:** `dsh plugin --profile web add @anionex/dsh-vision-toolkit`
|
|
14
|
+
|
|
15
|
+
**DSH Vision Toolkit brings [`agent-vision-toolkit`](https://github.com/Anionex/agent-vision-toolkit) into DeepSeek Harness as a native Profile Bundle.**
|
|
16
|
+
|
|
17
|
+
Give text-only DSH agents eyes—and keep vision in the harness—with intent-aware image Q&A, OCR, original-pixel grounding, UI restoration, pixel verification, managed Artifacts, and Web Settings. Ten independent tools replace shell glue with structured schemas and Agent-scoped progressive exposure.
|
|
18
|
+
|
|
19
|
+
**Upstream toolkit:** [Anionex/agent-vision-toolkit](https://github.com/Anionex/agent-vision-toolkit) · **Project website:** [agent-vision.anionex.me](https://agent-vision.anionex.me)
|
|
20
|
+
|
|
21
|
+
English | [中文](README.zh.md)
|
|
22
|
+
|
|
23
|
+
## Why this exists
|
|
24
|
+
|
|
25
|
+
`agent-vision-toolkit` treats vision as an Agent-callable capability rather than a property of the base model. Its method carries the reason for looking into the visual request, moves from the whole image to targeted regions, and verifies coordinates, colors, geometry, and differences with focused tools instead of accepting a generic description as evidence.
|
|
26
|
+
|
|
27
|
+
DSH Vision Toolkit preserves that method while replacing CLI installation and Bash argument construction with native schemas, DSH Credentials, lifecycle-managed runtime preparation, structured Session-log results, previewable Artifacts, dedicated Web cards, and Settings. The Agent loads one versioned Skill and receives the ten visual schemas only when the current task needs them.
|
|
28
|
+
|
|
29
|
+
The package delivers the committed P0 and P1 product scope. P2's stable `ctx.visionToolkit` service remains deliberately unpublished until an independent plugin becomes a real consumer; the internal runtime does not pretend that an unvalidated ecosystem API is stable.
|
|
30
|
+
|
|
31
|
+
## Proven use cases from agent-vision-toolkit
|
|
32
|
+
|
|
33
|
+
The first two panels are official upstream reference runs from the same pinned `agent-vision-toolkit` lineage packaged by this bundle. The image Q&A and screenshot-guided debugging panel is a live DeepSeek Harness Web session, showing the same workflows through DSH. See the [asset provenance record](assets/upstream/README.md) for the upstream source images.
|
|
34
|
+
|
|
35
|
+
### Infographic restoration: screenshot to editable HTML/CSS
|
|
36
|
+
|
|
37
|
+
<p align="center">
|
|
38
|
+
<img src="assets/upstream/infographic-reference.webp" width="49%" alt="Upstream reference screenshot of a three-stage model-training infographic." />
|
|
39
|
+
<img src="assets/upstream/infographic-result.webp" width="49%" alt="Upstream editable HTML and CSS reconstruction of the model-training infographic." />
|
|
40
|
+
</p>
|
|
41
|
+
|
|
42
|
+
*Left: source screenshot. Right: the editable HTML/CSS result from the upstream [infographic-restoration reference](https://github.com/Anionex/agent-vision-toolkit/blob/c27d1a300962b553c0884993c575cd3e819465ce/examples/infographic-restoration/how-is-the-model-trained.html).*
|
|
43
|
+
|
|
44
|
+
### UI restoration: sketch to working interface
|
|
45
|
+
|
|
46
|
+
<p align="center">
|
|
47
|
+
<img src="assets/upstream/ui-sketch.webp" width="49%" alt="Upstream hand-drawn JupyterLab workspace used as a UI restoration reference." />
|
|
48
|
+
<img src="assets/upstream/ui-result.webp" width="49%" alt="Upstream JupyterLab-style working interface reconstructed from the hand-drawn reference." />
|
|
49
|
+
</p>
|
|
50
|
+
|
|
51
|
+
*Left: hand-drawn input. Right: the upstream reconstructed interface; the complete method lives in the [UI restoration playbook](https://github.com/Anionex/agent-vision-toolkit/blob/c27d1a300962b553c0884993c575cd3e819465ce/skills/vision-tools/references/restore-ui.md).*
|
|
52
|
+
|
|
53
|
+
### Image Q&A and screenshot-guided debugging
|
|
54
|
+
|
|
55
|
+
<p align="center">
|
|
56
|
+
<img src="assets/dsh-conversation-image-qa.png" width="49%" alt="DSH Web session in which a text-only agent answers a focused question about a UI reference image." />
|
|
57
|
+
<img src="assets/dsh-conversation-screenshot-debugging.png" width="49%" alt="DSH Web session in which the agent uses a screenshot comparison to diagnose mismatched UI fields and recommend vision_pixel_diff." />
|
|
58
|
+
</p>
|
|
59
|
+
|
|
60
|
+
*Left: intent-aware image Q&A in DSH Web. Right: a DSH Web screenshot-debugging turn that lists the concrete UI differences and continues toward `vision_pixel_diff`. The upstream workflow source is the same [`agent-vision-toolkit` reference](https://github.com/Anionex/agent-vision-toolkit/blob/c27d1a300962b553c0884993c575cd3e819465ce/README.md#real-world-effects).*
|
|
61
|
+
|
|
62
|
+
DSH Vision Toolkit adds native tool schemas, versioned lifecycle, Credentials, structured Session results, Artifacts, Web presentation, Settings, and progressive exposure around these upstream capabilities. The next section is the reproducible proof executed and checked into this DSH repository.
|
|
63
|
+
|
|
64
|
+
## DSH-native proof: reference-to-pixel verification
|
|
65
|
+
|
|
66
|
+
The checked-in UI-restoration workflow renders an intentionally inaccurate HTML implementation, measures a `6.04%` pixel difference across six non-zero regions, iterates, and reaches an exact `0%` difference against the reference at `1200 × 720`.
|
|
67
|
+
|
|
68
|
+
<p>
|
|
69
|
+
<img src="examples/ui-restoration/assets/initial.png" width="49%" alt="Initial UI restoration candidate before Vision Toolkit iteration, with measurable layout and styling differences from the reference." />
|
|
70
|
+
<img src="examples/ui-restoration/assets/implementation.png" width="49%" alt="Final UI restoration output reproduced by the checked-in workflow with zero pixel difference from the reference." />
|
|
71
|
+
</p>
|
|
72
|
+
|
|
73
|
+
| Verified surface | Evidence |
|
|
74
|
+
|---|---|
|
|
75
|
+
| Product scope | 10 independent visual tools, matching `vision-tools` Skill, Artifacts, dedicated Web cards, and live Settings |
|
|
76
|
+
| Automated coverage | 17 Vitest files / 136 passing tests, plus a dependency-free portable package check |
|
|
77
|
+
| Real profiles | Clean temporary Web and Headless installation, activation, disable, re-enable, and uninstall |
|
|
78
|
+
| Visual acceptance | Reproducible HTML screenshot → pixel diff example with a final `0%` difference |
|
|
79
|
+
|
|
80
|
+
## Highlights
|
|
81
|
+
|
|
82
|
+
- **See images without bloating every prompt:** only `vision_toolkit_activate` is initially visible; loading `vision-tools` mounts ten independent schemas for that Agent and keeps version/health administration out of model context.
|
|
83
|
+
- **Act on coordinates instead of parsing prose:** grounding and detection return original-image pixel boxes, while every model-visible result remains structured text or JSON.
|
|
84
|
+
- **Deliver files, not temporary output:** crop, trace, OCR, pixel diff, foreground extraction, and HTML rendering produce described Artifacts that the Web client can preview, download, or open locally.
|
|
85
|
+
- **Keep runtime and credentials controlled:** DSH Credentials hold API keys, managed mode prepares an exact isolated Python environment, and a failed Settings candidate cannot replace the serving generation.
|
|
86
|
+
- **Close the visual verification loop:** local HTML rendering and pixel-diff ranking support reference → implementation → screenshot → measured iteration without a model-native image channel.
|
|
87
|
+
- **Use the same bundle in Web and Headless profiles:** Web adds cards, previews, Settings, and health actions; Headless receives the same tool semantics and complete structured results.
|
|
88
|
+
|
|
89
|
+
## Quick start
|
|
90
|
+
|
|
91
|
+
Prerequisites: DeepSeek Harness `0.1.0-rc.6` or a compatible later `0.1.x` release, Python 3.11+, and `pnpm` available to `dsh plugin`. Install the published bundle from npm, add it to the profiles you use, and confirm the bundle row:
|
|
92
|
+
|
|
93
|
+
```sh
|
|
94
|
+
dsh plugin --profile web add @anionex/dsh-vision-toolkit
|
|
95
|
+
dsh plugin --profile headless add @anionex/dsh-vision-toolkit
|
|
96
|
+
dsh --profile web --dump-config | grep vision-toolkit
|
|
97
|
+
dsh --profile headless --dump-config | grep vision-toolkit
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
Legacy profiles must use `nodeLinker: hoisted` and `autoInstallPeers: false` in their `pnpm-workspace.yaml`. An updated DSH launcher repairs these owned settings before `dsh plugin` runs; when using an older launcher, set them before installation so pnpm does not assemble a second Harness dependency graph inside the profile.
|
|
101
|
+
|
|
102
|
+
Restart a running Web profile, open **Settings → Vision Toolkit**, select a DSH Credential for remote tools, and explicitly run **Test connection**. In a conversation, make an image available as a workspace path, invoke `/vision-tools`, and ask the Agent to call a specific `vision_*` tool. Local crop, trace, pixel, color, foreground, and HTML operations do not require a visual API credential.
|
|
103
|
+
|
|
104
|
+
## How it works
|
|
105
|
+
|
|
106
|
+
```mermaid
|
|
107
|
+
flowchart LR
|
|
108
|
+
User["Workspace image or local HTML"] --> Skill["vision-tools Skill"]
|
|
109
|
+
Skill --> Activate["Agent-scoped activation"]
|
|
110
|
+
Activate --> Tools["10 independent vision_* tools"]
|
|
111
|
+
Tools --> Runtime["Shared VisionToolkitRuntime"]
|
|
112
|
+
Credentials["DSH Credentials"] --> Runtime
|
|
113
|
+
Settings["Web Settings and health"] --> Runtime
|
|
114
|
+
Runtime --> Upstream["Pinned agent-vision-toolkit"]
|
|
115
|
+
Runtime --> Remote["Configured vision API"]
|
|
116
|
+
Upstream --> Result["Text, coordinates, JSON"]
|
|
117
|
+
Remote --> Result
|
|
118
|
+
Runtime --> Artifacts["Workspace Artifacts"]
|
|
119
|
+
Result --> Session["Reconstructable Session log"]
|
|
120
|
+
Artifacts --> Web["Preview, download, or open file"]
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
Tool definitions call one runtime; the runtime validates paths, limits, credentials, cancellation, and deadlines before dispatching to the pinned upstream snapshot or configured OpenAI-compatible vision endpoint. Web presentation consumes the same structured results and Artifact descriptors, so it does not change Headless behavior. Health, connection testing, and version inspection stay in Settings rather than model tool schemas.
|
|
124
|
+
|
|
125
|
+
## Tools
|
|
126
|
+
|
|
127
|
+
| Tool | Execution | Structured result | Artifact delivery |
|
|
128
|
+
|---|---|---|---|
|
|
129
|
+
| `vision_glance` | Remote vision API | Description, targeted answer, OCR, or multi-image comparison | None |
|
|
130
|
+
| `vision_ground` | Remote vision API; optional local preview | Target, original-image dimensions, and pixel boxes | Optional labeled PNG |
|
|
131
|
+
| `vision_detect` | Remote vision API; optional local preview | Numbered element inventory and original-image pixel boxes | Optional numbered PNG |
|
|
132
|
+
| `vision_trace` | Local pinned vtracer pipeline | SVG geometry status, path count, scale, and size | SVG |
|
|
133
|
+
| `vision_crop` | Local Pillow pipeline | Applied pixel box, dimensions, format, and clamp status | PNG or JPEG |
|
|
134
|
+
| `vision_pixel_diff` | Local NumPy/Pillow pipeline | Difference percentage and ranked grid regions | PNG heatmap and JSON report |
|
|
135
|
+
| `vision_long_screenshot_ocr` | Local split/audit; remote OCR unless `splitOnly=true` | Chunk boundaries, reuse state, completion state, and run directory | Markdown, manifest, boundary audit, chunk PNGs, and OCR sidecars |
|
|
136
|
+
| `vision_extract_foreground` | Local pinned extraction pipeline | Selected box, component counts, foreground coverage, and dimensions | Transparent PNG |
|
|
137
|
+
| `vision_dominant_colors` | Local pinned color analysis | Extracted palette or pixel-backed candidate ranking | None |
|
|
138
|
+
| `vision_html_screenshot` | Local Chrome/Chromium/Edge adapter | Authorized source facts, viewport, and rendered dimensions | PNG |
|
|
139
|
+
|
|
140
|
+
The plugin does not reimplement visual algorithms. Its DSH-owned layer validates paths and limits, resolves credentials, calls the pinned upstream scripts with argv vectors, parses their exact output contracts, classifies failures, describes files, and projects results to the model and Web client.
|
|
141
|
+
|
|
142
|
+
## Progressive model exposure
|
|
143
|
+
|
|
144
|
+
Runtime readiness is profile-wide, but the ten visual execution schemas are Agent-scoped. Before an Agent loads `vision-tools`, the plugin contributes only the small `vision_toolkit_activate` bootstrap; the visual tools are absent from that Agent's request schema. A successful call to the standard `skill` tool with `name="vision-tools"` mounts all ten tools automatically for the next model step and hides the bootstrap. A direct `/vision-tools` invocation injects the Skill instructions; if the visual tools are still absent, those instructions require one `vision_toolkit_activate` call. Activation affects only that Agent, restores when the Session contains durable evidence matching the bundled Skill version, and lasts until the Agent or plugin is disposed.
|
|
145
|
+
|
|
146
|
+
Health checks, connection testing, and plugin/upstream version inspection are administrative Web Settings operations. `vision_toolkit_health` and `vision_toolkit_version` are not model tools and never enter an Agent's schema, including after visual-tool activation.
|
|
147
|
+
|
|
148
|
+
## Requirements
|
|
149
|
+
|
|
150
|
+
- DeepSeek Harness with a Web or Headless profile and `pnpm` available to `dsh plugin`.
|
|
151
|
+
- Python 3.11 or newer. Managed mode creates an isolated environment, so users do not install the upstream CLI or Python packages manually.
|
|
152
|
+
- Network access on the first managed-runtime activation unless the exact packages in `runtime/requirements.lock` are already available in the configured package cache.
|
|
153
|
+
- An OpenAI-compatible vision endpoint and DSH Credential for `vision_glance`, `vision_ground`, `vision_detect`, and non-split-only long-screenshot OCR. Local tools remain usable without that credential.
|
|
154
|
+
- Chrome, Chromium, or Edge only for `vision_html_screenshot`; all other tools remain available when no supported browser is installed.
|
|
155
|
+
- PNG, JPEG, GIF, or WebP inputs inside the session workspace or an explicitly configured `allowedDirs` root.
|
|
156
|
+
|
|
157
|
+
## Install and lifecycle
|
|
158
|
+
|
|
159
|
+
### Install
|
|
160
|
+
|
|
161
|
+
Install the bundle into each profile that should expose it:
|
|
162
|
+
|
|
163
|
+
```sh
|
|
164
|
+
dsh plugin --profile web add @anionex/dsh-vision-toolkit
|
|
165
|
+
dsh plugin --profile headless add @anionex/dsh-vision-toolkit
|
|
166
|
+
dsh --profile web --dump-config | grep vision-toolkit
|
|
167
|
+
dsh --profile headless --dump-config | grep vision-toolkit
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
Restart a long-lived Web profile after installation. The host discovers the built browser bundle from `package.json`'s `dsh.client` declaration at process startup; the legacy top-level `dshClient` field is not scanned.
|
|
171
|
+
|
|
172
|
+
The first managed start verifies the packaged upstream manifest and atomically prepares an isolated environment under `DSH_HOME/cache/dsh-vision-toolkit`. Only after preparation succeeds does the plugin publish the same-version `vision-tools` Skill and activation bootstrap; each Agent receives the execution tools only after loading that Skill. An initial preparation failure leaves the Web Settings repair surface available but exposes neither model capability nor a misleading Skill.
|
|
173
|
+
|
|
174
|
+
### Disable and re-enable
|
|
175
|
+
|
|
176
|
+
Set the bundle row to `disabled: true` in a profile patch or overlay:
|
|
177
|
+
|
|
178
|
+
```yaml
|
|
179
|
+
- id: vision-toolkit
|
|
180
|
+
disabled: true
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
Remove the flag or set it to `false` to re-enable the plugin. Disposal first cancels plugin-owned visual operations, then removes every Agent-scoped tool, the bootstrap, and the Skill; reactivation prepares the configured runtime before any model capability becomes visible. User configuration and completed Artifacts remain intact.
|
|
184
|
+
|
|
185
|
+
### Upgrade
|
|
186
|
+
|
|
187
|
+
For a registry installation, update the dependency through the profile package manager:
|
|
188
|
+
|
|
189
|
+
```sh
|
|
190
|
+
dsh plugin --profile web update @anionex/dsh-vision-toolkit
|
|
191
|
+
dsh plugin --profile headless update @anionex/dsh-vision-toolkit
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
For a local path installation, run `add` again against the replacement checkout or tarball. Settings remain in the profile's Settings provider. A candidate runtime is fully validated and prepared before it is persisted and made active; a failed or obsolete concurrent candidate cannot replace the current serving generation.
|
|
195
|
+
|
|
196
|
+
### Uninstall
|
|
197
|
+
|
|
198
|
+
```sh
|
|
199
|
+
dsh plugin --profile web remove @anionex/dsh-vision-toolkit
|
|
200
|
+
dsh plugin --profile headless remove @anionex/dsh-vision-toolkit
|
|
201
|
+
```
|
|
202
|
+
|
|
203
|
+
`dsh plugin remove` removes both the dependency and its bundle layer. The profile no longer exposes the activation bootstrap, Agent-scoped Vision Toolkit tools, or Skill entries. Managed cache data may be deleted separately when no profile uses the package; it is not active configuration and cannot register anything by itself.
|
|
204
|
+
|
|
205
|
+
## Configure
|
|
206
|
+
|
|
207
|
+
The bundle defaults to the managed runtime. A profile patch can override the provider and limits:
|
|
208
|
+
|
|
209
|
+
```yaml
|
|
210
|
+
- id: vision-toolkit
|
|
211
|
+
config:
|
|
212
|
+
provider:
|
|
213
|
+
baseUrl: https://api.inferera.com/v1
|
|
214
|
+
credential: VISION_API_KEY
|
|
215
|
+
model: gemini-3.6-flash
|
|
216
|
+
language: zh
|
|
217
|
+
timeoutMs: 60000
|
|
218
|
+
maxImageBytes: 10485760
|
|
219
|
+
maxImagePixels: 40000000
|
|
220
|
+
concurrency: 4
|
|
221
|
+
runtime:
|
|
222
|
+
mode: managed
|
|
223
|
+
allowedDirs: []
|
|
224
|
+
```
|
|
225
|
+
|
|
226
|
+
### Configuration fields
|
|
227
|
+
|
|
228
|
+
| Field | Default | Contract |
|
|
229
|
+
|---|---|---|
|
|
230
|
+
| `provider.baseUrl` | `https://api.inferera.com/v1` | OpenAI-compatible base URL; normalized without trailing slashes |
|
|
231
|
+
| `provider.credential` | `VISION_API_KEY` | DSH Credential reference, never a secret value |
|
|
232
|
+
| `provider.model` | `gemini-3.6-flash` | Multimodal model name sent to remote tools |
|
|
233
|
+
| `language` | `zh` | Vision output language: `zh` or `en` |
|
|
234
|
+
| `timeoutMs` | `60000` | Whole-operation deadline, 1000-600000 ms; each tool may request a narrower override |
|
|
235
|
+
| `maxImageBytes` | `10485760` | Encoded-byte limit per input image |
|
|
236
|
+
| `maxImagePixels` | `40000000` | Decoded-pixel limit per input image |
|
|
237
|
+
| `concurrency` | `4` | In-flight operations per session, 1-16 |
|
|
238
|
+
| `runtime.mode` | `managed` | `managed` uses the packaged snapshot; `external` accepts only the exact pin |
|
|
239
|
+
| `runtime.agentVisionToolkitPath` | unset | Required in `external` mode; exported exact snapshot or clean pinned Git checkout |
|
|
240
|
+
| `runtime.python` | unset | Optional Python 3.11+ bootstrap/interpreter override |
|
|
241
|
+
| `allowedDirs` | `[]` | Additional realpath-resolved input roots; the session workspace is always allowed |
|
|
242
|
+
|
|
243
|
+
### Credentials
|
|
244
|
+
|
|
245
|
+
Create or replace the referenced secret through DSH Credentials:
|
|
246
|
+
|
|
247
|
+
```sh
|
|
248
|
+
dsh credentials set VISION_API_KEY
|
|
249
|
+
```
|
|
250
|
+
|
|
251
|
+
The reference is stored in Settings; the value is not. Remote operations resolve it once per call and inject it only into that subprocess environment. The plugin excludes user `.env` files, checkout `.env` files, `PYTHONPATH`, `PYTHONHOME`, `VIRTUAL_ENV`, and user site-packages so ambient Python or upstream configuration cannot override the selected DSH provider. Logs, errors, tool results, Artifact metadata, and Settings responses never contain the secret.
|
|
252
|
+
|
|
253
|
+
### Managed and external runtimes
|
|
254
|
+
|
|
255
|
+
Managed mode verifies `vendor/agent-vision-toolkit/UPSTREAM_MANIFEST.json`, prefers `uv`, falls back to `venv` plus pip, installs exact versions from `runtime/requirements.lock`, coordinates concurrent preparation with a heartbeat lock, and publishes a staged environment only after all probes pass.
|
|
256
|
+
|
|
257
|
+
External mode is intended for development or controlled deployments:
|
|
258
|
+
|
|
259
|
+
```yaml
|
|
260
|
+
- id: vision-toolkit
|
|
261
|
+
config:
|
|
262
|
+
runtime:
|
|
263
|
+
mode: external
|
|
264
|
+
agentVisionToolkitPath: /opt/agent-vision-toolkit
|
|
265
|
+
python: python3.12
|
|
266
|
+
```
|
|
267
|
+
|
|
268
|
+
The path must be an exported copy matching the packaged manifest or the root of a clean Git checkout at `c27d1a300962b553c0884993c575cd3e819465ce`. Modified tracked files and untracked files are rejected because they can change or shadow the pinned Python behavior.
|
|
269
|
+
|
|
270
|
+
## Web Settings
|
|
271
|
+
|
|
272
|
+
The Web profile registers a Vision Toolkit Settings section for the provider URL, Credential reference, model, language, timeout, byte/pixel limits, concurrency, runtime mode, Python override, external source path, and allowed directories. It also shows plugin/upstream versions, the active runtime generation, non-secret Credential configured/source/writable facts, runtime paths, health results, and Artifact-route availability.
|
|
273
|
+
|
|
274
|
+
`Save and apply` validates the complete value, prepares the candidate Python/upstream runtime, commits the Settings revision, and only then atomically switches generations. A rejected candidate leaves the previous generation serving and is reported separately from a genuinely unavailable runtime. `Reload` always restores the authoritative saved value, even when its revision did not change, so a rejected browser draft is discarded. If initial startup cannot prepare a runtime, the Settings route remains available so a valid configuration can make the first generation operational. A stale browser revision receives a conflict instead of overwriting a newer save; reload before retrying. A read-only Settings provider allows inspection and health checks but disables saves.
|
|
275
|
+
|
|
276
|
+
`Run health check` performs local checks only. `Test connection` is an explicit action that sends the configured Credential to `GET /models`; it uploads no image and creates no completion. Plugin load and ordinary Settings reads never make that request.
|
|
277
|
+
|
|
278
|
+
Health, connection testing, and plugin/upstream version inspection are administrative Web Settings capabilities rather than model-facing tools, so their schemas never occupy an agent request.
|
|
279
|
+
|
|
280
|
+
## Artifacts and presentation
|
|
281
|
+
|
|
282
|
+
Artifact-producing tools write only under `<workspace>/.dsh-vision-toolkit/artifacts`, either as one validated file or an atomically committed run directory. Each model-visible descriptor contains the path, filename, MIME type, kind, description, source tool, preview intent, and byte size, so Headless agents can reuse the path in later calls without browser support. Before a traced SVG is committed, the runtime parses it as XML: standard declarations and comments are accepted, while doctypes, malformed or multi-root documents, a non-SVG namespace, and reported path/byte mismatches are rejected.
|
|
283
|
+
|
|
284
|
+
When the Web HTTP host is present, presentation-only metadata adds signed capability URLs for preview and download without altering the canonical tool result. Every read revalidates the signature, managed-root fence, path components, regular-file status, size, device/inode identity where available, extension, and MIME. SVG responses use a sandboxed no-resource CSP and the client renders them in a sandboxed iframe. Without an HTTP host, the same cards retain `Open file` through `openFile` and show the descriptor instead of inventing an inaccessible URL.
|
|
285
|
+
|
|
286
|
+
## Usage patterns
|
|
287
|
+
|
|
288
|
+
### Basic calls
|
|
289
|
+
|
|
290
|
+
```text
|
|
291
|
+
vision_glance images=["screenshot.png"] query="What error is shown?"
|
|
292
|
+
vision_ground image="screenshot.png" target="the send button" preview=true
|
|
293
|
+
vision_detect image="screenshot.png" category="buttons" preview=true
|
|
294
|
+
vision_crop image="screenshot.png" region="1067,841,1108,881"
|
|
295
|
+
vision_trace image="icon.png" color=true output="icon.svg"
|
|
296
|
+
vision_pixel_diff original="reference.png" rebuilt="actual.png" runName="comparison"
|
|
297
|
+
vision_long_screenshot_ocr image="page.png" mode="general" jobs=2
|
|
298
|
+
vision_extract_foreground image="logo.png" mode="color"
|
|
299
|
+
vision_dominant_colors image="screen.png" region="0,0,600,300" top=8
|
|
300
|
+
vision_html_screenshot source="implementation.html" width=1200 height=720
|
|
301
|
+
```
|
|
302
|
+
|
|
303
|
+
Common workflows are `vision_ground` → `vision_crop` → `vision_glance`, `vision_ground` → `vision_crop` → `vision_trace`, and reference image → `vision_html_screenshot` → `vision_pixel_diff`. Grounding and detection boxes always use original-image pixels (`x1/y1/x2/y2`).
|
|
304
|
+
|
|
305
|
+
### UI restoration example
|
|
306
|
+
|
|
307
|
+
The checked-in [UI restoration example](examples/ui-restoration/README.md) renders a reference, an intentionally inaccurate first implementation, and the final implementation through `vision_html_screenshot`, then compares both candidates through `vision_pixel_diff`:
|
|
308
|
+
|
|
309
|
+
```sh
|
|
310
|
+
npm run example:ui-restoration
|
|
311
|
+
npm run example:ui-restoration:write
|
|
312
|
+
```
|
|
313
|
+
|
|
314
|
+
The committed evidence records an initial `6.04%` difference across six non-zero worst regions and a final `0%` difference with no non-zero worst region. Check mode reproduces the tool path and verifies the committed assets; write mode intentionally refreshes the evidence.
|
|
315
|
+
|
|
316
|
+
## Security and execution model
|
|
317
|
+
|
|
318
|
+
- Inputs resolve against the session workspace and configured `allowedDirs`; realpath containment prevents traversal and symlink escape.
|
|
319
|
+
- Pillow decodes every image before a remote request and verifies bytes, pixels, dimensions, and extension/content agreement. Unsupported or oversized images fail before upload.
|
|
320
|
+
- Outputs use random staging files or directories inside the real managed destination, reject symbolic links, and commit only after format and contract validation.
|
|
321
|
+
- Remote vision prompts explicitly classify text and instructions visible inside images as untrusted content. The native tool descriptions and bundled skill likewise tell the text agent to treat derived descriptions, labels, and OCR as visual evidence rather than executable instructions.
|
|
322
|
+
- All upstream processes use argv vectors through `ctx.subprocess`, inherit caller cancellation, share one hard operation deadline, and terminate with the operation instead of continuing in the background. Plugin disposal aborts active calls before unregistering their tools.
|
|
323
|
+
- One live Session retains only the most recent successful `vision_glance` result. An immediate repeat reuses it only when image content, query/OCR mode, region, endpoint, model, language, and Credential are unchanged; failures and other Sessions never share the entry.
|
|
324
|
+
- Model-visible data is text, numbers, coordinates, structured JSON, and file descriptors. Tool calls/results remain reconstructable from the Session log; browser previews are presentation metadata only.
|
|
325
|
+
- Metrics include tool name, total/upstream duration, bounded image counts/bytes/pixels, cache hits, model, and error category; they exclude base64, authentication headers, secrets, and unbounded upstream output.
|
|
326
|
+
|
|
327
|
+
`vision_html_screenshot` accepts only authorized local `.html` or `.htm` files, disables network access in the pinned adapter, and launches a Chrome-family browser with `--headless=new`, `--use-mock-keychain`, `--incognito`, and a unique `--user-data-dir` under the system temporary directory. The profile is removed after every call, so headless rendering does not touch the user's daily Chrome profile or macOS login keychain.
|
|
328
|
+
|
|
329
|
+
## Troubleshooting
|
|
330
|
+
|
|
331
|
+
| Symptom | Resolution |
|
|
332
|
+
|---|---|
|
|
333
|
+
| `Model "..." does not support image input. (attachment-error)` | The image used DSH's native model-attachment channel, so a text-only model rejected the turn before the Skill or Vision Toolkit could run. Use DSH Paste Input's attachment button, paste, or drop flow so the file is copied into the session workspace and represented by a path, then invoke `/vision-tools`. Restart the Web profile and reload the page after installing or upgrading either browser plugin. |
|
|
334
|
+
| Credential reported missing | Run `dsh credentials set <REF>`, ensure `provider.credential` names that reference, then rerun health. Local-only tools do not need it. |
|
|
335
|
+
| Runtime preparation fails | Read the Settings runtime error, verify Python 3.11+, package-cache/network access, disk permissions, and the exact external pin. Save only after correcting the candidate; the active generation remains intact. |
|
|
336
|
+
| Chrome is not found | Install Chrome, Chromium, or Edge or configure an environment where one is discoverable. Only `vision_html_screenshot` is unavailable. |
|
|
337
|
+
| macOS displays a keychain dialog | Confirm the current built adapter is installed and no stale external `html_shot`/headless Chrome process is running. Current launches use a mock keychain and disposable profile; cancel the dialog rather than resetting the login keychain. |
|
|
338
|
+
| Input or output path is rejected | Move the file into the session workspace or add an intentional real directory to `allowedDirs`; remove escaping symlinks. Outputs accept a filename, not an absolute or nested path. |
|
|
339
|
+
| Vision service returns 401/403 | Replace the Credential value or select the correct reference and endpoint. Errors remain redacted. |
|
|
340
|
+
| Vision service returns 429 | Retry after the provider's rate-limit window or lower `concurrency`. The plugin does not silently switch providers. |
|
|
341
|
+
| Operation times out or is cancelled | Raise `timeoutMs` within 1000-600000 ms, reduce image/chunk work, or rerun after cancellation. The subprocess/request is stopped with the operation. |
|
|
342
|
+
| Settings save reports a conflict | Reload the section to obtain the current revision, reapply the intended edit, and save again. |
|
|
343
|
+
| Settings is read-only | Change the active Settings provider or edit the owning profile configuration; the plugin cannot bypass provider writability. |
|
|
344
|
+
| Artifact preview is unavailable | Use `Open file` or the model-visible path. Preview/download URLs exist only while a Web HTTP route is attached. |
|
|
345
|
+
|
|
346
|
+
## Development and verification
|
|
347
|
+
|
|
348
|
+
```sh
|
|
349
|
+
pnpm install --frozen-lockfile --trust-lockfile
|
|
350
|
+
pnpm run verify:portable
|
|
351
|
+
pnpm run build
|
|
352
|
+
pnpm test
|
|
353
|
+
pnpm run example:ui-restoration
|
|
354
|
+
pnpm pack --dry-run
|
|
355
|
+
```
|
|
356
|
+
|
|
357
|
+
`pnpm run verify:portable` is the dependency-free portable verification gate: it validates the vendored snapshot, package metadata and exports, committed JavaScript syntax, README links and images, required facade files, social-preview dimensions, and the dry-run tarball. The full TypeScript build and test suite run from this standalone checkout against the locked DSH `0.1.0-rc.6` registry packages; the client build also has a separate compiler face that resolves the packages' public exports without internal path aliases. The real Profile acceptance runs when compatible `dsh` and `pnpm` commands are on PATH, and CI requires that path instead of silently skipping it.
|
|
358
|
+
|
|
359
|
+
`pnpm run build` verifies the vendored manifest before emitting JavaScript, declarations, and the loader-compatible Web client. The package commits `lib/`, so installation from a checkout does not require a consumer-side build. The keyless real-profile test installs into a clean `DSH_HOME`, boots Headless, executes all five P0 tools plus representative P1 local/remote tools through real tool calls, verifies disable and re-enable behavior, and uninstalls the bundle. See the [requirements traceability reference](docs/requirements-traceability/README.md) for the implementation and verification home of every P0/P1 requirement.
|
|
360
|
+
|
|
361
|
+
Update the upstream snapshot only through `pnpm run upstream:sync -- <checkout>`, inspect the source and license, regenerate the manifest, and update the adapter compatibility tests and committed `lib/` in the same change. The runtime never fetches upstream `main`.
|
|
362
|
+
|
|
363
|
+
## Project status and scope
|
|
364
|
+
|
|
365
|
+
Version `0.1.4` is the current public npm release. P0 and P1 are product commitments in this package. P2 is a design threshold: no stable `ctx.visionToolkit` service, capability-discovery API, or provider ecosystem is published until at least one independent plugin consumes the internal capability shape. Web upload, drag-and-drop, camera/video/audio/document ingestion, interactive box editing, automatic GUI clicking, service clusters, model routing, model voting, and cross-session vision caches remain outside the current product.
|
|
366
|
+
|
|
367
|
+
## Community and About
|
|
368
|
+
|
|
369
|
+
- Read [CONTRIBUTING.md](CONTRIBUTING.md) before proposing code, protocol, or upstream-snapshot changes.
|
|
370
|
+
- Use [GitHub Issues](https://github.com/Anionex/dsh-vision-toolkit/issues) for reproducible bugs, focused feature requests, and usage questions; use [SUPPORT.md](SUPPORT.md) to choose the right channel.
|
|
371
|
+
- Report vulnerabilities privately through the process in [SECURITY.md](SECURITY.md), never in a public issue.
|
|
372
|
+
- Follow releases and compatibility notes in [CHANGELOG.md](CHANGELOG.md).
|
|
373
|
+
- Optional sponsorship is described transparently in [FUNDING.md](FUNDING.md); support does not purchase roadmap priority or private support.
|
|
374
|
+
- Use the upstream [project website](https://agent-vision.anionex.me) and [repository](https://github.com/Anionex/agent-vision-toolkit) for the general toolkit, cross-harness integrations, visual-task playbooks, and reference runs.
|
|
375
|
+
- Star, share, contribute to, or sponsor `agent-vision-toolkit` if its algorithms or methods save time; DSH-specific bugs and integration requests belong in this repository.
|
|
376
|
+
|
|
377
|
+
[`agent-vision-toolkit`](https://github.com/Anionex/agent-vision-toolkit) was created by [Anionex](https://anionex.me/). This repository maintains its native DeepSeek Harness integration: DSH owns lifecycle, security, structured schemas, Credentials, Artifacts, and Web presentation, while the upstream project remains the home of the visual algorithms and reusable playbooks.
|
|
378
|
+
|
|
379
|
+
If you would like to follow my future work, [follow me on X](https://x.com/anion_ex) or [GitHub](https://github.com/Anionex).
|
|
380
|
+
|
|
381
|
+
## License
|
|
382
|
+
|
|
383
|
+
The plugin is MIT-licensed. The packaged `agent-vision-toolkit` snapshot retains its upstream MIT license in `vendor/agent-vision-toolkit/LICENSE` and remains the sole implementation of its visual algorithms.
|