@anionex/dsh-vision-toolkit 0.1.5

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (152) hide show
  1. package/LICENSE +21 -0
  2. package/README.i18n.yaml +6 -0
  3. package/README.md +383 -0
  4. package/README.zh.md +383 -0
  5. package/assets/dsh-conversation-artifact.png +0 -0
  6. package/assets/dsh-conversation-image-qa-top.png +0 -0
  7. package/assets/dsh-conversation-image-qa.png +0 -0
  8. package/assets/dsh-conversation-pixel-diff.png +0 -0
  9. package/assets/dsh-conversation-screenshot-debugging-top.png +0 -0
  10. package/assets/dsh-conversation-screenshot-debugging.png +0 -0
  11. package/assets/dsh-conversation-tool-call.png +0 -0
  12. package/assets/dsh-conversation-vision-trace.png +0 -0
  13. package/assets/hero.png +0 -0
  14. package/assets/social-preview.png +0 -0
  15. package/assets/upstream/README.md +16 -0
  16. package/assets/upstream/image-qa.webp +0 -0
  17. package/assets/upstream/infographic-reference.webp +0 -0
  18. package/assets/upstream/infographic-result.webp +0 -0
  19. package/assets/upstream/screenshot-debugging.webp +0 -0
  20. package/assets/upstream/ui-result.webp +0 -0
  21. package/assets/upstream/ui-sketch.webp +0 -0
  22. package/assets/vision-settings.png +0 -0
  23. package/cordis.patch.yml +6 -0
  24. package/docs/assets/vision-settings.png +0 -0
  25. package/docs/requirements-traceability/README.i18n.yaml +6 -0
  26. package/docs/requirements-traceability/README.md +75 -0
  27. package/docs/requirements-traceability/README.zh.md +75 -0
  28. package/examples/ui-restoration/README.i18n.yaml +6 -0
  29. package/examples/ui-restoration/README.md +70 -0
  30. package/examples/ui-restoration/README.zh.md +70 -0
  31. package/examples/ui-restoration/assets/final-heatmap.png +0 -0
  32. package/examples/ui-restoration/assets/final-report.json +83 -0
  33. package/examples/ui-restoration/assets/implementation.png +0 -0
  34. package/examples/ui-restoration/assets/initial-heatmap.png +0 -0
  35. package/examples/ui-restoration/assets/initial-report.json +83 -0
  36. package/examples/ui-restoration/assets/initial.png +0 -0
  37. package/examples/ui-restoration/assets/metrics.json +12 -0
  38. package/examples/ui-restoration/assets/reference.png +0 -0
  39. package/examples/ui-restoration/implementation.html +94 -0
  40. package/examples/ui-restoration/initial.html +57 -0
  41. package/lib/artifact-access.js +369 -0
  42. package/lib/artifact-access.js.map +1 -0
  43. package/lib/artifacts.js +56 -0
  44. package/lib/artifacts.js.map +1 -0
  45. package/lib/client.js +952 -0
  46. package/lib/client.js.map +1 -0
  47. package/lib/config.js +117 -0
  48. package/lib/config.js.map +1 -0
  49. package/lib/errors.js +56 -0
  50. package/lib/errors.js.map +1 -0
  51. package/lib/exposure.js +213 -0
  52. package/lib/exposure.js.map +1 -0
  53. package/lib/index.js +97 -0
  54. package/lib/index.js.map +1 -0
  55. package/lib/paste-images.js +199 -0
  56. package/lib/paste-images.js.map +1 -0
  57. package/lib/paths.js +325 -0
  58. package/lib/paths.js.map +1 -0
  59. package/lib/runtime-install.js +601 -0
  60. package/lib/runtime-install.js.map +1 -0
  61. package/lib/runtime-manager.js +126 -0
  62. package/lib/runtime-manager.js.map +1 -0
  63. package/lib/runtime.js +1344 -0
  64. package/lib/runtime.js.map +1 -0
  65. package/lib/skill.js +139 -0
  66. package/lib/skill.js.map +1 -0
  67. package/lib/tools.js +528 -0
  68. package/lib/tools.js.map +1 -0
  69. package/lib/types/artifact-access.d.ts +61 -0
  70. package/lib/types/artifact-access.d.ts.map +1 -0
  71. package/lib/types/artifacts.d.ts +42 -0
  72. package/lib/types/artifacts.d.ts.map +1 -0
  73. package/lib/types/client/index.d.ts +179 -0
  74. package/lib/types/client/index.d.ts.map +1 -0
  75. package/lib/types/client/paste-images.d.ts +57 -0
  76. package/lib/types/client/paste-images.d.ts.map +1 -0
  77. package/lib/types/config.d.ts +73 -0
  78. package/lib/types/config.d.ts.map +1 -0
  79. package/lib/types/errors.d.ts +35 -0
  80. package/lib/types/errors.d.ts.map +1 -0
  81. package/lib/types/exposure.d.ts +40 -0
  82. package/lib/types/exposure.d.ts.map +1 -0
  83. package/lib/types/index.d.ts +18 -0
  84. package/lib/types/index.d.ts.map +1 -0
  85. package/lib/types/paste-images.d.ts +21 -0
  86. package/lib/types/paste-images.d.ts.map +1 -0
  87. package/lib/types/paths.d.ts +107 -0
  88. package/lib/types/paths.d.ts.map +1 -0
  89. package/lib/types/runtime-install.d.ts +49 -0
  90. package/lib/types/runtime-install.d.ts.map +1 -0
  91. package/lib/types/runtime-manager.d.ts +60 -0
  92. package/lib/types/runtime-manager.d.ts.map +1 -0
  93. package/lib/types/runtime.d.ts +389 -0
  94. package/lib/types/runtime.d.ts.map +1 -0
  95. package/lib/types/skill.d.ts +15 -0
  96. package/lib/types/skill.d.ts.map +1 -0
  97. package/lib/types/tools.d.ts +22 -0
  98. package/lib/types/tools.d.ts.map +1 -0
  99. package/lib/types/upstream.d.ts +207 -0
  100. package/lib/types/upstream.d.ts.map +1 -0
  101. package/lib/types/version.d.ts +15 -0
  102. package/lib/types/version.d.ts.map +1 -0
  103. package/lib/types/web-request.d.ts +4 -0
  104. package/lib/types/web-request.d.ts.map +1 -0
  105. package/lib/types/web.d.ts +74 -0
  106. package/lib/types/web.d.ts.map +1 -0
  107. package/lib/upstream.js +675 -0
  108. package/lib/upstream.js.map +1 -0
  109. package/lib/version.js +18 -0
  110. package/lib/version.js.map +1 -0
  111. package/lib/web-request.js +20 -0
  112. package/lib/web-request.js.map +1 -0
  113. package/lib/web.js +244 -0
  114. package/lib/web.js.map +1 -0
  115. package/package.json +139 -0
  116. package/runtime/requirements.lock +3 -0
  117. package/src/artifact-access.ts +386 -0
  118. package/src/artifacts.ts +85 -0
  119. package/src/client/index.tsx +866 -0
  120. package/src/client/paste-images.tsx +426 -0
  121. package/src/config.ts +177 -0
  122. package/src/errors.ts +62 -0
  123. package/src/exposure.ts +227 -0
  124. package/src/index.ts +122 -0
  125. package/src/paste-images.ts +234 -0
  126. package/src/paths.ts +348 -0
  127. package/src/runtime-install.ts +723 -0
  128. package/src/runtime-manager.ts +166 -0
  129. package/src/runtime.ts +1783 -0
  130. package/src/skill.ts +143 -0
  131. package/src/tools.ts +668 -0
  132. package/src/upstream.ts +861 -0
  133. package/src/version.ts +37 -0
  134. package/src/web-request.ts +17 -0
  135. package/src/web.ts +329 -0
  136. package/vendor/agent-vision-toolkit/CHANGELOG.md +16 -0
  137. package/vendor/agent-vision-toolkit/LICENSE +21 -0
  138. package/vendor/agent-vision-toolkit/README.md +399 -0
  139. package/vendor/agent-vision-toolkit/UPSTREAM_MANIFEST.json +89 -0
  140. package/vendor/agent-vision-toolkit/bin/crop +90 -0
  141. package/vendor/agent-vision-toolkit/bin/detect +13 -0
  142. package/vendor/agent-vision-toolkit/bin/glance +93 -0
  143. package/vendor/agent-vision-toolkit/bin/ground +13 -0
  144. package/vendor/agent-vision-toolkit/bin/trace +129 -0
  145. package/vendor/agent-vision-toolkit/detect.py +56 -0
  146. package/vendor/agent-vision-toolkit/ground.py +216 -0
  147. package/vendor/agent-vision-toolkit/skills/vision-tools/scripts/dominant_colors.py +224 -0
  148. package/vendor/agent-vision-toolkit/skills/vision-tools/scripts/extract_fg.py +278 -0
  149. package/vendor/agent-vision-toolkit/skills/vision-tools/scripts/html_shot.py +108 -0
  150. package/vendor/agent-vision-toolkit/skills/vision-tools/scripts/long_screenshot_ocr.py +1245 -0
  151. package/vendor/agent-vision-toolkit/skills/vision-tools/scripts/pixel_diff.py +88 -0
  152. package/vendor/agent-vision-toolkit/vision_client.py +156 -0
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Anionex
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,6 @@
1
+ # Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
2
+ # side as of the last confirmed-consistent state. Both languages carry equal authority;
3
+ # after editing either side, bring the other along and re-record with:
4
+ # pnpm run verify-translation-pairing --write README.md
5
+ README.md: 1b6de29f6b3b5e7373014301e2c33600f07e30ee
6
+ README.zh.md: c97d447ebfa8b776820069270b79d3db09c9daa5
package/README.md ADDED
@@ -0,0 +1,383 @@
1
+ ![DSH Vision Toolkit — native visual engineering for text-only DeepSeek Harness agents](assets/hero.png)
2
+
3
+ # DSH Vision Toolkit
4
+
5
+ [![X (Twitter)](https://img.shields.io/badge/-@anion__ex-000000?style=flat-square&logo=x&logoColor=white)](https://x.com/anion_ex)
6
+ [![Release v0.1.4](https://img.shields.io/badge/release-v0.1.4-5B4CF0?style=flat-square)](https://github.com/Anionex/dsh-vision-toolkit/releases/tag/v0.1.4)
7
+ [![Verified: 136 tests](https://img.shields.io/badge/verified-136%20tests-2EA44F?style=flat-square)](tests)
8
+ [![License: MIT](https://img.shields.io/badge/license-MIT-0B7285?style=flat-square)](LICENSE)
9
+ [![Node.js](https://img.shields.io/badge/Node.js-%5E22.19%20%7C%20%3E%3D24-339933?style=flat-square&logo=nodedotjs&logoColor=white)](package.json)
10
+ [![Python](https://img.shields.io/badge/Python-3.11%2B-3776AB?style=flat-square&logo=python&logoColor=white)](runtime/requirements.lock)
11
+ [![DSH profiles](https://img.shields.io/badge/DSH-Web%20%2B%20Headless-5B4CF0?style=flat-square)](cordis.patch.yml)
12
+
13
+ **Install:** `dsh plugin --profile web add @anionex/dsh-vision-toolkit`
14
+
15
+ **DSH Vision Toolkit brings [`agent-vision-toolkit`](https://github.com/Anionex/agent-vision-toolkit) into DeepSeek Harness as a native Profile Bundle.**
16
+
17
+ Give text-only DSH agents eyes—and keep vision in the harness—with intent-aware image Q&A, OCR, original-pixel grounding, UI restoration, pixel verification, managed Artifacts, and Web Settings. Ten independent tools replace shell glue with structured schemas and Agent-scoped progressive exposure.
18
+
19
+ **Upstream toolkit:** [Anionex/agent-vision-toolkit](https://github.com/Anionex/agent-vision-toolkit) · **Project website:** [agent-vision.anionex.me](https://agent-vision.anionex.me)
20
+
21
+ English | [中文](README.zh.md)
22
+
23
+ ## Why this exists
24
+
25
+ `agent-vision-toolkit` treats vision as an Agent-callable capability rather than a property of the base model. Its method carries the reason for looking into the visual request, moves from the whole image to targeted regions, and verifies coordinates, colors, geometry, and differences with focused tools instead of accepting a generic description as evidence.
26
+
27
+ DSH Vision Toolkit preserves that method while replacing CLI installation and Bash argument construction with native schemas, DSH Credentials, lifecycle-managed runtime preparation, structured Session-log results, previewable Artifacts, dedicated Web cards, and Settings. The Agent loads one versioned Skill and receives the ten visual schemas only when the current task needs them.
28
+
29
+ The package delivers the committed P0 and P1 product scope. P2's stable `ctx.visionToolkit` service remains deliberately unpublished until an independent plugin becomes a real consumer; the internal runtime does not pretend that an unvalidated ecosystem API is stable.
30
+
31
+ ## Proven use cases from agent-vision-toolkit
32
+
33
+ The first two panels are official upstream reference runs from the same pinned `agent-vision-toolkit` lineage packaged by this bundle. The image Q&A and screenshot-guided debugging panel is a live DeepSeek Harness Web session, showing the same workflows through DSH. See the [asset provenance record](assets/upstream/README.md) for the upstream source images.
34
+
35
+ ### Infographic restoration: screenshot to editable HTML/CSS
36
+
37
+ <p align="center">
38
+ <img src="assets/upstream/infographic-reference.webp" width="49%" alt="Upstream reference screenshot of a three-stage model-training infographic." />
39
+ <img src="assets/upstream/infographic-result.webp" width="49%" alt="Upstream editable HTML and CSS reconstruction of the model-training infographic." />
40
+ </p>
41
+
42
+ *Left: source screenshot. Right: the editable HTML/CSS result from the upstream [infographic-restoration reference](https://github.com/Anionex/agent-vision-toolkit/blob/c27d1a300962b553c0884993c575cd3e819465ce/examples/infographic-restoration/how-is-the-model-trained.html).*
43
+
44
+ ### UI restoration: sketch to working interface
45
+
46
+ <p align="center">
47
+ <img src="assets/upstream/ui-sketch.webp" width="49%" alt="Upstream hand-drawn JupyterLab workspace used as a UI restoration reference." />
48
+ <img src="assets/upstream/ui-result.webp" width="49%" alt="Upstream JupyterLab-style working interface reconstructed from the hand-drawn reference." />
49
+ </p>
50
+
51
+ *Left: hand-drawn input. Right: the upstream reconstructed interface; the complete method lives in the [UI restoration playbook](https://github.com/Anionex/agent-vision-toolkit/blob/c27d1a300962b553c0884993c575cd3e819465ce/skills/vision-tools/references/restore-ui.md).*
52
+
53
+ ### Image Q&A and screenshot-guided debugging
54
+
55
+ <p align="center">
56
+ <img src="assets/dsh-conversation-image-qa.png" width="49%" alt="DSH Web session in which a text-only agent answers a focused question about a UI reference image." />
57
+ <img src="assets/dsh-conversation-screenshot-debugging.png" width="49%" alt="DSH Web session in which the agent uses a screenshot comparison to diagnose mismatched UI fields and recommend vision_pixel_diff." />
58
+ </p>
59
+
60
+ *Left: intent-aware image Q&A in DSH Web. Right: a DSH Web screenshot-debugging turn that lists the concrete UI differences and continues toward `vision_pixel_diff`. The upstream workflow source is the same [`agent-vision-toolkit` reference](https://github.com/Anionex/agent-vision-toolkit/blob/c27d1a300962b553c0884993c575cd3e819465ce/README.md#real-world-effects).*
61
+
62
+ DSH Vision Toolkit adds native tool schemas, versioned lifecycle, Credentials, structured Session results, Artifacts, Web presentation, Settings, and progressive exposure around these upstream capabilities. The next section is the reproducible proof executed and checked into this DSH repository.
63
+
64
+ ## DSH-native proof: reference-to-pixel verification
65
+
66
+ The checked-in UI-restoration workflow renders an intentionally inaccurate HTML implementation, measures a `6.04%` pixel difference across six non-zero regions, iterates, and reaches an exact `0%` difference against the reference at `1200 × 720`.
67
+
68
+ <p>
69
+ <img src="examples/ui-restoration/assets/initial.png" width="49%" alt="Initial UI restoration candidate before Vision Toolkit iteration, with measurable layout and styling differences from the reference." />
70
+ <img src="examples/ui-restoration/assets/implementation.png" width="49%" alt="Final UI restoration output reproduced by the checked-in workflow with zero pixel difference from the reference." />
71
+ </p>
72
+
73
+ | Verified surface | Evidence |
74
+ |---|---|
75
+ | Product scope | 10 independent visual tools, matching `vision-tools` Skill, Artifacts, dedicated Web cards, and live Settings |
76
+ | Automated coverage | 17 Vitest files / 136 passing tests, plus a dependency-free portable package check |
77
+ | Real profiles | Clean temporary Web and Headless installation, activation, disable, re-enable, and uninstall |
78
+ | Visual acceptance | Reproducible HTML screenshot → pixel diff example with a final `0%` difference |
79
+
80
+ ## Highlights
81
+
82
+ - **See images without bloating every prompt:** only `vision_toolkit_activate` is initially visible; loading `vision-tools` mounts ten independent schemas for that Agent and keeps version/health administration out of model context.
83
+ - **Act on coordinates instead of parsing prose:** grounding and detection return original-image pixel boxes, while every model-visible result remains structured text or JSON.
84
+ - **Deliver files, not temporary output:** crop, trace, OCR, pixel diff, foreground extraction, and HTML rendering produce described Artifacts that the Web client can preview, download, or open locally.
85
+ - **Keep runtime and credentials controlled:** DSH Credentials hold API keys, managed mode prepares an exact isolated Python environment, and a failed Settings candidate cannot replace the serving generation.
86
+ - **Close the visual verification loop:** local HTML rendering and pixel-diff ranking support reference → implementation → screenshot → measured iteration without a model-native image channel.
87
+ - **Use the same bundle in Web and Headless profiles:** Web adds cards, previews, Settings, and health actions; Headless receives the same tool semantics and complete structured results.
88
+
89
+ ## Quick start
90
+
91
+ Prerequisites: DeepSeek Harness `0.1.0-rc.6` or a compatible later `0.1.x` release, Python 3.11+, and `pnpm` available to `dsh plugin`. Install the published bundle from npm, add it to the profiles you use, and confirm the bundle row:
92
+
93
+ ```sh
94
+ dsh plugin --profile web add @anionex/dsh-vision-toolkit
95
+ dsh plugin --profile headless add @anionex/dsh-vision-toolkit
96
+ dsh --profile web --dump-config | grep vision-toolkit
97
+ dsh --profile headless --dump-config | grep vision-toolkit
98
+ ```
99
+
100
+ Legacy profiles must use `nodeLinker: hoisted` and `autoInstallPeers: false` in their `pnpm-workspace.yaml`. An updated DSH launcher repairs these owned settings before `dsh plugin` runs; when using an older launcher, set them before installation so pnpm does not assemble a second Harness dependency graph inside the profile.
101
+
102
+ Restart a running Web profile, open **Settings → Vision Toolkit**, select a DSH Credential for remote tools, and explicitly run **Test connection**. In a conversation, make an image available as a workspace path, invoke `/vision-tools`, and ask the Agent to call a specific `vision_*` tool. Local crop, trace, pixel, color, foreground, and HTML operations do not require a visual API credential.
103
+
104
+ ## How it works
105
+
106
+ ```mermaid
107
+ flowchart LR
108
+ User["Workspace image or local HTML"] --> Skill["vision-tools Skill"]
109
+ Skill --> Activate["Agent-scoped activation"]
110
+ Activate --> Tools["10 independent vision_* tools"]
111
+ Tools --> Runtime["Shared VisionToolkitRuntime"]
112
+ Credentials["DSH Credentials"] --> Runtime
113
+ Settings["Web Settings and health"] --> Runtime
114
+ Runtime --> Upstream["Pinned agent-vision-toolkit"]
115
+ Runtime --> Remote["Configured vision API"]
116
+ Upstream --> Result["Text, coordinates, JSON"]
117
+ Remote --> Result
118
+ Runtime --> Artifacts["Workspace Artifacts"]
119
+ Result --> Session["Reconstructable Session log"]
120
+ Artifacts --> Web["Preview, download, or open file"]
121
+ ```
122
+
123
+ Tool definitions call one runtime; the runtime validates paths, limits, credentials, cancellation, and deadlines before dispatching to the pinned upstream snapshot or configured OpenAI-compatible vision endpoint. Web presentation consumes the same structured results and Artifact descriptors, so it does not change Headless behavior. Health, connection testing, and version inspection stay in Settings rather than model tool schemas.
124
+
125
+ ## Tools
126
+
127
+ | Tool | Execution | Structured result | Artifact delivery |
128
+ |---|---|---|---|
129
+ | `vision_glance` | Remote vision API | Description, targeted answer, OCR, or multi-image comparison | None |
130
+ | `vision_ground` | Remote vision API; optional local preview | Target, original-image dimensions, and pixel boxes | Optional labeled PNG |
131
+ | `vision_detect` | Remote vision API; optional local preview | Numbered element inventory and original-image pixel boxes | Optional numbered PNG |
132
+ | `vision_trace` | Local pinned vtracer pipeline | SVG geometry status, path count, scale, and size | SVG |
133
+ | `vision_crop` | Local Pillow pipeline | Applied pixel box, dimensions, format, and clamp status | PNG or JPEG |
134
+ | `vision_pixel_diff` | Local NumPy/Pillow pipeline | Difference percentage and ranked grid regions | PNG heatmap and JSON report |
135
+ | `vision_long_screenshot_ocr` | Local split/audit; remote OCR unless `splitOnly=true` | Chunk boundaries, reuse state, completion state, and run directory | Markdown, manifest, boundary audit, chunk PNGs, and OCR sidecars |
136
+ | `vision_extract_foreground` | Local pinned extraction pipeline | Selected box, component counts, foreground coverage, and dimensions | Transparent PNG |
137
+ | `vision_dominant_colors` | Local pinned color analysis | Extracted palette or pixel-backed candidate ranking | None |
138
+ | `vision_html_screenshot` | Local Chrome/Chromium/Edge adapter | Authorized source facts, viewport, and rendered dimensions | PNG |
139
+
140
+ The plugin does not reimplement visual algorithms. Its DSH-owned layer validates paths and limits, resolves credentials, calls the pinned upstream scripts with argv vectors, parses their exact output contracts, classifies failures, describes files, and projects results to the model and Web client.
141
+
142
+ ## Progressive model exposure
143
+
144
+ Runtime readiness is profile-wide, but the ten visual execution schemas are Agent-scoped. Before an Agent loads `vision-tools`, the plugin contributes only the small `vision_toolkit_activate` bootstrap; the visual tools are absent from that Agent's request schema. A successful call to the standard `skill` tool with `name="vision-tools"` mounts all ten tools automatically for the next model step and hides the bootstrap. A direct `/vision-tools` invocation injects the Skill instructions; if the visual tools are still absent, those instructions require one `vision_toolkit_activate` call. Activation affects only that Agent, restores when the Session contains durable evidence matching the bundled Skill version, and lasts until the Agent or plugin is disposed.
145
+
146
+ Health checks, connection testing, and plugin/upstream version inspection are administrative Web Settings operations. `vision_toolkit_health` and `vision_toolkit_version` are not model tools and never enter an Agent's schema, including after visual-tool activation.
147
+
148
+ ## Requirements
149
+
150
+ - DeepSeek Harness with a Web or Headless profile and `pnpm` available to `dsh plugin`.
151
+ - Python 3.11 or newer. Managed mode creates an isolated environment, so users do not install the upstream CLI or Python packages manually.
152
+ - Network access on the first managed-runtime activation unless the exact packages in `runtime/requirements.lock` are already available in the configured package cache.
153
+ - An OpenAI-compatible vision endpoint and DSH Credential for `vision_glance`, `vision_ground`, `vision_detect`, and non-split-only long-screenshot OCR. Local tools remain usable without that credential.
154
+ - Chrome, Chromium, or Edge only for `vision_html_screenshot`; all other tools remain available when no supported browser is installed.
155
+ - PNG, JPEG, GIF, or WebP inputs inside the session workspace or an explicitly configured `allowedDirs` root.
156
+
157
+ ## Install and lifecycle
158
+
159
+ ### Install
160
+
161
+ Install the bundle into each profile that should expose it:
162
+
163
+ ```sh
164
+ dsh plugin --profile web add @anionex/dsh-vision-toolkit
165
+ dsh plugin --profile headless add @anionex/dsh-vision-toolkit
166
+ dsh --profile web --dump-config | grep vision-toolkit
167
+ dsh --profile headless --dump-config | grep vision-toolkit
168
+ ```
169
+
170
+ Restart a long-lived Web profile after installation. The host discovers the built browser bundle from `package.json`'s `dsh.client` declaration at process startup; the legacy top-level `dshClient` field is not scanned.
171
+
172
+ The first managed start verifies the packaged upstream manifest and atomically prepares an isolated environment under `DSH_HOME/cache/dsh-vision-toolkit`. Only after preparation succeeds does the plugin publish the same-version `vision-tools` Skill and activation bootstrap; each Agent receives the execution tools only after loading that Skill. An initial preparation failure leaves the Web Settings repair surface available but exposes neither model capability nor a misleading Skill.
173
+
174
+ ### Disable and re-enable
175
+
176
+ Set the bundle row to `disabled: true` in a profile patch or overlay:
177
+
178
+ ```yaml
179
+ - id: vision-toolkit
180
+ disabled: true
181
+ ```
182
+
183
+ Remove the flag or set it to `false` to re-enable the plugin. Disposal first cancels plugin-owned visual operations, then removes every Agent-scoped tool, the bootstrap, and the Skill; reactivation prepares the configured runtime before any model capability becomes visible. User configuration and completed Artifacts remain intact.
184
+
185
+ ### Upgrade
186
+
187
+ For a registry installation, update the dependency through the profile package manager:
188
+
189
+ ```sh
190
+ dsh plugin --profile web update @anionex/dsh-vision-toolkit
191
+ dsh plugin --profile headless update @anionex/dsh-vision-toolkit
192
+ ```
193
+
194
+ For a local path installation, run `add` again against the replacement checkout or tarball. Settings remain in the profile's Settings provider. A candidate runtime is fully validated and prepared before it is persisted and made active; a failed or obsolete concurrent candidate cannot replace the current serving generation.
195
+
196
+ ### Uninstall
197
+
198
+ ```sh
199
+ dsh plugin --profile web remove @anionex/dsh-vision-toolkit
200
+ dsh plugin --profile headless remove @anionex/dsh-vision-toolkit
201
+ ```
202
+
203
+ `dsh plugin remove` removes both the dependency and its bundle layer. The profile no longer exposes the activation bootstrap, Agent-scoped Vision Toolkit tools, or Skill entries. Managed cache data may be deleted separately when no profile uses the package; it is not active configuration and cannot register anything by itself.
204
+
205
+ ## Configure
206
+
207
+ The bundle defaults to the managed runtime. A profile patch can override the provider and limits:
208
+
209
+ ```yaml
210
+ - id: vision-toolkit
211
+ config:
212
+ provider:
213
+ baseUrl: https://api.inferera.com/v1
214
+ credential: VISION_API_KEY
215
+ model: gemini-3.6-flash
216
+ language: zh
217
+ timeoutMs: 60000
218
+ maxImageBytes: 10485760
219
+ maxImagePixels: 40000000
220
+ concurrency: 4
221
+ runtime:
222
+ mode: managed
223
+ allowedDirs: []
224
+ ```
225
+
226
+ ### Configuration fields
227
+
228
+ | Field | Default | Contract |
229
+ |---|---|---|
230
+ | `provider.baseUrl` | `https://api.inferera.com/v1` | OpenAI-compatible base URL; normalized without trailing slashes |
231
+ | `provider.credential` | `VISION_API_KEY` | DSH Credential reference, never a secret value |
232
+ | `provider.model` | `gemini-3.6-flash` | Multimodal model name sent to remote tools |
233
+ | `language` | `zh` | Vision output language: `zh` or `en` |
234
+ | `timeoutMs` | `60000` | Whole-operation deadline, 1000-600000 ms; each tool may request a narrower override |
235
+ | `maxImageBytes` | `10485760` | Encoded-byte limit per input image |
236
+ | `maxImagePixels` | `40000000` | Decoded-pixel limit per input image |
237
+ | `concurrency` | `4` | In-flight operations per session, 1-16 |
238
+ | `runtime.mode` | `managed` | `managed` uses the packaged snapshot; `external` accepts only the exact pin |
239
+ | `runtime.agentVisionToolkitPath` | unset | Required in `external` mode; exported exact snapshot or clean pinned Git checkout |
240
+ | `runtime.python` | unset | Optional Python 3.11+ bootstrap/interpreter override |
241
+ | `allowedDirs` | `[]` | Additional realpath-resolved input roots; the session workspace is always allowed |
242
+
243
+ ### Credentials
244
+
245
+ Create or replace the referenced secret through DSH Credentials:
246
+
247
+ ```sh
248
+ dsh credentials set VISION_API_KEY
249
+ ```
250
+
251
+ The reference is stored in Settings; the value is not. Remote operations resolve it once per call and inject it only into that subprocess environment. The plugin excludes user `.env` files, checkout `.env` files, `PYTHONPATH`, `PYTHONHOME`, `VIRTUAL_ENV`, and user site-packages so ambient Python or upstream configuration cannot override the selected DSH provider. Logs, errors, tool results, Artifact metadata, and Settings responses never contain the secret.
252
+
253
+ ### Managed and external runtimes
254
+
255
+ Managed mode verifies `vendor/agent-vision-toolkit/UPSTREAM_MANIFEST.json`, prefers `uv`, falls back to `venv` plus pip, installs exact versions from `runtime/requirements.lock`, coordinates concurrent preparation with a heartbeat lock, and publishes a staged environment only after all probes pass.
256
+
257
+ External mode is intended for development or controlled deployments:
258
+
259
+ ```yaml
260
+ - id: vision-toolkit
261
+ config:
262
+ runtime:
263
+ mode: external
264
+ agentVisionToolkitPath: /opt/agent-vision-toolkit
265
+ python: python3.12
266
+ ```
267
+
268
+ The path must be an exported copy matching the packaged manifest or the root of a clean Git checkout at `c27d1a300962b553c0884993c575cd3e819465ce`. Modified tracked files and untracked files are rejected because they can change or shadow the pinned Python behavior.
269
+
270
+ ## Web Settings
271
+
272
+ The Web profile registers a Vision Toolkit Settings section for the provider URL, Credential reference, model, language, timeout, byte/pixel limits, concurrency, runtime mode, Python override, external source path, and allowed directories. It also shows plugin/upstream versions, the active runtime generation, non-secret Credential configured/source/writable facts, runtime paths, health results, and Artifact-route availability.
273
+
274
+ `Save and apply` validates the complete value, prepares the candidate Python/upstream runtime, commits the Settings revision, and only then atomically switches generations. A rejected candidate leaves the previous generation serving and is reported separately from a genuinely unavailable runtime. `Reload` always restores the authoritative saved value, even when its revision did not change, so a rejected browser draft is discarded. If initial startup cannot prepare a runtime, the Settings route remains available so a valid configuration can make the first generation operational. A stale browser revision receives a conflict instead of overwriting a newer save; reload before retrying. A read-only Settings provider allows inspection and health checks but disables saves.
275
+
276
+ `Run health check` performs local checks only. `Test connection` is an explicit action that sends the configured Credential to `GET /models`; it uploads no image and creates no completion. Plugin load and ordinary Settings reads never make that request.
277
+
278
+ Health, connection testing, and plugin/upstream version inspection are administrative Web Settings capabilities rather than model-facing tools, so their schemas never occupy an agent request.
279
+
280
+ ## Artifacts and presentation
281
+
282
+ Artifact-producing tools write only under `<workspace>/.dsh-vision-toolkit/artifacts`, either as one validated file or an atomically committed run directory. Each model-visible descriptor contains the path, filename, MIME type, kind, description, source tool, preview intent, and byte size, so Headless agents can reuse the path in later calls without browser support. Before a traced SVG is committed, the runtime parses it as XML: standard declarations and comments are accepted, while doctypes, malformed or multi-root documents, a non-SVG namespace, and reported path/byte mismatches are rejected.
283
+
284
+ When the Web HTTP host is present, presentation-only metadata adds signed capability URLs for preview and download without altering the canonical tool result. Every read revalidates the signature, managed-root fence, path components, regular-file status, size, device/inode identity where available, extension, and MIME. SVG responses use a sandboxed no-resource CSP and the client renders them in a sandboxed iframe. Without an HTTP host, the same cards retain `Open file` through `openFile` and show the descriptor instead of inventing an inaccessible URL.
285
+
286
+ ## Usage patterns
287
+
288
+ ### Basic calls
289
+
290
+ ```text
291
+ vision_glance images=["screenshot.png"] query="What error is shown?"
292
+ vision_ground image="screenshot.png" target="the send button" preview=true
293
+ vision_detect image="screenshot.png" category="buttons" preview=true
294
+ vision_crop image="screenshot.png" region="1067,841,1108,881"
295
+ vision_trace image="icon.png" color=true output="icon.svg"
296
+ vision_pixel_diff original="reference.png" rebuilt="actual.png" runName="comparison"
297
+ vision_long_screenshot_ocr image="page.png" mode="general" jobs=2
298
+ vision_extract_foreground image="logo.png" mode="color"
299
+ vision_dominant_colors image="screen.png" region="0,0,600,300" top=8
300
+ vision_html_screenshot source="implementation.html" width=1200 height=720
301
+ ```
302
+
303
+ Common workflows are `vision_ground` → `vision_crop` → `vision_glance`, `vision_ground` → `vision_crop` → `vision_trace`, and reference image → `vision_html_screenshot` → `vision_pixel_diff`. Grounding and detection boxes always use original-image pixels (`x1/y1/x2/y2`).
304
+
305
+ ### UI restoration example
306
+
307
+ The checked-in [UI restoration example](examples/ui-restoration/README.md) renders a reference, an intentionally inaccurate first implementation, and the final implementation through `vision_html_screenshot`, then compares both candidates through `vision_pixel_diff`:
308
+
309
+ ```sh
310
+ npm run example:ui-restoration
311
+ npm run example:ui-restoration:write
312
+ ```
313
+
314
+ The committed evidence records an initial `6.04%` difference across six non-zero worst regions and a final `0%` difference with no non-zero worst region. Check mode reproduces the tool path and verifies the committed assets; write mode intentionally refreshes the evidence.
315
+
316
+ ## Security and execution model
317
+
318
+ - Inputs resolve against the session workspace and configured `allowedDirs`; realpath containment prevents traversal and symlink escape.
319
+ - Pillow decodes every image before a remote request and verifies bytes, pixels, dimensions, and extension/content agreement. Unsupported or oversized images fail before upload.
320
+ - Outputs use random staging files or directories inside the real managed destination, reject symbolic links, and commit only after format and contract validation.
321
+ - Remote vision prompts explicitly classify text and instructions visible inside images as untrusted content. The native tool descriptions and bundled skill likewise tell the text agent to treat derived descriptions, labels, and OCR as visual evidence rather than executable instructions.
322
+ - All upstream processes use argv vectors through `ctx.subprocess`, inherit caller cancellation, share one hard operation deadline, and terminate with the operation instead of continuing in the background. Plugin disposal aborts active calls before unregistering their tools.
323
+ - One live Session retains only the most recent successful `vision_glance` result. An immediate repeat reuses it only when image content, query/OCR mode, region, endpoint, model, language, and Credential are unchanged; failures and other Sessions never share the entry.
324
+ - Model-visible data is text, numbers, coordinates, structured JSON, and file descriptors. Tool calls/results remain reconstructable from the Session log; browser previews are presentation metadata only.
325
+ - Metrics include tool name, total/upstream duration, bounded image counts/bytes/pixels, cache hits, model, and error category; they exclude base64, authentication headers, secrets, and unbounded upstream output.
326
+
327
+ `vision_html_screenshot` accepts only authorized local `.html` or `.htm` files, disables network access in the pinned adapter, and launches a Chrome-family browser with `--headless=new`, `--use-mock-keychain`, `--incognito`, and a unique `--user-data-dir` under the system temporary directory. The profile is removed after every call, so headless rendering does not touch the user's daily Chrome profile or macOS login keychain.
328
+
329
+ ## Troubleshooting
330
+
331
+ | Symptom | Resolution |
332
+ |---|---|
333
+ | `Model "..." does not support image input. (attachment-error)` | The image used DSH's native model-attachment channel, so a text-only model rejected the turn before the Skill or Vision Toolkit could run. Use DSH Paste Input's attachment button, paste, or drop flow so the file is copied into the session workspace and represented by a path, then invoke `/vision-tools`. Restart the Web profile and reload the page after installing or upgrading either browser plugin. |
334
+ | Credential reported missing | Run `dsh credentials set <REF>`, ensure `provider.credential` names that reference, then rerun health. Local-only tools do not need it. |
335
+ | Runtime preparation fails | Read the Settings runtime error, verify Python 3.11+, package-cache/network access, disk permissions, and the exact external pin. Save only after correcting the candidate; the active generation remains intact. |
336
+ | Chrome is not found | Install Chrome, Chromium, or Edge or configure an environment where one is discoverable. Only `vision_html_screenshot` is unavailable. |
337
+ | macOS displays a keychain dialog | Confirm the current built adapter is installed and no stale external `html_shot`/headless Chrome process is running. Current launches use a mock keychain and disposable profile; cancel the dialog rather than resetting the login keychain. |
338
+ | Input or output path is rejected | Move the file into the session workspace or add an intentional real directory to `allowedDirs`; remove escaping symlinks. Outputs accept a filename, not an absolute or nested path. |
339
+ | Vision service returns 401/403 | Replace the Credential value or select the correct reference and endpoint. Errors remain redacted. |
340
+ | Vision service returns 429 | Retry after the provider's rate-limit window or lower `concurrency`. The plugin does not silently switch providers. |
341
+ | Operation times out or is cancelled | Raise `timeoutMs` within 1000-600000 ms, reduce image/chunk work, or rerun after cancellation. The subprocess/request is stopped with the operation. |
342
+ | Settings save reports a conflict | Reload the section to obtain the current revision, reapply the intended edit, and save again. |
343
+ | Settings is read-only | Change the active Settings provider or edit the owning profile configuration; the plugin cannot bypass provider writability. |
344
+ | Artifact preview is unavailable | Use `Open file` or the model-visible path. Preview/download URLs exist only while a Web HTTP route is attached. |
345
+
346
+ ## Development and verification
347
+
348
+ ```sh
349
+ pnpm install --frozen-lockfile --trust-lockfile
350
+ pnpm run verify:portable
351
+ pnpm run build
352
+ pnpm test
353
+ pnpm run example:ui-restoration
354
+ pnpm pack --dry-run
355
+ ```
356
+
357
+ `pnpm run verify:portable` is the dependency-free portable verification gate: it validates the vendored snapshot, package metadata and exports, committed JavaScript syntax, README links and images, required facade files, social-preview dimensions, and the dry-run tarball. The full TypeScript build and test suite run from this standalone checkout against the locked DSH `0.1.0-rc.6` registry packages; the client build also has a separate compiler face that resolves the packages' public exports without internal path aliases. The real Profile acceptance runs when compatible `dsh` and `pnpm` commands are on PATH, and CI requires that path instead of silently skipping it.
358
+
359
+ `pnpm run build` verifies the vendored manifest before emitting JavaScript, declarations, and the loader-compatible Web client. The package commits `lib/`, so installation from a checkout does not require a consumer-side build. The keyless real-profile test installs into a clean `DSH_HOME`, boots Headless, executes all five P0 tools plus representative P1 local/remote tools through real tool calls, verifies disable and re-enable behavior, and uninstalls the bundle. See the [requirements traceability reference](docs/requirements-traceability/README.md) for the implementation and verification home of every P0/P1 requirement.
360
+
361
+ Update the upstream snapshot only through `pnpm run upstream:sync -- <checkout>`, inspect the source and license, regenerate the manifest, and update the adapter compatibility tests and committed `lib/` in the same change. The runtime never fetches upstream `main`.
362
+
363
+ ## Project status and scope
364
+
365
+ Version `0.1.4` is the current public npm release. P0 and P1 are product commitments in this package. P2 is a design threshold: no stable `ctx.visionToolkit` service, capability-discovery API, or provider ecosystem is published until at least one independent plugin consumes the internal capability shape. Web upload, drag-and-drop, camera/video/audio/document ingestion, interactive box editing, automatic GUI clicking, service clusters, model routing, model voting, and cross-session vision caches remain outside the current product.
366
+
367
+ ## Community and About
368
+
369
+ - Read [CONTRIBUTING.md](CONTRIBUTING.md) before proposing code, protocol, or upstream-snapshot changes.
370
+ - Use [GitHub Issues](https://github.com/Anionex/dsh-vision-toolkit/issues) for reproducible bugs, focused feature requests, and usage questions; use [SUPPORT.md](SUPPORT.md) to choose the right channel.
371
+ - Report vulnerabilities privately through the process in [SECURITY.md](SECURITY.md), never in a public issue.
372
+ - Follow releases and compatibility notes in [CHANGELOG.md](CHANGELOG.md).
373
+ - Optional sponsorship is described transparently in [FUNDING.md](FUNDING.md); support does not purchase roadmap priority or private support.
374
+ - Use the upstream [project website](https://agent-vision.anionex.me) and [repository](https://github.com/Anionex/agent-vision-toolkit) for the general toolkit, cross-harness integrations, visual-task playbooks, and reference runs.
375
+ - Star, share, contribute to, or sponsor `agent-vision-toolkit` if its algorithms or methods save time; DSH-specific bugs and integration requests belong in this repository.
376
+
377
+ [`agent-vision-toolkit`](https://github.com/Anionex/agent-vision-toolkit) was created by [Anionex](https://anionex.me/). This repository maintains its native DeepSeek Harness integration: DSH owns lifecycle, security, structured schemas, Credentials, Artifacts, and Web presentation, while the upstream project remains the home of the visual algorithms and reusable playbooks.
378
+
379
+ If you would like to follow my future work, [follow me on X](https://x.com/anion_ex) or [GitHub](https://github.com/Anionex).
380
+
381
+ ## License
382
+
383
+ The plugin is MIT-licensed. The packaged `agent-vision-toolkit` snapshot retains its upstream MIT license in `vendor/agent-vision-toolkit/LICENSE` and remains the sole implementation of its visual algorithms.