@mengruo/dsh-vision-toolkit 0.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (216) hide show
  1. package/LICENSE +21 -0
  2. package/README.i18n.yaml +6 -0
  3. package/README.md +356 -0
  4. package/README.zh.md +354 -0
  5. package/assets/community-group-qr.png +0 -0
  6. package/assets/dsh-conversation-artifact.png +0 -0
  7. package/assets/dsh-conversation-image-qa-top.png +0 -0
  8. package/assets/dsh-conversation-image-qa.png +0 -0
  9. package/assets/dsh-conversation-pixel-diff.png +0 -0
  10. package/assets/dsh-conversation-screenshot-debugging-top.png +0 -0
  11. package/assets/dsh-conversation-screenshot-debugging.png +0 -0
  12. package/assets/dsh-conversation-tool-call.png +0 -0
  13. package/assets/dsh-conversation-vision-trace.png +0 -0
  14. package/assets/dsh-view-example.png +0 -0
  15. package/assets/hero-v2.png +0 -0
  16. package/assets/logo_aihubmix.png +0 -0
  17. package/assets/python-bootstrap.json +48 -0
  18. package/assets/skill/SKILL.md +329 -0
  19. package/assets/skill/UPSTREAM.json +71 -0
  20. package/assets/skill/references/gui.md +88 -0
  21. package/assets/skill/references/long-screenshot-ocr.md +77 -0
  22. package/assets/skill/references/restore-graphic.md +84 -0
  23. package/assets/skill/references/restore-structure.md +45 -0
  24. package/assets/skill/references/restore-ui.md +202 -0
  25. package/assets/social-preview.png +0 -0
  26. package/assets/upstream/README.md +18 -0
  27. package/assets/upstream/focus-hint-comparison-1.webp +0 -0
  28. package/assets/upstream/focus-hint-comparison-2.webp +0 -0
  29. package/assets/upstream/infographic-reference.webp +0 -0
  30. package/assets/upstream/infographic-result.webp +0 -0
  31. package/assets/upstream/ui-fast-restore-reference.webp +0 -0
  32. package/assets/upstream/ui-fast-restore-result.webp +0 -0
  33. package/assets/upstream/ui-result.webp +0 -0
  34. package/assets/upstream/ui-sketch.webp +0 -0
  35. package/assets/vision-model-test.png +0 -0
  36. package/assets/vision-settings.png +0 -0
  37. package/assets/wechat-reward.png +0 -0
  38. package/cordis.patch.yml +6 -0
  39. package/docs/aihubmix-gemini-vision.i18n.yaml +6 -0
  40. package/docs/aihubmix-gemini-vision.md +138 -0
  41. package/docs/aihubmix-gemini-vision.zh.md +138 -0
  42. package/docs/assets/aihubmix-api-keys.png +0 -0
  43. package/docs/assets/aihubmix-create-key.png +0 -0
  44. package/docs/assets/aihubmix-free-vision-model.png +0 -0
  45. package/docs/assets/aihubmix-home.png +0 -0
  46. package/docs/assets/aihubmix-sign-up.png +0 -0
  47. package/docs/assets/groq-console-home.png +0 -0
  48. package/docs/assets/groq-console-keys-login.png +0 -0
  49. package/docs/assets/groq-docs-quickstart.png +0 -0
  50. package/docs/assets/groq-docs-qwen3.6.png +0 -0
  51. package/docs/assets/groq-docs-vision.png +0 -0
  52. package/docs/assets/vision-settings.png +0 -0
  53. package/docs/dsh-desktop-install.md +67 -0
  54. package/docs/dsh-desktop-install.zh.md +67 -0
  55. package/docs/groq-qwen3.6-vision.i18n.yaml +6 -0
  56. package/docs/groq-qwen3.6-vision.md +236 -0
  57. package/docs/groq-qwen3.6-vision.zh.md +236 -0
  58. package/docs/python-runtime.i18n.yaml +6 -0
  59. package/docs/python-runtime.md +89 -0
  60. package/docs/python-runtime.zh.md +89 -0
  61. package/docs/requirements-traceability/README.i18n.yaml +6 -0
  62. package/docs/requirements-traceability/README.md +75 -0
  63. package/docs/requirements-traceability/README.zh.md +75 -0
  64. package/examples/ui-restoration/README.i18n.yaml +6 -0
  65. package/examples/ui-restoration/README.md +70 -0
  66. package/examples/ui-restoration/README.zh.md +70 -0
  67. package/examples/ui-restoration/assets/final-heatmap.png +0 -0
  68. package/examples/ui-restoration/assets/final-report.json +83 -0
  69. package/examples/ui-restoration/assets/implementation.png +0 -0
  70. package/examples/ui-restoration/assets/initial-heatmap.png +0 -0
  71. package/examples/ui-restoration/assets/initial-report.json +83 -0
  72. package/examples/ui-restoration/assets/initial.png +0 -0
  73. package/examples/ui-restoration/assets/metrics.json +12 -0
  74. package/examples/ui-restoration/assets/reference.png +0 -0
  75. package/examples/ui-restoration/implementation.html +94 -0
  76. package/examples/ui-restoration/initial.html +57 -0
  77. package/lib/artifact-access.js +369 -0
  78. package/lib/artifact-access.js.map +1 -0
  79. package/lib/artifacts.js +56 -0
  80. package/lib/artifacts.js.map +1 -0
  81. package/lib/client.js +2099 -0
  82. package/lib/client.js.map +1 -0
  83. package/lib/config.js +172 -0
  84. package/lib/config.js.map +1 -0
  85. package/lib/defaults.js +6 -0
  86. package/lib/defaults.js.map +1 -0
  87. package/lib/errors.js +56 -0
  88. package/lib/errors.js.map +1 -0
  89. package/lib/evidence-cache.js +350 -0
  90. package/lib/evidence-cache.js.map +1 -0
  91. package/lib/exposure.js +247 -0
  92. package/lib/exposure.js.map +1 -0
  93. package/lib/image-input-variants.js +882 -0
  94. package/lib/image-input-variants.js.map +1 -0
  95. package/lib/index.js +105 -0
  96. package/lib/index.js.map +1 -0
  97. package/lib/paste-images.js +217 -0
  98. package/lib/paste-images.js.map +1 -0
  99. package/lib/paths.js +354 -0
  100. package/lib/paths.js.map +1 -0
  101. package/lib/plugin-update.js +1003 -0
  102. package/lib/plugin-update.js.map +1 -0
  103. package/lib/runtime-install.js +964 -0
  104. package/lib/runtime-install.js.map +1 -0
  105. package/lib/runtime-manager.js +133 -0
  106. package/lib/runtime-manager.js.map +1 -0
  107. package/lib/runtime.js +1678 -0
  108. package/lib/runtime.js.map +1 -0
  109. package/lib/skill.js +25 -0
  110. package/lib/skill.js.map +1 -0
  111. package/lib/tools.js +549 -0
  112. package/lib/tools.js.map +1 -0
  113. package/lib/types/artifact-access.d.ts +61 -0
  114. package/lib/types/artifact-access.d.ts.map +1 -0
  115. package/lib/types/artifacts.d.ts +42 -0
  116. package/lib/types/artifacts.d.ts.map +1 -0
  117. package/lib/types/client/display-config.d.ts +24 -0
  118. package/lib/types/client/display-config.d.ts.map +1 -0
  119. package/lib/types/client/index.d.ts +350 -0
  120. package/lib/types/client/index.d.ts.map +1 -0
  121. package/lib/types/client/model-variants-hider.d.ts +40 -0
  122. package/lib/types/client/model-variants-hider.d.ts.map +1 -0
  123. package/lib/types/client/paste-images.d.ts +126 -0
  124. package/lib/types/client/paste-images.d.ts.map +1 -0
  125. package/lib/types/config.d.ts +123 -0
  126. package/lib/types/config.d.ts.map +1 -0
  127. package/lib/types/defaults.d.ts +6 -0
  128. package/lib/types/defaults.d.ts.map +1 -0
  129. package/lib/types/errors.d.ts +35 -0
  130. package/lib/types/errors.d.ts.map +1 -0
  131. package/lib/types/evidence-cache.d.ts +91 -0
  132. package/lib/types/evidence-cache.d.ts.map +1 -0
  133. package/lib/types/exposure.d.ts +50 -0
  134. package/lib/types/exposure.d.ts.map +1 -0
  135. package/lib/types/image-input-variants.d.ts +144 -0
  136. package/lib/types/image-input-variants.d.ts.map +1 -0
  137. package/lib/types/index.d.ts +19 -0
  138. package/lib/types/index.d.ts.map +1 -0
  139. package/lib/types/paste-images.d.ts +73 -0
  140. package/lib/types/paste-images.d.ts.map +1 -0
  141. package/lib/types/paths.d.ts +117 -0
  142. package/lib/types/paths.d.ts.map +1 -0
  143. package/lib/types/plugin-update.d.ts +119 -0
  144. package/lib/types/plugin-update.d.ts.map +1 -0
  145. package/lib/types/runtime-install.d.ts +92 -0
  146. package/lib/types/runtime-install.d.ts.map +1 -0
  147. package/lib/types/runtime-manager.d.ts +60 -0
  148. package/lib/types/runtime-manager.d.ts.map +1 -0
  149. package/lib/types/runtime.d.ts +412 -0
  150. package/lib/types/runtime.d.ts.map +1 -0
  151. package/lib/types/skill.d.ts +14 -0
  152. package/lib/types/skill.d.ts.map +1 -0
  153. package/lib/types/tools.d.ts +35 -0
  154. package/lib/types/tools.d.ts.map +1 -0
  155. package/lib/types/upstream.d.ts +233 -0
  156. package/lib/types/upstream.d.ts.map +1 -0
  157. package/lib/types/version.d.ts +15 -0
  158. package/lib/types/version.d.ts.map +1 -0
  159. package/lib/types/web-request.d.ts +11 -0
  160. package/lib/types/web-request.d.ts.map +1 -0
  161. package/lib/types/web.d.ts +118 -0
  162. package/lib/types/web.d.ts.map +1 -0
  163. package/lib/upstream.js +860 -0
  164. package/lib/upstream.js.map +1 -0
  165. package/lib/version.js +18 -0
  166. package/lib/version.js.map +1 -0
  167. package/lib/web-request.js +29 -0
  168. package/lib/web-request.js.map +1 -0
  169. package/lib/web.js +462 -0
  170. package/lib/web.js.map +1 -0
  171. package/package.json +149 -0
  172. package/patches/vision-tools-dsh.patch +970 -0
  173. package/runtime/requirements.lock +3 -0
  174. package/src/artifact-access.ts +386 -0
  175. package/src/artifacts.ts +85 -0
  176. package/src/client/display-config.ts +62 -0
  177. package/src/client/index.tsx +1580 -0
  178. package/src/client/model-variants-hider.ts +159 -0
  179. package/src/client/paste-images.tsx +747 -0
  180. package/src/config.ts +290 -0
  181. package/src/defaults.ts +5 -0
  182. package/src/errors.ts +62 -0
  183. package/src/evidence-cache.ts +418 -0
  184. package/src/exposure.ts +265 -0
  185. package/src/image-input-variants.ts +996 -0
  186. package/src/index.ts +141 -0
  187. package/src/paste-images.ts +286 -0
  188. package/src/paths.ts +383 -0
  189. package/src/plugin-update.ts +1143 -0
  190. package/src/runtime-install.ts +1161 -0
  191. package/src/runtime-manager.ts +174 -0
  192. package/src/runtime.ts +2189 -0
  193. package/src/skill.ts +35 -0
  194. package/src/tools.ts +691 -0
  195. package/src/upstream.ts +1078 -0
  196. package/src/version.ts +37 -0
  197. package/src/web-request.ts +27 -0
  198. package/src/web.ts +594 -0
  199. package/vendor/agent-vision-toolkit/CHANGELOG.md +30 -0
  200. package/vendor/agent-vision-toolkit/LICENSE +21 -0
  201. package/vendor/agent-vision-toolkit/README.md +405 -0
  202. package/vendor/agent-vision-toolkit/UPSTREAM_MANIFEST.json +94 -0
  203. package/vendor/agent-vision-toolkit/bin/crop +90 -0
  204. package/vendor/agent-vision-toolkit/bin/detect +13 -0
  205. package/vendor/agent-vision-toolkit/bin/glance +93 -0
  206. package/vendor/agent-vision-toolkit/bin/ground +13 -0
  207. package/vendor/agent-vision-toolkit/bin/trace +129 -0
  208. package/vendor/agent-vision-toolkit/detect.py +60 -0
  209. package/vendor/agent-vision-toolkit/ground.py +267 -0
  210. package/vendor/agent-vision-toolkit/skills/vision-tools/scripts/dominant_colors.py +224 -0
  211. package/vendor/agent-vision-toolkit/skills/vision-tools/scripts/extract_fg.py +278 -0
  212. package/vendor/agent-vision-toolkit/skills/vision-tools/scripts/html_shot.py +420 -0
  213. package/vendor/agent-vision-toolkit/skills/vision-tools/scripts/long_screenshot_ocr.py +1245 -0
  214. package/vendor/agent-vision-toolkit/skills/vision-tools/scripts/pixel_diff.py +88 -0
  215. package/vendor/agent-vision-toolkit/tests/test_vision_client.py +450 -0
  216. package/vendor/agent-vision-toolkit/vision_client.py +310 -0
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Anionex
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,6 @@
1
+ # Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
2
+ # side as of the last confirmed-consistent state. Both languages carry equal authority;
3
+ # after editing either side, bring the other along and re-record with:
4
+ # pnpm run verify-translation-pairing --write dsh-vision-toolkit/README.md
5
+ README.md: f38885c9ef8b326808f9a43c8433e9e6a5c929db
6
+ README.zh.md: 06e3133487ea5465a8c81e04a1a98308217c8432
package/README.md ADDED
@@ -0,0 +1,356 @@
1
+ <p align="center">
2
+ <img src="assets/hero-v2.png" alt="DSH Vision Toolkit helps text-only DeepSeek Harness agents understand images and complete visual tasks" />
3
+ </p>
4
+
5
+ <div align="center">
6
+
7
+ # DSH Vision Toolkit
8
+
9
+ <a href="https://trendshift.io/repositories/149708?utm_source=trendshift-badge&amp;utm_medium=badge&amp;utm_campaign=badge-trendshift-149708" target="_blank" rel="noopener noreferrer"><img src="https://trendshift.io/api/badge/trendshift/repositories/149708/daily?language=TypeScript" alt="Anionex%2Fdsh-vision-toolkit | Trendshift" width="250" height="55"/></a>
10
+
11
+ [![Recommended by dshfind](https://img.shields.io/badge/recommended%20by-dshfind-FFD700?style=flat-square)](https://dshfind.com/en/plugins/Anionex/dsh-vision-toolkit)
12
+ [![dshfind score: 94 — highest-rated plugin](https://img.shields.io/badge/dshfind%20score-94%20%7C%20highest--rated%20plugin-5B4CF0?style=flat-square)](https://dshfind.com/en/plugins/Anionex/dsh-vision-toolkit)
13
+ [![agentic leaderboard](https://www.theagenticleaderboard.com/badges/new/dsh-vision-toolkit.svg)](https://www.theagenticleaderboard.com)
14
+
15
+ [![npm](https://img.shields.io/npm/v/@mengruo/dsh-vision-toolkit?style=flat-square&color=5B4CF0)](https://www.npmjs.com/package/@mengruo/dsh-vision-toolkit)
16
+ [![MIT](https://img.shields.io/badge/license-MIT-0B7285?style=flat-square)](LICENSE)
17
+ [![DSH](https://img.shields.io/badge/DSH-Web%20%2B%20Headless-5B4CF0?style=flat-square)](cordis.patch.yml)
18
+
19
+ **A more powerful vision toolkit—give text-only models in DeepSeek Harness eyes: image Q&A, long-screenshot OCR, UI restoration, and GUI visual tasks in one toolkit and Skill.**
20
+
21
+ 🚀 Paste an image and ask directly | Install with one command | Built-in free vision | Broad use cases
22
+
23
+ [Highlights](#highlights) | [Quick start](#quick-start-three-steps) | [Toolbox](#toolbox) | [Configuration and limits](#configuration-and-limits) | [Troubleshooting](#troubleshooting) | [Community](#development-and-community)
24
+
25
+ 🌐 **English** | [中文](README.zh.md)
26
+
27
+ </div>
28
+
29
+ 🏆 This project is the first comprehensive vision-tool plugin in the DeepSeek Harness ecosystem: it was initiated before internal beta and built during the beta with reference to [`agent-vision-toolkit`](https://github.com/Anionex/agent-vision-toolkit).
30
+
31
+ > **Original work:** The system and division of responsibilities behind these visual tools, together with the `vision-skills` Skill, were personally created and continuously refined by the author through long-term real-world use and repeated iteration.
32
+
33
+ > If this project helps you or gives you some inspiration, feel free to star 🌟 & fork.
34
+
35
+ ## Highlights
36
+
37
+ - **Paste an image and ask directly.** In DSH Web, pasting an image switches the text-only model to its `(Vision Toolkit)` variant automatically — no manual path copying or model changes. Native thumbnails, session history, and workspace paths stay intact; Web can preview artifacts.
38
+ - **One command to install.** The built-in free Gemini 3.7 Flash vision service is ready after installation, with no API key required.
39
+ - **Built-in free vision quota.** The shared service works immediately after installation with a quota of **100 images per machine per day**.
40
+ - **Not just a caption — the content that matters.** The model does not produce a generic description; it extracts evidence around the current task, such as “Where is the error?” or “Where is the button?”.
41
+ - **A battle-tested visual-task methodology.** The bundled Skill tells the agent what to look at for different visual tasks, which tool to choose, how to proceed, and how to verify the result.
42
+
43
+ [`agent-vision-toolkit`](https://github.com/Anionex/agent-vision-toolkit) gives an agent more than image captions: it can read, locate, crop, trace, rebuild, and verify visual work. DSH Vision Toolkit is its native DeepSeek Harness integration, bringing that workflow into Web and Headless Profiles.
44
+
45
+ This project has two layers:
46
+
47
+ 1. **Visual tools and a Skill:** the agent learns when to inspect, ground, OCR, crop, trace, or compare pixels.
48
+ 2. **Native DSH integration:** those capabilities live inside Profiles, sessions, Settings, Artifacts, and the Web UI, with a free Gemini 3.7 Flash vision service ready after installation.
49
+
50
+ > **Install and use it immediately.** The default setup includes a free Gemini 3.7 Flash vision service and requires no API key.
51
+
52
+ ```sh
53
+ dsh plugin --profile web add @mengruo/dsh-vision-toolkit
54
+ ```
55
+
56
+ **Upstream toolkit:** [Anionex/agent-vision-toolkit](https://github.com/Anionex/agent-vision-toolkit) · **Project website:** [agent-vision.anionex.me](https://agent-vision.anionex.me)
57
+
58
+ ## ❤️ Sponsor
59
+
60
+ > Want to sponsor this project? See [FUNDING.md](FUNDING.md) or email davidyang042@gmail.com.
61
+
62
+ <details open>
63
+ <summary>Click to collapse</summary>
64
+
65
+ <table>
66
+ <tr>
67
+ <td width="220"><a href="https://aihubmix.com/?aff=5wj6sgx8"><img src="assets/logo_aihubmix.png" alt="AIHubMix" height="48"></a></td>
68
+ <td>Thanks to <a href="https://aihubmix.com/?aff=5wj6sgx8">AIHubMix</a> for sponsoring this project! AIHubMix is a stable, high-concurrency AI model API gateway that connects Claude, GPT, Gemini, DeepSeek, and other mainstream models through a single API key, compatible with multiple protocols, with <b>free model options</b> available. To sign up, use the <a href="https://aihubmix.com/?aff=5wj6sgx8">AIHubMix entry</a> outside mainland China or the <a href="https://inferera.com/?aff=5wj6sgx8">Inferera entry</a> within mainland China.</td>
69
+ </tr>
70
+ </table>
71
+
72
+ </details>
73
+
74
+ **Contents**
75
+
76
+ - [Highlights](#highlights)
77
+ - [Recent updates](#recent-updates)
78
+ - [Who it is for](#who-it-is-for)
79
+ - [See it in action](#see-it-in-action)
80
+ - [Quick start: three steps](#quick-start-three-steps)
81
+ - [Toolbox](#toolbox)
82
+ - [Configuration and limits](#configuration-and-limits)
83
+ - [Troubleshooting](#troubleshooting)
84
+ - [Donation](#donation)
85
+ - [Development and community](#development-and-community)
86
+
87
+ ## Recent updates
88
+
89
+ - **2026-08-20 · AIHubMix setup guide:** Added a screenshot-based guide for getting an API key through the Inferera entry and using the free Gemini 3.7 Flash vision model; Settings now links directly to this guide.
90
+ - **2026-08-19 · Transparent routing by default:** The model selector keeps one entry per model with the original name, and image input (paste, history, `read_image`) works without manually switching to a `(Vision Toolkit)` variant. Disable “Transparent variant routing” in advanced settings → image input to restore the explicit entries.
91
+ - **2026-08-16 · Windows Python:** Added Microsoft Store Python support, fixing first-time isolated-runtime setup failures for affected Windows users.
92
+ - **2026-08-17 · Free vision upgrade:** Switched the built-in no-key service to Gemini 3.7 Flash and fixed Qwen/Gemini bounding-box coordinate order.
93
+ - **2026-08-16 · Image paste:** Text-only routes now switch to a `(Vision Toolkit)` variant and keep a workspace path, fixing blocked pastes and images that could not be reused later.
94
+ - **2026-08-16 · More shared capacity:** Expanded the free service capacity to reduce peak-time `429` responses.
95
+ - **2026-08-16 · Real model test:** Added a full image-request test in Settings, fixing the false confidence caused by a successful `/models` request to a model that still cannot process images.
96
+
97
+ ## Who it is for
98
+
99
+ 1. Want an interaction experience similar to a multimodal model: paste an image directly and ask a question or make a request.
100
+ 2. Want more than image Q&A — complete more complex, high-value visual tasks such as turning a sketch into a front-end page, converting an image into HTML, or extracting chat messages from long screenshots; more scenarios are added over time.
101
+
102
+ The bundled `vision-skills` Skill carries the complete upstream playbooks, explaining when to use each workflow, in what order to call the tools, and how to verify the result:
103
+
104
+ | Playbook | What the agent learns to do |
105
+ | --- | --- |
106
+ | [Read long screenshots, chat histories, and scrolling pages](assets/skill/references/long-screenshot-ocr.md) | Find low-content cut bands, OCR each chunk in order, preserve chat speakers/timestamps/quotes, merge only duplicated overlap, and surface risky boundaries for verification |
107
+ | [Rebuild a UI from a screenshot or design](assets/skill/references/restore-ui.md) | Reuse project components and assets first, then combine code-native UI, extracted visuals, rendered screenshots, and visual comparison to align a page or component |
108
+ | [Restore an icon, logo, illustration, or other graphic](assets/skill/references/restore-graphic.md) | Extract a transparent PNG from the source image, or rebuild an editable/scalable SVG when needed, then verify shape, color, and alpha edges |
109
+ | [Turn a sketch, diagram, or whiteboard into structured code](assets/skill/references/restore-structure.md) | Recover nodes, labels, connections, and directions as editable Mermaid, Graphviz, or another structured representation |
110
+ | [Operate a GUI from screenshots](assets/skill/references/gui.md) | Locate a control, perform one action, capture the screen again, and verify the resulting state before continuing |
111
+
112
+ ## See it in action
113
+
114
+ ### Paste an image directly into DSH
115
+
116
+ <p align="center">
117
+ <img src="assets/dsh-view-example.png" width="82%" alt="A text-only DeepSeek model answering a question about a pasted image through Vision Toolkit in DSH Web" />
118
+ </p>
119
+
120
+ *Paste an image into the conversation. A text-only model can switch to its `Vision Toolkit` variant and inspect the image in the context of the user's question.*
121
+
122
+ ### Screenshot to editable page
123
+
124
+ <p align="center">
125
+ <img src="assets/upstream/infographic-reference.webp" width="49%" alt="Reference infographic screenshot used for restoration" />
126
+ <img src="assets/upstream/infographic-result.webp" width="49%" alt="Editable HTML and CSS reconstruction created from the reference screenshot" />
127
+ </p>
128
+
129
+ > Prompt example: “(Use vision-skills) Rebuild this image into HTML.”
130
+
131
+ *Left: the reference screenshot. Right: an editable HTML/CSS result. The result can continue into screenshot rendering and pixel comparison instead of ending as an image description.*
132
+
133
+ ### Sketch to working interface
134
+
135
+ <p align="center">
136
+ <img src="assets/upstream/ui-sketch.webp" width="49%" alt="Hand-drawn JupyterLab interface used as the restoration reference" />
137
+ <img src="assets/upstream/ui-result.webp" width="49%" alt="Working JupyterLab-style interface reconstructed from the sketch" />
138
+ </p>
139
+
140
+ *Left: a hand-drawn reference. Right: the working interface reconstructed from it.*
141
+
142
+ > Prompt example: “(Use vision-skills) Turn this sketch into a working front-end page.”
143
+
144
+ ### Fast UI restoration: an approximate first pass
145
+
146
+ <p align="center">
147
+ <img src="assets/upstream/ui-fast-restore-reference.webp" width="49%" alt="Original YouMind homepage used as the fast UI restoration reference" />
148
+ <img src="assets/upstream/ui-fast-restore-result.webp" width="49%" alt="Approximate YouMind homepage produced with fast UI restoration mode" />
149
+ </p>
150
+
151
+ > Prompt example: “(Use vision-skills) Quickly rebuild this image into HTML.”
152
+
153
+ *Left: the original page. Right: a fast reconstruction that preserves the main layout, content, and visual hierarchy while allowing approximate colors and library icons. Fast mode targets a first screenshot in about three minutes.*
154
+
155
+ ## Quick start: three steps
156
+
157
+ ### 1. Install
158
+
159
+ ```sh
160
+ dsh plugin --profile web add @mengruo/dsh-vision-toolkit
161
+ ```
162
+
163
+ You can install it into a Headless Profile too:
164
+
165
+ ```sh
166
+ dsh plugin --profile headless add @mengruo/dsh-vision-toolkit
167
+ ```
168
+
169
+ Using DSH Desktop? It bundles its own `dsh` CLI and intentionally does not add it to your system PATH. Open **DSH Terminal** from the tray and run the command there, targeting the Desktop profile:
170
+
171
+ ```sh
172
+ dsh plugin --profile desktop add @mengruo/dsh-vision-toolkit
173
+ ```
174
+
175
+ Then restart DSH Desktop. The built-in plugin marketplace in DSH Desktop 2.0.1 has known installation issues; the terminal command above is the reliable path until a fixed Desktop release is available.
176
+
177
+ For the full Desktop install, update, and troubleshooting walkthrough, see [Installing and updating in DSH Desktop](docs/dsh-desktop-install.md).
178
+
179
+ ### 2. Restart and check it
180
+
181
+ Restart a running Web Profile, then open **Settings → Vision Toolkit**. The free provider is already configured; run **Test vision model** to confirm it is reachable.
182
+
183
+ The first start prepares an isolated runtime: the plugin prefers a system Python 3.11+; when none is found, it downloads a hash-verified standalone Python (about 35 MB) from the domestic mirror (`dsh-vision-python-bootstrap-1317715800.cos.ap-guangzhou.myqcloud.com`) on first use, falling back to the GitHub release when the mirror is unreachable. The locked runtime dependencies (Pillow, NumPy, vtracer) are installed from the Tencent Cloud PyPI mirror (`mirrors.cloud.tencent.com/pypi/simple`) first and fall back to the official PyPI index. A normal installation does not require an `agent-vision-toolkit` source checkout or a local path setting.
184
+
185
+ ### 3. Paste an image and describe the outcome you want
186
+
187
+ Paste a screenshot into the conversation or place an image in the session workspace, then invoke `/vision-skills`. For example:
188
+
189
+ ```text
190
+ Inspect this screenshot. Explain the error and tell me what to fix first.
191
+ Find the login button in the top-right corner, return original pixel coordinates, and make a boxed preview.
192
+ Crop this icon and convert it to SVG.
193
+ Rebuild the page from reference.png. After each pass, render it and run a pixel diff until the major differences are gone.
194
+ ```
195
+
196
+ ## Toolbox
197
+
198
+ The plugin provides 10 tools that can be called independently or composed into a workflow:
199
+
200
+ | Tool | Best question to ask | Main result |
201
+ | --- | --- | --- |
202
+ | `vision_glance` | “What is happening in this image?” | Focused answer, description, OCR, or multi-image comparison |
203
+ | `vision_ground` | “Where is the thing I need?” | Original pixel coordinates and optional boxed preview |
204
+ | `vision_detect` | “Which buttons, icons, or elements are present?” | Numbered element inventory, coordinates, and optional preview |
205
+ | `vision_crop` | “Extract this region as its own image” | PNG or JPEG crop |
206
+ | `vision_trace` | “Turn this shape into an editable vector” | SVG |
207
+ | `vision_pixel_diff` | “Where does the implementation differ from the reference?” | Difference percentage, ranked regions, heatmap, and JSON |
208
+ | `vision_long_screenshot_ocr` | “Read this entire long screenshot” | Markdown, chunks, manifest, and audit output |
209
+ | `vision_extract_foreground` | “Remove the background from this subject” | Transparent PNG |
210
+ | `vision_dominant_colors` | “Which colors dominate this area?” | Palette or ranked candidate colors |
211
+ | `vision_html_screenshot` | “Render this local page at an exact viewport or capture the full page” | PNG and optional CSS `pageHeight` |
212
+
213
+ Coordinates always use original-image pixels in `x1,y1,x2,y2` form, so grounding output can feed directly into cropping, tracing, or later automation.
214
+
215
+ For a long HTML document, pass `fullPage=true`. The requested width and height remain the layout viewport, while the resulting PNG covers the complete document and reports `pageHeight` in CSS pixels.
216
+
217
+ ## How it works
218
+
219
+ The plugin keeps image understanding and deterministic local image processing in one Agent workflow. The diagram below shows the implementation boundary.
220
+
221
+ ### Descriptions that keep the task in view
222
+
223
+ Most vision bridges for text-only models ask a multimodal model for a generic description and hand it to the text model, adding a semantic layer where information is lost. Vision Toolkit instead recovers **why the agent wants to look at the image**: the user message or the model's stated reason becomes a focus hint passed to the vision model. The result is a task-aware description that emphasizes what matters for the current step — with fewer tokens, higher accuracy, and faster responses.
224
+
225
+ <p align="center">
226
+ <img src="assets/upstream/focus-hint-comparison-1.webp" width="49%" alt="Generic image descriptions compared with task-aware vision using a focus hint - part 1" />
227
+ <img src="assets/upstream/focus-hint-comparison-2.webp" width="49%" alt="Generic image descriptions compared with task-aware vision using a focus hint - part 2" />
228
+ </p>
229
+
230
+ **Architecture and image-input behavior**
231
+
232
+ ```mermaid
233
+ flowchart LR
234
+ Image["Screenshot or local HTML"] --> Skill["vision-skills Skill"]
235
+ Skill --> Agent["Text agent selects a task"]
236
+ Agent --> Vision["Use a vision model when image understanding is needed"]
237
+ Agent --> Local["Run crop, SVG, and pixel work locally"]
238
+ Vision --> Result["Answer, OCR, coordinates"]
239
+ Local --> Artifact["PNG, SVG, heatmap, JSON"]
240
+ Result --> Session["Continue reasoning and acting"]
241
+ Artifact --> Session
242
+ ```
243
+
244
+ The visual capabilities come from a packaged, pinned `agent-vision-toolkit` snapshot. The DSH plugin handles installation, session-scoped tool exposure, Credentials, path checks, cancellation, timeouts, result files, and Web presentation. The runtime never fetches upstream `main` in the background.
245
+
246
+ The bundled `vision-skills` Skill is the DSH adapter of the upstream
247
+ `vision-tools` Skill: its `SKILL.md` plus all five upstream playbooks. Tool
248
+ names, argument syntax, Artifact delivery, progressive exposure, and DSH
249
+ path/lifecycle boundaries are adapted; the upstream tool-selection rules,
250
+ coarse-to-fine method, and task SOPs remain intact. The exact upstream Skill
251
+ commit, source hashes, adapted hashes, and reviewable adapter patch are
252
+ recorded in `assets/skill/UPSTREAM.json` and `patches/vision-tools-dsh.patch`.
253
+
254
+ For routes that DSH positively identifies as text-only, the plugin registers a sibling `<model> (Vision Toolkit)` variant. By default, pasting an image in DSH Web switches to that variant and gives the model both a reusable workspace path and a visual description focused on the current task.
255
+
256
+ ## Configuration and limits
257
+
258
+ ### Built-in free service
259
+
260
+ The default setup uses:
261
+
262
+ ```text
263
+ Base URL: https://vision.anionex.me/v1
264
+ Model: gemini-3.7-flash
265
+ API Key: https://agent-vision.anionex.me (filled automatically)
266
+ ```
267
+
268
+ Requests that still use the previous `qwen/qwen3.6-27b` model name remain compatible and are routed to the Qwen backend.
269
+
270
+ This is a shared zero-configuration entry point, not an unlimited private endpoint. Request safeguards include:
271
+
272
+ | Limit | Current value |
273
+ | --- | --- |
274
+ | Daily quota | 100 images per machine per day |
275
+ | Images per request | Up to 5 |
276
+ | Image size | 4 MiB per image |
277
+ | Decoded pixels | 20,000,000 per image |
278
+ | Output | Up to 4,096 tokens per request |
279
+
280
+ These safeguards prevent unusually large requests from monopolizing memory or request time. When shared capacity is reached, the service returns a readable `429` response with `Retry-After` instead of collapsing into an unexplained model failure.
281
+
282
+ Existing clients that still send `api_key="free"` remain compatible.
283
+
284
+ ### Bring your own vision model
285
+
286
+ For higher quotas, private endpoints, or another model, change the provider in **Settings → Vision Toolkit** and store the API key as a DSH Credential. Settings stores the Credential reference and never reads the saved secret back into the browser.
287
+
288
+ **Step-by-step AIHubMix tutorial:** [Get an AIHubMix API key and use free Gemini 3.7 Flash for vision](docs/aihubmix-gemini-vision.md). It includes screenshots for account/API-key setup, the exact Vision Toolkit settings, free-model selection, and troubleshooting.
289
+
290
+ You can also configure a Profile patch:
291
+
292
+ ```yaml
293
+ - id: vision-toolkit
294
+ config:
295
+ provider:
296
+ baseUrl: https://api.example.com/v1
297
+ credential: MY_VISION_KEY
298
+ model: your-vision-model
299
+ protocol: openai
300
+ ```
301
+
302
+ OpenAI Chat Completions-compatible endpoints and Anthropic Messages are supported. The Web Settings panel exposes the full provider, runtime, timeout, image-limit, and image-input-variant configuration.
303
+
304
+ For a trusted internal endpoint that uses a self-signed certificate or MITM proxy, start the DSH process with `VISION_SSL_VERIFY=0`. The plugin forwards that value to the isolated Python runtime; certificate verification remains enabled when the variable is unset or has any other value. The false values `false`, `off`, `no`, `none`, and `disabled` are also accepted, case-insensitively.
305
+
306
+ ### Configure the Python runtime
307
+
308
+ Most users never need to configure the Python runtime: the plugin prefers a system Python 3.11+ and otherwise downloads a pinned standalone Python automatically from the domestic mirror, falling back to GitHub when the mirror is unreachable.
309
+
310
+ For advanced setups — overriding `runtime.python`, using `runtime.mode: external`, verifying the runtime, or allowing additional input directories — see [Python runtime configuration](docs/python-runtime.md).
311
+
312
+ ## Troubleshooting
313
+
314
+ | Problem | What to do |
315
+ | --- | --- |
316
+ | The vision-model test fails with `Vision API returned an incompatible response structure` | The base URL usually needs a path prefix. Local OpenAI-compatible services such as LM Studio and Ollama should be entered as `http://127.0.0.1:1234/v1` (include `/v1`); the plugin appends `/chat/completions`, and a port-only address hits an unknown endpoint and returns this error |
317
+ | Pasting an image still says the model does not support image input | Restart the Web Profile, refresh the page, and confirm the selected route has the `(Vision Toolkit)` suffix. You can also place the image in the session workspace and invoke `/vision-skills` |
318
+ | The free service returns 429 | Wait for the `Retry-After` interval, or switch to your own endpoint when you need stable higher volume |
319
+ | The image exceeds a size or pixel limit | Crop or resize it first; the error identifies whether bytes or decoded pixels caused the rejection |
320
+ | A custom Credential is missing | Enter the API key in **Settings → Vision Toolkit** and confirm the Credential name matches the provider configuration |
321
+ | First-time runtime setup fails | The standalone-Python download needs network and disk access (domestic mirror first, GitHub fallback). Check connectivity or package-cache access, or install Python 3.11+ / configure `runtime.python` in Settings, then retry the model test |
322
+ | Chrome is not found | Install Chrome, Chromium, or Edge. Only HTML screenshot rendering is unavailable; the other tools still work |
323
+ | DSH Desktop says `dsh` is not recognized, or its built-in marketplace install fails | Open **DSH Terminal** from the tray, run `dsh plugin --profile desktop add @mengruo/dsh-vision-toolkit`, then restart DSH Desktop. The Desktop 2.0.1 marketplace has known install issues, so the terminal command is the reliable path for now |
324
+ | An artifact cannot be previewed | Use **Open file** or the workspace path in the result. Preview URLs exist only while the Web route is available |
325
+
326
+ ## FAQ
327
+
328
+ **Will adding a vision model significantly increase costs?**
329
+
330
+ No. Each inspection sends only the necessary intent and the image to the multimodal model, and context does not accumulate across calls, so the added cost stays small. To reduce it further, a locally deployed small multimodal side model (for example the Gemma 4 or Qwen 3.5/3.6 series) can provide the vision capability.
331
+
332
+ ## Donation
333
+
334
+ If this project is valuable to you, you are welcome to buy the developer a coffee ☕️
335
+
336
+ <img width="240" alt="WeChat reward code" src="assets/wechat-reward.png" />
337
+
338
+ ## Development and community
339
+
340
+ - Read [CONTRIBUTING.md](CONTRIBUTING.md) before contributing.
341
+ - Use [GitHub Issues](https://github.com/mengruoa/dsh-vision-toolkit/issues) for bugs, focused feature requests, and usage questions; see [SUPPORT.md](SUPPORT.md) for channel guidance.
342
+ - Report vulnerabilities privately through [SECURITY.md](SECURITY.md).
343
+ - See [CHANGELOG.md](CHANGELOG.md) for releases and [FUNDING.md](FUNDING.md) for sponsorship details.
344
+ - Visit upstream [agent-vision-toolkit](https://github.com/Anionex/agent-vision-toolkit) for the general toolkit, cross-agent integrations, and visual-task playbooks.
345
+
346
+ <p align="center">
347
+ <img src="assets/community-group-qr.png" alt="QR code for the agent-vision-toolkit community group" width="240" />
348
+ </p>
349
+
350
+ I'm [anionex](https://anionex.me/), an AI-native developer who once ranked **No. 3** on GitHub's global developer trending list, with more than 16k stars across my projects. If you would like to follow my future work, [follow me on GitHub](https://github.com/Anionex).
351
+
352
+ [`agent-vision-toolkit`](https://github.com/Anionex/agent-vision-toolkit) was created by [Anionex](https://anionex.me/). This repository maintains its native DeepSeek Harness integration.
353
+
354
+ ## License
355
+
356
+ The plugin is available under the [MIT License](LICENSE). The packaged upstream snapshot retains its original MIT license in [`vendor/agent-vision-toolkit/LICENSE`](vendor/agent-vision-toolkit/LICENSE).