@anionex/dsh-vision-toolkit 0.1.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.i18n.yaml +6 -0
- package/README.md +383 -0
- package/README.zh.md +383 -0
- package/assets/dsh-conversation-artifact.png +0 -0
- package/assets/dsh-conversation-image-qa-top.png +0 -0
- package/assets/dsh-conversation-image-qa.png +0 -0
- package/assets/dsh-conversation-pixel-diff.png +0 -0
- package/assets/dsh-conversation-screenshot-debugging-top.png +0 -0
- package/assets/dsh-conversation-screenshot-debugging.png +0 -0
- package/assets/dsh-conversation-tool-call.png +0 -0
- package/assets/dsh-conversation-vision-trace.png +0 -0
- package/assets/hero.png +0 -0
- package/assets/social-preview.png +0 -0
- package/assets/upstream/README.md +16 -0
- package/assets/upstream/image-qa.webp +0 -0
- package/assets/upstream/infographic-reference.webp +0 -0
- package/assets/upstream/infographic-result.webp +0 -0
- package/assets/upstream/screenshot-debugging.webp +0 -0
- package/assets/upstream/ui-result.webp +0 -0
- package/assets/upstream/ui-sketch.webp +0 -0
- package/assets/vision-settings.png +0 -0
- package/cordis.patch.yml +6 -0
- package/docs/assets/vision-settings.png +0 -0
- package/docs/requirements-traceability/README.i18n.yaml +6 -0
- package/docs/requirements-traceability/README.md +75 -0
- package/docs/requirements-traceability/README.zh.md +75 -0
- package/examples/ui-restoration/README.i18n.yaml +6 -0
- package/examples/ui-restoration/README.md +70 -0
- package/examples/ui-restoration/README.zh.md +70 -0
- package/examples/ui-restoration/assets/final-heatmap.png +0 -0
- package/examples/ui-restoration/assets/final-report.json +83 -0
- package/examples/ui-restoration/assets/implementation.png +0 -0
- package/examples/ui-restoration/assets/initial-heatmap.png +0 -0
- package/examples/ui-restoration/assets/initial-report.json +83 -0
- package/examples/ui-restoration/assets/initial.png +0 -0
- package/examples/ui-restoration/assets/metrics.json +12 -0
- package/examples/ui-restoration/assets/reference.png +0 -0
- package/examples/ui-restoration/implementation.html +94 -0
- package/examples/ui-restoration/initial.html +57 -0
- package/lib/artifact-access.js +369 -0
- package/lib/artifact-access.js.map +1 -0
- package/lib/artifacts.js +56 -0
- package/lib/artifacts.js.map +1 -0
- package/lib/client.js +952 -0
- package/lib/client.js.map +1 -0
- package/lib/config.js +117 -0
- package/lib/config.js.map +1 -0
- package/lib/errors.js +56 -0
- package/lib/errors.js.map +1 -0
- package/lib/exposure.js +213 -0
- package/lib/exposure.js.map +1 -0
- package/lib/index.js +97 -0
- package/lib/index.js.map +1 -0
- package/lib/paste-images.js +199 -0
- package/lib/paste-images.js.map +1 -0
- package/lib/paths.js +325 -0
- package/lib/paths.js.map +1 -0
- package/lib/runtime-install.js +601 -0
- package/lib/runtime-install.js.map +1 -0
- package/lib/runtime-manager.js +126 -0
- package/lib/runtime-manager.js.map +1 -0
- package/lib/runtime.js +1344 -0
- package/lib/runtime.js.map +1 -0
- package/lib/skill.js +139 -0
- package/lib/skill.js.map +1 -0
- package/lib/tools.js +528 -0
- package/lib/tools.js.map +1 -0
- package/lib/types/artifact-access.d.ts +61 -0
- package/lib/types/artifact-access.d.ts.map +1 -0
- package/lib/types/artifacts.d.ts +42 -0
- package/lib/types/artifacts.d.ts.map +1 -0
- package/lib/types/client/index.d.ts +179 -0
- package/lib/types/client/index.d.ts.map +1 -0
- package/lib/types/client/paste-images.d.ts +57 -0
- package/lib/types/client/paste-images.d.ts.map +1 -0
- package/lib/types/config.d.ts +73 -0
- package/lib/types/config.d.ts.map +1 -0
- package/lib/types/errors.d.ts +35 -0
- package/lib/types/errors.d.ts.map +1 -0
- package/lib/types/exposure.d.ts +40 -0
- package/lib/types/exposure.d.ts.map +1 -0
- package/lib/types/index.d.ts +18 -0
- package/lib/types/index.d.ts.map +1 -0
- package/lib/types/paste-images.d.ts +21 -0
- package/lib/types/paste-images.d.ts.map +1 -0
- package/lib/types/paths.d.ts +107 -0
- package/lib/types/paths.d.ts.map +1 -0
- package/lib/types/runtime-install.d.ts +49 -0
- package/lib/types/runtime-install.d.ts.map +1 -0
- package/lib/types/runtime-manager.d.ts +60 -0
- package/lib/types/runtime-manager.d.ts.map +1 -0
- package/lib/types/runtime.d.ts +389 -0
- package/lib/types/runtime.d.ts.map +1 -0
- package/lib/types/skill.d.ts +15 -0
- package/lib/types/skill.d.ts.map +1 -0
- package/lib/types/tools.d.ts +22 -0
- package/lib/types/tools.d.ts.map +1 -0
- package/lib/types/upstream.d.ts +207 -0
- package/lib/types/upstream.d.ts.map +1 -0
- package/lib/types/version.d.ts +15 -0
- package/lib/types/version.d.ts.map +1 -0
- package/lib/types/web-request.d.ts +4 -0
- package/lib/types/web-request.d.ts.map +1 -0
- package/lib/types/web.d.ts +74 -0
- package/lib/types/web.d.ts.map +1 -0
- package/lib/upstream.js +675 -0
- package/lib/upstream.js.map +1 -0
- package/lib/version.js +18 -0
- package/lib/version.js.map +1 -0
- package/lib/web-request.js +20 -0
- package/lib/web-request.js.map +1 -0
- package/lib/web.js +244 -0
- package/lib/web.js.map +1 -0
- package/package.json +139 -0
- package/runtime/requirements.lock +3 -0
- package/src/artifact-access.ts +386 -0
- package/src/artifacts.ts +85 -0
- package/src/client/index.tsx +866 -0
- package/src/client/paste-images.tsx +426 -0
- package/src/config.ts +177 -0
- package/src/errors.ts +62 -0
- package/src/exposure.ts +227 -0
- package/src/index.ts +122 -0
- package/src/paste-images.ts +234 -0
- package/src/paths.ts +348 -0
- package/src/runtime-install.ts +723 -0
- package/src/runtime-manager.ts +166 -0
- package/src/runtime.ts +1783 -0
- package/src/skill.ts +143 -0
- package/src/tools.ts +668 -0
- package/src/upstream.ts +861 -0
- package/src/version.ts +37 -0
- package/src/web-request.ts +17 -0
- package/src/web.ts +329 -0
- package/vendor/agent-vision-toolkit/CHANGELOG.md +16 -0
- package/vendor/agent-vision-toolkit/LICENSE +21 -0
- package/vendor/agent-vision-toolkit/README.md +399 -0
- package/vendor/agent-vision-toolkit/UPSTREAM_MANIFEST.json +89 -0
- package/vendor/agent-vision-toolkit/bin/crop +90 -0
- package/vendor/agent-vision-toolkit/bin/detect +13 -0
- package/vendor/agent-vision-toolkit/bin/glance +93 -0
- package/vendor/agent-vision-toolkit/bin/ground +13 -0
- package/vendor/agent-vision-toolkit/bin/trace +129 -0
- package/vendor/agent-vision-toolkit/detect.py +56 -0
- package/vendor/agent-vision-toolkit/ground.py +216 -0
- package/vendor/agent-vision-toolkit/skills/vision-tools/scripts/dominant_colors.py +224 -0
- package/vendor/agent-vision-toolkit/skills/vision-tools/scripts/extract_fg.py +278 -0
- package/vendor/agent-vision-toolkit/skills/vision-tools/scripts/html_shot.py +108 -0
- package/vendor/agent-vision-toolkit/skills/vision-tools/scripts/long_screenshot_ocr.py +1245 -0
- package/vendor/agent-vision-toolkit/skills/vision-tools/scripts/pixel_diff.py +88 -0
- package/vendor/agent-vision-toolkit/vision_client.py +156 -0
package/README.zh.md
ADDED
|
@@ -0,0 +1,383 @@
|
|
|
1
|
+

|
|
2
|
+
|
|
3
|
+
# DSH Vision Toolkit
|
|
4
|
+
|
|
5
|
+
[](https://x.com/anion_ex)
|
|
6
|
+
[](https://github.com/Anionex/dsh-vision-toolkit/releases/tag/v0.1.4)
|
|
7
|
+
[](tests)
|
|
8
|
+
[](LICENSE)
|
|
9
|
+
[](package.json)
|
|
10
|
+
[](runtime/requirements.lock)
|
|
11
|
+
[](cordis.patch.yml)
|
|
12
|
+
|
|
13
|
+
**安装:** `dsh plugin --profile web add @anionex/dsh-vision-toolkit`
|
|
14
|
+
|
|
15
|
+
**DSH Vision Toolkit 将 [`agent-vision-toolkit`](https://github.com/Anionex/agent-vision-toolkit) 作为原生 Profile Bundle 带入 DeepSeek Harness。**
|
|
16
|
+
|
|
17
|
+
让纯文本 DSH Agent 真正看见,并知道当前任务应该看哪里:通过带意图的图片问答、OCR、原图像素定位、UI 还原、像素验证、托管产物和 Web Settings 完成视觉闭环。10 个独立工具以结构化 schema 和 Agent 级渐进暴露取代 Shell 拼接。
|
|
18
|
+
|
|
19
|
+
**上游工具箱:** [Anionex/agent-vision-toolkit](https://github.com/Anionex/agent-vision-toolkit) · **项目网站:** [agent-vision.anionex.me](https://agent-vision.anionex.me)
|
|
20
|
+
|
|
21
|
+
[English](README.md) | 中文
|
|
22
|
+
|
|
23
|
+
## 为什么需要它
|
|
24
|
+
|
|
25
|
+
`agent-vision-toolkit` 把视觉视为 Agent 可调用的能力,而不是基础模型自带的天赋。它会把“为什么要看这张图”带入视觉请求,从全局逐步收敛到目标区域,并用专用工具验证坐标、颜色、轮廓和差异,不把泛化描述直接当成证据。
|
|
26
|
+
|
|
27
|
+
DSH Vision Toolkit 保留这套方法,并用原生 schema、DSH Credentials、受生命周期管理的运行时准备、可从 Session 日志重建的结构化结果、可预览产物、专用 Web 卡片和 Settings 取代 CLI 安装与 Bash 参数拼接。Agent 加载一个带版本的 Skill,只有当前任务需要视觉时才会获得 10 个工具 schema。
|
|
28
|
+
|
|
29
|
+
本包完整交付已承诺的 P0 与 P1 产品范围。P2 的稳定 `ctx.visionToolkit` 服务会等到独立插件成为真实消费方后再发布;内部运行时不会把未经验证的生态接口伪装为稳定契约。
|
|
30
|
+
|
|
31
|
+
## agent-vision-toolkit 已验证的真实用例
|
|
32
|
+
|
|
33
|
+
前两张图是本 Bundle 所打包 `agent-vision-toolkit` 固定版本同一代码线上的官方实跑结果;图片问答与截图辅助排障这一张则是在 DeepSeek Harness Web 中实际运行的会话,展示 DSH 中的同一套工作流。上游图片来源见[素材溯源记录](assets/upstream/README.md)。
|
|
34
|
+
|
|
35
|
+
### 信息图还原:从截图到可编辑 HTML/CSS
|
|
36
|
+
|
|
37
|
+
<p align="center">
|
|
38
|
+
<img src="assets/upstream/infographic-reference.webp" width="49%" alt="上游用于还原的三阶段模型训练信息图原始截图。" />
|
|
39
|
+
<img src="assets/upstream/infographic-result.webp" width="49%" alt="上游使用 HTML 和 CSS 还原出的可编辑模型训练信息图。" />
|
|
40
|
+
</p>
|
|
41
|
+
|
|
42
|
+
*左:原始截图;右:上游[信息图还原示例](https://github.com/Anionex/agent-vision-toolkit/blob/c27d1a300962b553c0884993c575cd3e819465ce/examples/infographic-restoration/how-is-the-model-trained.html)生成的可编辑 HTML/CSS 结果。*
|
|
43
|
+
|
|
44
|
+
### UI 还原:从手绘稿到可用界面
|
|
45
|
+
|
|
46
|
+
<p align="center">
|
|
47
|
+
<img src="assets/upstream/ui-sketch.webp" width="49%" alt="上游用于 UI 还原的手绘 JupyterLab 工作区参考图。" />
|
|
48
|
+
<img src="assets/upstream/ui-result.webp" width="49%" alt="上游依据手绘参考图还原出的 JupyterLab 风格可用界面。" />
|
|
49
|
+
</p>
|
|
50
|
+
|
|
51
|
+
*左:手绘输入;右:上游还原出的界面,完整方法见 [UI 还原 playbook](https://github.com/Anionex/agent-vision-toolkit/blob/c27d1a300962b553c0884993c575cd3e819465ce/skills/vision-tools/references/restore-ui.md)。*
|
|
52
|
+
|
|
53
|
+
### 图片问答与截图辅助排障
|
|
54
|
+
|
|
55
|
+
<p align="center">
|
|
56
|
+
<img src="assets/dsh-conversation-image-qa.png" width="49%" alt="DSH Web 会话中,纯文本 Agent 针对 UI 参考图回答聚焦问题。" />
|
|
57
|
+
<img src="assets/dsh-conversation-screenshot-debugging.png" width="49%" alt="DSH Web 会话中,Agent 根据截图对比定位 UI 字段差异并建议继续运行 vision_pixel_diff。" />
|
|
58
|
+
</p>
|
|
59
|
+
|
|
60
|
+
*左:DSH Web 中带意图的图片问答;右:DSH Web 中通过截图对比定位 UI 字段差异,并继续向 `vision_pixel_diff` 推进。上游工作流来源仍为 [`agent-vision-toolkit` 官方实跑](https://github.com/Anionex/agent-vision-toolkit/blob/c27d1a300962b553c0884993c575cd3e819465ce/README.md#real-world-effects)。*
|
|
61
|
+
|
|
62
|
+
DSH Vision Toolkit 在这些上游能力之外增加原生工具 schema、版本化生命周期、Credentials、结构化 Session 结果、产物、Web 展示、Settings 和渐进暴露。下一节展示由本 DSH 仓库实际执行并提交的可复现实证。
|
|
63
|
+
|
|
64
|
+
## DSH 原生实证:从参考图到像素级一致
|
|
65
|
+
|
|
66
|
+
仓库中的 UI 还原流程会渲染一个故意不准确的 HTML 实现,测得 `6.04%` 像素差异和 6 个非零差异区域;经过迭代后,在 `1200 × 720` 下达到相对参考图精确 `0%` 的差异。
|
|
67
|
+
|
|
68
|
+
<p>
|
|
69
|
+
<img src="examples/ui-restoration/assets/initial.png" width="49%" alt="Vision Toolkit 迭代前的 UI 还原候选,与参考图仍有可测量的布局和样式差异。" />
|
|
70
|
+
<img src="examples/ui-restoration/assets/implementation.png" width="49%" alt="仓库内可复现流程生成的最终 UI 还原结果,与参考图达到零像素差异。" />
|
|
71
|
+
</p>
|
|
72
|
+
|
|
73
|
+
| 已验证范围 | 证据 |
|
|
74
|
+
|---|---|
|
|
75
|
+
| 产品范围 | 10 个独立视觉工具、匹配的 `vision-tools` Skill、产物、专用 Web 卡片和实时 Settings |
|
|
76
|
+
| 自动化覆盖 | 17 个 Vitest 文件 / 136 项通过测试,以及不依赖 DSH 开发树的可移植包检查 |
|
|
77
|
+
| 真实 Profile | 干净临时 Web 与 Headless 安装、激活、禁用、重新启用和卸载 |
|
|
78
|
+
| 视觉验收 | 可复现的 HTML 截图 → 像素对比示例,最终差异为 `0%` |
|
|
79
|
+
|
|
80
|
+
## 亮点
|
|
81
|
+
|
|
82
|
+
- **看图但不让每轮提示词膨胀:** 初始只暴露 `vision_toolkit_activate`;加载 `vision-tools` 后,10 个独立 schema 才挂到当前 Agent,版本和健康管理始终不进入模型上下文。
|
|
83
|
+
- **直接使用坐标,而不是解析自然语言:** 定位和元素盘点返回原图像素框,所有模型可见结果保持为结构化文字或 JSON。
|
|
84
|
+
- **交付正式文件,而不是临时输出:** 裁剪、SVG 恢复、OCR、像素对比、前景提取和 HTML 渲染会生成带描述的产物,Web 客户端可预览、下载或在本地打开。
|
|
85
|
+
- **受控管理运行时与凭据:** API Key 由 DSH Credentials 保管;managed 模式准备精确隔离的 Python 环境;失败的 Settings 候选不会替换当前服务 generation。
|
|
86
|
+
- **闭合视觉验证循环:** 本地 HTML 渲染和像素差异排序支持参考图 → 实现 → 截图 → 度量迭代,不依赖模型原生图片通道。
|
|
87
|
+
- **同一 Bundle 同时服务 Web 与 Headless:** Web 增加卡片、预览、Settings 和健康操作;Headless 保持相同工具语义和完整结构化结果。
|
|
88
|
+
|
|
89
|
+
## 快速开始
|
|
90
|
+
|
|
91
|
+
前置条件:DeepSeek Harness `0.1.0-rc.6` 或兼容的后续 `0.1.x` 版本、Python 3.11+,并确保 `dsh plugin` 可以使用 `pnpm`。从 npm 安装已发布的 Bundle,将其加入所需 Profile,并确认 Bundle 行已经挂载:
|
|
92
|
+
|
|
93
|
+
```sh
|
|
94
|
+
dsh plugin --profile web add @anionex/dsh-vision-toolkit
|
|
95
|
+
dsh plugin --profile headless add @anionex/dsh-vision-toolkit
|
|
96
|
+
dsh --profile web --dump-config | grep vision-toolkit
|
|
97
|
+
dsh --profile headless --dump-config | grep vision-toolkit
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
旧 Profile 的 `pnpm-workspace.yaml` 必须使用 `nodeLinker: hoisted` 和 `autoInstallPeers: false`。更新后的 DSH launcher 会在 `dsh plugin` 运行前修复这两个自有设置;使用旧 launcher 时,应在安装前手动设置,避免 pnpm 在 Profile 内组装第二套 Harness 依赖图。
|
|
101
|
+
|
|
102
|
+
安装后重启正在运行的 Web Profile,打开 **设置 → 视觉工具**,为远程工具选择 DSH Credential,并显式执行**测试连接**。在会话中把图片放进工作区路径,调用 `/vision-tools`,再让 Agent 使用明确的 `vision_*` 工具。本地裁剪、SVG、像素、颜色、前景和 HTML 操作不需要视觉 API Credential。
|
|
103
|
+
|
|
104
|
+
## 工作原理
|
|
105
|
+
|
|
106
|
+
```mermaid
|
|
107
|
+
flowchart LR
|
|
108
|
+
User["Workspace image or local HTML"] --> Skill["vision-tools Skill"]
|
|
109
|
+
Skill --> Activate["Agent-scoped activation"]
|
|
110
|
+
Activate --> Tools["10 independent vision_* tools"]
|
|
111
|
+
Tools --> Runtime["Shared VisionToolkitRuntime"]
|
|
112
|
+
Credentials["DSH Credentials"] --> Runtime
|
|
113
|
+
Settings["Web Settings and health"] --> Runtime
|
|
114
|
+
Runtime --> Upstream["Pinned agent-vision-toolkit"]
|
|
115
|
+
Runtime --> Remote["Configured vision API"]
|
|
116
|
+
Upstream --> Result["Text, coordinates, JSON"]
|
|
117
|
+
Remote --> Result
|
|
118
|
+
Runtime --> Artifacts["Workspace Artifacts"]
|
|
119
|
+
Result --> Session["Reconstructable Session log"]
|
|
120
|
+
Artifacts --> Web["Preview, download, or open file"]
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
所有工具定义都调用同一个 Runtime;Runtime 在分发到固定上游快照或已配置的 OpenAI 兼容视觉端点前,统一验证路径、限制、Credential、取消和超时。Web 展示读取相同的结构化结果与产物描述,因此不会改变 Headless 语义。健康、连接测试和版本检查只留在 Settings,不进入模型工具 schema。
|
|
124
|
+
|
|
125
|
+
## 工具
|
|
126
|
+
|
|
127
|
+
| 工具 | 执行方式 | 结构化结果 | 产物交付 |
|
|
128
|
+
|---|---|---|---|
|
|
129
|
+
| `vision_glance` | 远程视觉 API | 描述、针对性回答、OCR 或多图比较 | 无 |
|
|
130
|
+
| `vision_ground` | 远程视觉 API;可选本地预览 | 目标、原图尺寸和像素框 | 可选标注 PNG |
|
|
131
|
+
| `vision_detect` | 远程视觉 API;可选本地预览 | 带编号的元素清单和原图像素框 | 可选编号 PNG |
|
|
132
|
+
| `vision_trace` | 本地固定 vtracer 流水线 | SVG 几何状态、路径数、缩放和大小 | SVG |
|
|
133
|
+
| `vision_crop` | 本地 Pillow 流水线 | 实际像素框、尺寸、格式和裁剪边界状态 | PNG 或 JPEG |
|
|
134
|
+
| `vision_pixel_diff` | 本地 NumPy/Pillow 流水线 | 差异比例和排序后的网格区域 | PNG 热力图和 JSON 报告 |
|
|
135
|
+
| `vision_long_screenshot_ocr` | 本地切分/审计;除 `splitOnly=true` 外执行远程 OCR | 分块边界、复用状态、完成状态和运行目录 | Markdown、manifest、边界审计、分块 PNG 和 OCR 伴随文件 |
|
|
136
|
+
| `vision_extract_foreground` | 本地固定提取流水线 | 选区、连通分量数、前景覆盖率和尺寸 | 透明 PNG |
|
|
137
|
+
| `vision_dominant_colors` | 本地固定颜色分析 | 提取的调色板或有像素证据的候选色排序 | 无 |
|
|
138
|
+
| `vision_html_screenshot` | 本地 Chrome/Chromium/Edge 适配器 | 已授权源文件信息、视口和渲染尺寸 | PNG |
|
|
139
|
+
|
|
140
|
+
插件不重新实现视觉算法。DSH 侧只负责验证路径与限制、解析 Credential、用 argv 向量调用固定上游脚本、解析精确输出契约、分类失败、描述文件,并把结果投影给模型和 Web 客户端。
|
|
141
|
+
|
|
142
|
+
## 渐进式模型暴露
|
|
143
|
+
|
|
144
|
+
运行时就绪状态属于整个 Profile,但 10 个视觉执行工具的 schema 属于具体 Agent。Agent 加载 `vision-tools` 前,插件只贡献很小的 `vision_toolkit_activate` 引导工具;该 Agent 的请求 schema 中没有视觉执行工具。标准 `skill` 工具以 `name="vision-tools"` 成功加载后,会为下一模型步骤自动挂载全部 10 个工具并隐藏引导工具。直接调用 `/vision-tools` 会注入 skill 指令;如果此时视觉工具仍不可见,这些指令要求调用一次 `vision_toolkit_activate`。激活只影响当前 Agent;Session 中存在与打包 skill 版本匹配的持久证据时可以恢复,并持续到 Agent 或插件被释放。
|
|
145
|
+
|
|
146
|
+
健康检查、连接测试以及插件/上游版本检查属于 Web Settings 管理操作。`vision_toolkit_health` 和 `vision_toolkit_version` 不是模型工具,即使视觉执行工具已经激活,也永远不会进入 Agent schema。
|
|
147
|
+
|
|
148
|
+
## 运行要求
|
|
149
|
+
|
|
150
|
+
- 启用 Web 或 Headless Profile 的 DeepSeek Harness,并确保 `dsh plugin` 可以使用 `pnpm`。
|
|
151
|
+
- Python 3.11 或更高版本。Managed 模式会创建隔离环境,用户无需手工安装上游 CLI(命令行界面)或 Python 包。
|
|
152
|
+
- 首次启用 managed 运行时需要联网;如果配置的软件包缓存已有 `runtime/requirements.lock` 中的精确版本,则无需联网。
|
|
153
|
+
- `vision_glance`、`vision_ground`、`vision_detect` 和非仅切分长截图 OCR 需要 OpenAI 兼容视觉端点及 DSH Credential。本地工具无需该 Credential 也可使用。
|
|
154
|
+
- 只有 `vision_html_screenshot` 需要 Chrome、Chromium 或 Edge;未安装受支持浏览器时,其他工具保持可用。
|
|
155
|
+
- 输入必须是会话工作区或显式 `allowedDirs` 根目录内的 PNG、JPEG、GIF 或 WebP。
|
|
156
|
+
|
|
157
|
+
## 安装与生命周期
|
|
158
|
+
|
|
159
|
+
### 安装
|
|
160
|
+
|
|
161
|
+
将 Bundle 安装到需要暴露能力的每个 Profile:
|
|
162
|
+
|
|
163
|
+
```sh
|
|
164
|
+
dsh plugin --profile web add @anionex/dsh-vision-toolkit
|
|
165
|
+
dsh plugin --profile headless add @anionex/dsh-vision-toolkit
|
|
166
|
+
dsh --profile web --dump-config | grep vision-toolkit
|
|
167
|
+
dsh --profile headless --dump-config | grep vision-toolkit
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
安装后需要重启长期运行的 Web Profile。宿主在进程启动时通过 `package.json` 的 `dsh.client` 声明发现已构建的浏览器 Bundle;旧的顶层 `dshClient` 字段不会被扫描。
|
|
171
|
+
|
|
172
|
+
首次 managed 启动会验证打包的上游 manifest(元数据清单),并在 `DSH_HOME/cache/dsh-vision-toolkit` 下原子准备隔离环境。插件只在准备成功后发布同版本的 `vision-tools` skill 与激活引导工具;每个 Agent 只有在加载该 skill 后才获得执行工具。初次准备失败时,Web Settings 修复入口仍然可用,但插件不会暴露任何模型能力或误导模型的 skill。
|
|
173
|
+
|
|
174
|
+
### 禁用与重新启用
|
|
175
|
+
|
|
176
|
+
在 Profile patch 或 overlay 中把 Bundle 行设为 `disabled: true`:
|
|
177
|
+
|
|
178
|
+
```yaml
|
|
179
|
+
- id: vision-toolkit
|
|
180
|
+
disabled: true
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
删除该字段或设为 `false` 即可重新启用。资源释放会先取消插件拥有的视觉操作,再移除全部 Agent 级工具、引导工具和 skill;重新启用时,配置的运行时准备完成后才会暴露任何模型能力。用户配置和已完成的产物会保留。
|
|
184
|
+
|
|
185
|
+
### 升级
|
|
186
|
+
|
|
187
|
+
通过注册表安装时,使用 Profile 的包管理命令更新依赖:
|
|
188
|
+
|
|
189
|
+
```sh
|
|
190
|
+
dsh plugin --profile web update @anionex/dsh-vision-toolkit
|
|
191
|
+
dsh plugin --profile headless update @anionex/dsh-vision-toolkit
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
通过本地路径安装时,对替换后的 checkout 或 tarball 再次执行 `add`。Settings 保存在 Profile 的 Settings 提供方中。候选运行时完成验证和准备后才会持久化并启用;失败候选或已经陈旧的并发候选无法替换当前服务 generation。
|
|
195
|
+
|
|
196
|
+
### 卸载
|
|
197
|
+
|
|
198
|
+
```sh
|
|
199
|
+
dsh plugin --profile web remove @anionex/dsh-vision-toolkit
|
|
200
|
+
dsh plugin --profile headless remove @anionex/dsh-vision-toolkit
|
|
201
|
+
```
|
|
202
|
+
|
|
203
|
+
`dsh plugin remove` 会同时移除依赖及其 Bundle 层。Profile 随即不再暴露激活引导工具、Agent 级 Vision Toolkit 工具或 skill 条目。没有 Profile 使用本包时可以另行删除 managed 缓存;缓存不是活动配置,无法自行注册任何能力。
|
|
204
|
+
|
|
205
|
+
## 配置
|
|
206
|
+
|
|
207
|
+
Bundle 默认使用 managed 运行时。Profile patch 可以覆盖提供方与限制:
|
|
208
|
+
|
|
209
|
+
```yaml
|
|
210
|
+
- id: vision-toolkit
|
|
211
|
+
config:
|
|
212
|
+
provider:
|
|
213
|
+
baseUrl: https://api.inferera.com/v1
|
|
214
|
+
credential: VISION_API_KEY
|
|
215
|
+
model: gemini-3.6-flash
|
|
216
|
+
language: zh
|
|
217
|
+
timeoutMs: 60000
|
|
218
|
+
maxImageBytes: 10485760
|
|
219
|
+
maxImagePixels: 40000000
|
|
220
|
+
concurrency: 4
|
|
221
|
+
runtime:
|
|
222
|
+
mode: managed
|
|
223
|
+
allowedDirs: []
|
|
224
|
+
```
|
|
225
|
+
|
|
226
|
+
### 配置字段
|
|
227
|
+
|
|
228
|
+
| 字段 | 默认值 | 契约 |
|
|
229
|
+
|---|---|---|
|
|
230
|
+
| `provider.baseUrl` | `https://api.inferera.com/v1` | OpenAI 兼容基础 URL;去除结尾斜杠后使用 |
|
|
231
|
+
| `provider.credential` | `VISION_API_KEY` | DSH Credential 引用,不是密钥值 |
|
|
232
|
+
| `provider.model` | `gemini-3.6-flash` | 远程工具使用的多模态模型名 |
|
|
233
|
+
| `language` | `zh` | 视觉输出语言:`zh` 或 `en` |
|
|
234
|
+
| `timeoutMs` | `60000` | 完整操作截止时间,1000-600000 毫秒;每个工具可请求更窄的覆盖值 |
|
|
235
|
+
| `maxImageBytes` | `10485760` | 每张输入图片的编码字节上限 |
|
|
236
|
+
| `maxImagePixels` | `40000000` | 每张输入图片的解码像素上限 |
|
|
237
|
+
| `concurrency` | `4` | 每个会话内的并发操作数,1-16 |
|
|
238
|
+
| `runtime.mode` | `managed` | `managed` 使用打包快照;`external` 只接受精确固定版本 |
|
|
239
|
+
| `runtime.agentVisionToolkitPath` | 未设置 | `external` 模式必填;必须是精确导出快照或固定 commit 的干净 Git checkout |
|
|
240
|
+
| `runtime.python` | 未设置 | 可选的 Python 3.11+ 引导程序/解释器覆盖值 |
|
|
241
|
+
| `allowedDirs` | `[]` | 额外的 realpath 解析输入根目录;会话工作区始终允许 |
|
|
242
|
+
|
|
243
|
+
### Credential
|
|
244
|
+
|
|
245
|
+
通过 DSH Credentials 创建或替换引用指向的密钥:
|
|
246
|
+
|
|
247
|
+
```sh
|
|
248
|
+
dsh credentials set VISION_API_KEY
|
|
249
|
+
```
|
|
250
|
+
|
|
251
|
+
Settings 只保存引用,不保存值。每次远程操作都会重新解析引用,并只把值注入对应子进程环境。插件排除用户 `.env`、checkout `.env`、`PYTHONPATH`、`PYTHONHOME`、`VIRTUAL_ENV` 和用户 site-packages,避免环境中的 Python 或上游配置覆盖选定的 DSH 提供方。日志、错误、工具结果、产物元数据和 Settings 响应都不包含密钥。
|
|
252
|
+
|
|
253
|
+
### Managed 与 external 运行时
|
|
254
|
+
|
|
255
|
+
Managed 模式会验证 `vendor/agent-vision-toolkit/UPSTREAM_MANIFEST.json`,优先使用 `uv`,回退到 `venv` 加 pip,按 `runtime/requirements.lock` 安装精确版本,通过 heartbeat 锁协调并发准备,并只在全部探针通过后发布 staging 环境。
|
|
256
|
+
|
|
257
|
+
External 模式用于开发或受控部署:
|
|
258
|
+
|
|
259
|
+
```yaml
|
|
260
|
+
- id: vision-toolkit
|
|
261
|
+
config:
|
|
262
|
+
runtime:
|
|
263
|
+
mode: external
|
|
264
|
+
agentVisionToolkitPath: /opt/agent-vision-toolkit
|
|
265
|
+
python: python3.12
|
|
266
|
+
```
|
|
267
|
+
|
|
268
|
+
该路径必须是与打包 manifest 一致的导出副本,或 commit `c27d1a300962b553c0884993c575cd3e819465ce` 的干净 Git checkout 根目录。插件拒绝已修改的 tracked 文件和 untracked 文件,因为它们可能改变或遮蔽固定 Python 行为。
|
|
269
|
+
|
|
270
|
+
## Web Settings
|
|
271
|
+
|
|
272
|
+
Web Profile 会注册 Vision Toolkit Settings 分区,可配置提供方 URL、Credential 引用、模型、语言、超时、字节/像素限制、并发数、运行时模式、Python 覆盖值、external 源码路径和允许目录。该页面还会显示插件/上游版本、当前运行时 generation、不含密钥的 Credential configured/source/writable 状态、运行时路径、健康检查结果和产物路由可用性。
|
|
273
|
+
|
|
274
|
+
“保存并应用”会验证完整配置,准备候选 Python/上游运行时,提交 Settings revision,最后才原子切换 generation。候选被拒绝时,之前的 generation 继续服务,页面也会把这种状态与运行时确实不可用区分开来。“重新加载”始终恢复后端已保存的权威值,即使 revision 没有变化也会丢弃被拒绝的浏览器草稿。初始启动无法准备运行时时,Settings 路由仍可用于提交有效配置并激活首个 generation。陈旧浏览器 revision 不会覆盖较新的保存结果,而是返回冲突;刷新后再重试。只读 Settings 提供方允许查看和健康检查,但禁用保存。
|
|
275
|
+
|
|
276
|
+
“运行健康检查”只执行本地检查。“测试连接”是显式操作,会把已配置 Credential 发送到 `GET /models`;它不会上传图片,也不会创建 completion。插件加载和普通 Settings 读取不会发送该请求。
|
|
277
|
+
|
|
278
|
+
健康检查、连接测试以及插件/上游版本检查属于 Web Settings 管理能力,而不是模型工具,因此其 schema 永远不会占用 agent 请求上下文。
|
|
279
|
+
|
|
280
|
+
## 产物与展示
|
|
281
|
+
|
|
282
|
+
会生成产物的工具只能写入 `<workspace>/.dsh-vision-toolkit/artifacts`,写入形式为单个已验证文件或原子提交的运行目录。每个模型可见产物描述都包含路径、文件名、MIME 类型、种类、说明、来源工具、预览意图和字节数,因此 Headless agent 无需浏览器支持,也能在后续调用中复用该路径。提交 trace SVG 前,运行时会把它作为 XML 解析:允许标准声明与注释,但拒绝 doctype、格式错误或多根文档、非 SVG namespace,以及上游报告与实际路径数/字节数不一致的结果。
|
|
283
|
+
|
|
284
|
+
存在 Web HTTP 宿主时,仅供展示的元数据会加入带签名的预览和下载能力 URL,而不改变规范工具结果。每次读取都会重新验证签名、managed 根目录围栏、路径组件、普通文件状态、大小、可用时的 device/inode 身份、扩展名和 MIME。SVG 响应使用禁止外部资源的 sandbox CSP,客户端通过 sandbox iframe 渲染。没有 HTTP 宿主时,同一张卡片保留 `openFile` 提供的“打开文件”能力,并显示产物描述,不会伪造无法访问的 URL。
|
|
285
|
+
|
|
286
|
+
## 使用方式
|
|
287
|
+
|
|
288
|
+
### 基础调用
|
|
289
|
+
|
|
290
|
+
```text
|
|
291
|
+
vision_glance images=["screenshot.png"] query="What error is shown?"
|
|
292
|
+
vision_ground image="screenshot.png" target="the send button" preview=true
|
|
293
|
+
vision_detect image="screenshot.png" category="buttons" preview=true
|
|
294
|
+
vision_crop image="screenshot.png" region="1067,841,1108,881"
|
|
295
|
+
vision_trace image="icon.png" color=true output="icon.svg"
|
|
296
|
+
vision_pixel_diff original="reference.png" rebuilt="actual.png" runName="comparison"
|
|
297
|
+
vision_long_screenshot_ocr image="page.png" mode="general" jobs=2
|
|
298
|
+
vision_extract_foreground image="logo.png" mode="color"
|
|
299
|
+
vision_dominant_colors image="screen.png" region="0,0,600,300" top=8
|
|
300
|
+
vision_html_screenshot source="implementation.html" width=1200 height=720
|
|
301
|
+
```
|
|
302
|
+
|
|
303
|
+
常见工作流包括 `vision_ground` → `vision_crop` → `vision_glance`、`vision_ground` → `vision_crop` → `vision_trace`,以及参考图 → `vision_html_screenshot` → `vision_pixel_diff`。Grounding 和 detection 坐标始终使用原图像素(`x1/y1/x2/y2`)。
|
|
304
|
+
|
|
305
|
+
### UI 还原示例
|
|
306
|
+
|
|
307
|
+
已提交的 [UI 还原示例](examples/ui-restoration/README.md) 通过 `vision_html_screenshot` 渲染参考页面、故意不准确的初版实现和最终实现,再通过 `vision_pixel_diff` 比较两个候选结果:
|
|
308
|
+
|
|
309
|
+
```sh
|
|
310
|
+
npm run example:ui-restoration
|
|
311
|
+
npm run example:ui-restoration:write
|
|
312
|
+
```
|
|
313
|
+
|
|
314
|
+
已提交证据记录初版差异为 `6.04%`,有 6 个非零最差区域;最终差异为 `0%`,没有非零最差区域。Check 模式会复现工具调用路径并验证已提交资源;write 模式会有意刷新证据。
|
|
315
|
+
|
|
316
|
+
## 安全与执行模型
|
|
317
|
+
|
|
318
|
+
- 输入相对会话工作区和配置的 `allowedDirs` 解析;realpath containment 阻止路径穿越和符号链接逃逸。
|
|
319
|
+
- 每张图片都会在远程请求前由 Pillow 解码,并校验字节、像素、尺寸以及扩展名与内容是否一致。不支持或过大的图片会在上传前失败。
|
|
320
|
+
- 输出使用真实 managed 目标目录中的随机 staging 文件或目录,拒绝符号链接,并只在格式与契约验证通过后提交。
|
|
321
|
+
- 远程视觉提示词明确将图片中的文字和指令归类为不可信内容。原生工具描述与打包 skill 同样要求文本 agent 只把衍生描述、标签和 OCR 当作视觉证据,而不是可执行指令。
|
|
322
|
+
- 所有上游进程都通过 `ctx.subprocess` 使用 argv 向量,继承调用方取消信号,共享一个完整操作硬截止时间,并随操作终止,不会继续在后台运行。插件释放会在注销对应工具前中止活动调用。
|
|
323
|
+
- 一个活动会话只保留最近一次成功的 `vision_glance` 结果。只有图片内容、问题/OCR 模式、区域、端点、模型、语言和 Credential 都未改变时,紧接着的重复调用才会复用该结果;失败调用和其他会话绝不共享此条目。
|
|
324
|
+
- 模型可见数据仅包含文本、数字、坐标、结构化 JSON 和文件描述。工具调用/结果可以从会话日志重建;浏览器预览只属于展示元数据。
|
|
325
|
+
- 指标包含工具名、总耗时/上游耗时、有界图片数量/字节/像素、缓存命中、模型和错误类别;不包含 base64、鉴权头、密钥或无界上游输出。
|
|
326
|
+
|
|
327
|
+
`vision_html_screenshot` 只接受已授权的本地 `.html` 或 `.htm` 文件,在固定适配器中禁用网络,并使用 `--headless=new`、`--use-mock-keychain`、`--incognito` 和系统临时目录内的唯一 `--user-data-dir` 启动 Chrome 系浏览器。每次调用后都会删除该 profile,因此无头渲染不会接触用户日常 Chrome Profile 或 macOS 登录钥匙串。
|
|
328
|
+
|
|
329
|
+
## 故障排查
|
|
330
|
+
|
|
331
|
+
| 症状 | 解决方法 |
|
|
332
|
+
|---|---|
|
|
333
|
+
| `Model "..." does not support image input. (attachment-error)` | 图片走了 DSH 的模型原生附件通道,纯文本模型会在 Skill 或 Vision Toolkit 运行前拒绝该轮。请使用 DSH Paste Input 的附件按钮、粘贴或拖放流程,让文件先复制到会话工作区并以路径形式进入消息,再调用 `/vision-tools`。安装或升级任一浏览器插件后,需要重启 Web Profile 并刷新页面。 |
|
|
334
|
+
| Credential 显示缺失 | 执行 `dsh credentials set <REF>`,确认 `provider.credential` 指向该引用,再重新运行健康检查。本地工具不需要它。 |
|
|
335
|
+
| 运行时准备失败 | 查看 Settings 中的运行时错误,检查 Python 3.11+、软件包缓存/网络、磁盘权限和精确 external 固定版本。修正候选后再保存;当前 generation 不受影响。 |
|
|
336
|
+
| 找不到 Chrome | 安装 Chrome、Chromium 或 Edge,或让其中一个可被运行环境发现。只有 `vision_html_screenshot` 不可用。 |
|
|
337
|
+
| macOS 弹出钥匙串对话框 | 确认安装的是当前构建产物,且没有遗留的外部 `html_shot`/headless Chrome 进程。当前启动使用 mock keychain 和一次性 profile;取消对话框,不要重置登录钥匙串。 |
|
|
338
|
+
| 输入或输出路径被拒绝 | 把文件移入会话工作区,或有意将真实目录加入 `allowedDirs`;移除会逃逸的符号链接。输出参数只接受文件名,不接受绝对路径或嵌套路径。 |
|
|
339
|
+
| 视觉服务返回 401/403 | 替换 Credential 值,或选择正确的引用和端点。错误内容保持脱敏。 |
|
|
340
|
+
| 视觉服务返回 429 | 等待提供方限流窗口结束后重试,或降低 `concurrency`。插件不会静默切换提供方。 |
|
|
341
|
+
| 操作超时或被取消 | 在 1000-600000 毫秒范围内提高 `timeoutMs`、减少图片/分块工作量,或在取消后重新执行。子进程/请求会随操作停止。 |
|
|
342
|
+
| Settings 保存冲突 | 重新加载分区以取得当前 revision,重新应用目标修改,再次保存。 |
|
|
343
|
+
| Settings 只读 | 更换活动 Settings 提供方,或编辑其拥有的 Profile 配置;插件不能绕过提供方可写性。 |
|
|
344
|
+
| 无法预览产物 | 使用“打开文件”或模型可见路径。只有 Web HTTP 路由已挂载时才存在预览/下载 URL。 |
|
|
345
|
+
|
|
346
|
+
## 开发与验证
|
|
347
|
+
|
|
348
|
+
```sh
|
|
349
|
+
pnpm install --frozen-lockfile --trust-lockfile
|
|
350
|
+
pnpm run verify:portable
|
|
351
|
+
pnpm run build
|
|
352
|
+
pnpm test
|
|
353
|
+
pnpm run example:ui-restoration
|
|
354
|
+
pnpm pack --dry-run
|
|
355
|
+
```
|
|
356
|
+
|
|
357
|
+
`pnpm run verify:portable` 是不依赖外部开发包的可移植验证门禁:验证上游快照、package 元数据与 exports、已提交 JavaScript 语法、README 链接和图片、必需的开源门面文件、social preview 尺寸以及 dry-run tarball。完整 TypeScript 构建和测试会在这个独立 checkout 中直接使用 lockfile 锁定的 DSH `0.1.0-rc.6` registry 包;客户端构建还通过独立 compiler face 验证这些包的公开 exports,不使用内部路径 alias。PATH 中存在兼容的 `dsh` 与 `pnpm` 时会执行真实 Profile 验收,CI 会强制要求该路径,而不会静默跳过。
|
|
358
|
+
|
|
359
|
+
`pnpm run build` 会先验证 vendored manifest,再生成 JavaScript、声明文件和 loader 兼容 Web 客户端。本包提交 `lib/`,因此从 checkout 安装时不要求消费方构建。无真实 Key 的真实 Profile 测试会安装到干净 `DSH_HOME`、启动 Headless、通过真实工具调用执行全部五个 P0 工具和具有代表性的 P1 本地/远程工具、验证禁用与重新启用行为,并卸载 Bundle。每项 P0/P1 需求对应的实现与验证位置见[需求追踪参考](docs/requirements-traceability/README.md)。
|
|
360
|
+
|
|
361
|
+
更新上游快照时只能执行 `pnpm run upstream:sync -- <checkout>`,检查源码和许可证,重新生成 manifest,并在同一变更中更新适配器兼容性测试和已提交 `lib/`。运行时绝不拉取上游 `main`。
|
|
362
|
+
|
|
363
|
+
## 项目状态与范围
|
|
364
|
+
|
|
365
|
+
版本 `0.1.4` 是当前公开 npm 发布。P0 和 P1 是本包的产品承诺。P2 是设计门槛:至少一个独立插件消费内部能力形态前,不发布稳定 `ctx.visionToolkit` 服务、能力发现 API 或提供方生态。Web 上传、拖拽、摄像头/视频/音频/文档输入、交互式标注框编辑、GUI 自动点击、远程服务集群、模型路由、模型投票和跨会话视觉缓存不属于当前产品范围。
|
|
366
|
+
|
|
367
|
+
## 社区与关于
|
|
368
|
+
|
|
369
|
+
- 提交代码、协议或上游快照变更前,请先阅读 [CONTRIBUTING.md](CONTRIBUTING.md)。
|
|
370
|
+
- 可复现缺陷、范围明确的功能建议和使用问题请提交到 [GitHub Issues](https://github.com/Anionex/dsh-vision-toolkit/issues);如何选择渠道见 [SUPPORT.md](SUPPORT.md)。
|
|
371
|
+
- 安全漏洞必须按 [SECURITY.md](SECURITY.md) 私下报告,不要创建公开 Issue。
|
|
372
|
+
- 版本与兼容性变化记录在 [CHANGELOG.md](CHANGELOG.md)。
|
|
373
|
+
- 可选赞助方式与用途见 [FUNDING.md](FUNDING.md);赞助不购买路线图优先级或私有支持。
|
|
374
|
+
- 通用工具箱、跨 Harness 接入、视觉任务 playbook 和官方实跑案例请访问上游[项目网站](https://agent-vision.anionex.me)与[代码仓库](https://github.com/Anionex/agent-vision-toolkit)。
|
|
375
|
+
- 如果 `agent-vision-toolkit` 的算法或方法节省了时间,欢迎为上游 star、分享、贡献或赞助;DSH 专属缺陷和集成需求请提交到本仓库。
|
|
376
|
+
|
|
377
|
+
[`agent-vision-toolkit`](https://github.com/Anionex/agent-vision-toolkit) 由 [Anionex](https://anionex.me/) 创建。本仓库维护它面向 DeepSeek Harness 的原生集成:DSH 侧负责生命周期、安全、结构化 schema、Credentials、产物和 Web 展示;视觉算法与可复用 playbook 继续由上游项目维护。
|
|
378
|
+
|
|
379
|
+
如果你想了解我后续的更多工作,欢迎在 [X](https://x.com/anion_ex) 或 [GitHub](https://github.com/Anionex) 关注我。
|
|
380
|
+
|
|
381
|
+
## 许可证
|
|
382
|
+
|
|
383
|
+
插件采用 MIT 许可。打包的 `agent-vision-toolkit` 快照在 `vendor/agent-vision-toolkit/LICENSE` 保留上游 MIT 许可证,并继续作为视觉算法的唯一实现。
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
package/assets/hero.png
ADDED
|
Binary file
|
|
Binary file
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# Upstream reference images
|
|
2
|
+
|
|
3
|
+
These files are README-sized derivatives of official [`Anionex/agent-vision-toolkit`](https://github.com/Anionex/agent-vision-toolkit) reference assets. They document upstream use cases; this DSH integration does not claim to have rerun or reproduced those results.
|
|
4
|
+
|
|
5
|
+
The source files exist at the packaged runtime pin [`c27d1a300962b553c0884993c575cd3e819465ce`](https://github.com/Anionex/agent-vision-toolkit/tree/c27d1a300962b553c0884993c575cd3e819465ce) and were confirmed byte-identical in upstream commit [`7eafd51e7e62bd14f72627c69f2c11601c508f88`](https://github.com/Anionex/agent-vision-toolkit/tree/7eafd51e7e62bd14f72627c69f2c11601c508f88) on August 11, 2026.
|
|
6
|
+
|
|
7
|
+
| Local derivative | Upstream source |
|
|
8
|
+
|---|---|
|
|
9
|
+
| `infographic-reference.webp` | `assets/infographic-restore-reference.png` |
|
|
10
|
+
| `infographic-result.webp` | `assets/infographic-restore-result.png` |
|
|
11
|
+
| `ui-sketch.webp` | `assets/ui-restore-sketch.png` |
|
|
12
|
+
| `ui-result.webp` | `assets/ui-restore-result.png` |
|
|
13
|
+
| `image-qa.webp` | `assets/effect-1.jpg` |
|
|
14
|
+
| `screenshot-debugging.webp` | `assets/effect-2.jpg` |
|
|
15
|
+
|
|
16
|
+
Each derivative preserves the complete frame, limits the longest edge to 1200 pixels, removes metadata, and uses WebP quality 90 for repository delivery. The originals and visual algorithms remain covered by the upstream [MIT License](https://github.com/Anionex/agent-vision-toolkit/blob/c27d1a300962b553c0884993c575cd3e819465ce/LICENSE).
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
package/cordis.patch.yml
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
1
|
+
# dsh-vision-toolkit bundle patch: mounts the plugin into a profile layer stack.
|
|
2
|
+
# Runtime defaults live at the plugin's config boundary. Users override them in
|
|
3
|
+
# their profile patch row with the same id.
|
|
4
|
+
- insert:
|
|
5
|
+
- id: vision-toolkit
|
|
6
|
+
name: '@anionex/dsh-vision-toolkit'
|
|
Binary file
|
|
@@ -0,0 +1,6 @@
|
|
|
1
|
+
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
|
2
|
+
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
|
3
|
+
# after editing either side, bring the other along and re-record with:
|
|
4
|
+
# pnpm run verify-translation-pairing --write dsh-vision-toolkit/docs/requirements-traceability/README.md
|
|
5
|
+
README.md: 881cdb364929b3017d184d946193d77a8712c2e2
|
|
6
|
+
README.zh.md: e5153d28f266762a19b2f35e8e88bfa08d979673
|
|
@@ -0,0 +1,75 @@
|
|
|
1
|
+
# Vision Toolkit requirements traceability
|
|
2
|
+
|
|
3
|
+
English | [中文](README.zh.md)
|
|
4
|
+
|
|
5
|
+
This reference maps the DSH Vision Toolkit product brief's committed P0/P1 requirements to their owning implementation, automated coverage, and runnable acceptance evidence. A row marked **Delivered** is part of the package contract; P2/P3 rows document intentional product boundaries rather than unfinished P0/P1 work.
|
|
6
|
+
|
|
7
|
+
## P0 product requirements
|
|
8
|
+
|
|
9
|
+
| Requirement | Status | Owning implementation | Verification |
|
|
10
|
+
|---|---|---|---|
|
|
11
|
+
| P0-1 Standard Profile Bundle | **Delivered** | [`package.json`](../../package.json), [`cordis.patch.yml`](../../cordis.patch.yml), [`src/index.ts`](../../src/index.ts), committed `lib/` | [`tests/package-layout.spec.ts`](../../tests/package-layout.spec.ts), [`tests/profile-install.e2e.spec.ts`](../../tests/profile-install.e2e.spec.ts), clean `--dump-config`, Web/Headless boot, disable, and removal checks |
|
|
12
|
+
| P0-2 Upstream reuse | **Delivered** | [`src/version.ts`](../../src/version.ts), [`src/runtime-install.ts`](../../src/runtime-install.ts), [`scripts/upstream-manifest.mjs`](../../scripts/upstream-manifest.mjs), [`scripts/sync-upstream.mjs`](../../scripts/sync-upstream.mjs), [`vendor/agent-vision-toolkit/UPSTREAM_MANIFEST.json`](../../vendor/agent-vision-toolkit/UPSTREAM_MANIFEST.json) | [`tests/upstream.spec.ts`](../../tests/upstream.spec.ts), [`tests/runtime-install.spec.ts`](../../tests/runtime-install.spec.ts), build-time manifest verification |
|
|
13
|
+
| P0-3 Native tools | **Delivered** | Independent definitions in [`src/tools.ts`](../../src/tools.ts), Agent-scoped publication in [`src/exposure.ts`](../../src/exposure.ts), shared execution in [`src/runtime.ts`](../../src/runtime.ts), upstream adapter in [`src/upstream.ts`](../../src/upstream.ts) | [`tests/tools.spec.ts`](../../tests/tools.spec.ts), [`tests/runtime.spec.ts`](../../tests/runtime.spec.ts), [`tests/upstream.spec.ts`](../../tests/upstream.spec.ts), [`tests/vision-prompt-guard.spec.ts`](../../tests/vision-prompt-guard.spec.ts), real-profile progressive calls for all five P0 tools |
|
|
14
|
+
| P0-4 Configuration and Credentials | **Delivered** | [`src/config.ts`](../../src/config.ts), per-operation resolution in [`src/runtime.ts`](../../src/runtime.ts), isolated environment construction in [`src/upstream.ts`](../../src/upstream.ts) | [`tests/config.spec.ts`](../../tests/config.spec.ts), Credential/error/redaction cases in [`tests/runtime.spec.ts`](../../tests/runtime.spec.ts) and [`tests/errors.spec.ts`](../../tests/errors.spec.ts) |
|
|
15
|
+
| P0-5 Skill lifecycle | **Delivered** | Readiness ordering, lifecycle abort, and global disposer in [`src/index.ts`](../../src/index.ts); per-Agent activation, restoration, and teardown in [`src/exposure.ts`](../../src/exposure.ts); bundled content in [`src/skill.ts`](../../src/skill.ts) | Agent isolation, native/direct/Code Mode activation, Session restoration, in-flight cancellation, and disposal cases in [`tests/tools.spec.ts`](../../tests/tools.spec.ts); progressive exposure plus disable/uninstall paths in [`tests/profile-install.e2e.spec.ts`](../../tests/profile-install.e2e.spec.ts) |
|
|
16
|
+
| P0-6 Text-only model results | **Delivered** | JSON output schemas and pure renderers in [`src/tools.ts`](../../src/tools.ts); canonical Artifact descriptors in [`src/artifacts.ts`](../../src/artifacts.ts); schema visibility derived from durable Skill-load evidence in [`src/exposure.ts`](../../src/exposure.ts) | Tool schema/presentation assertions in [`tests/tools.spec.ts`](../../tests/tools.spec.ts), request-by-request schema and model-facing transcript assertions in [`tests/profile-install.e2e.spec.ts`](../../tests/profile-install.e2e.spec.ts) |
|
|
17
|
+
| P0-7 Stable errors | **Delivered** | [`src/errors.ts`](../../src/errors.ts), boundary validation in [`src/paths.ts`](../../src/paths.ts), [`src/runtime.ts`](../../src/runtime.ts), and [`src/upstream.ts`](../../src/upstream.ts) | [`tests/errors.spec.ts`](../../tests/errors.spec.ts), [`tests/paths.spec.ts`](../../tests/paths.spec.ts), parser, timeout, cancellation, credential, and capacity cases in runtime/upstream tests |
|
|
18
|
+
| P0-8 Tests and documentation | **Delivered** | Bilingual [`README.md`](../../README.md), package tests, managed runtime, example, and committed build output | `pnpm run build`, `pnpm test`, `pnpm pack --dry-run`, translation/Markdown gates, and the keyless clean-profile e2e |
|
|
19
|
+
|
|
20
|
+
## P1 product requirements
|
|
21
|
+
|
|
22
|
+
| Requirement | Status | Owning implementation | Verification |
|
|
23
|
+
|---|---|---|---|
|
|
24
|
+
| P1-1 Artifact delivery | **Delivered** | Descriptor creation in [`src/artifacts.ts`](../../src/artifacts.ts), fenced atomic paths in [`src/paths.ts`](../../src/paths.ts), signed capability delivery in [`src/artifact-access.ts`](../../src/artifact-access.ts) | [`tests/artifacts.spec.ts`](../../tests/artifacts.spec.ts), [`tests/artifact-access.spec.ts`](../../tests/artifact-access.spec.ts), Artifact-producing runtime/profile tests |
|
|
25
|
+
| P1-2 Extended tools | **Delivered** | `vision_pixel_diff`, `vision_long_screenshot_ocr`, `vision_extract_foreground`, `vision_dominant_colors`, and `vision_html_screenshot` in [`src/tools.ts`](../../src/tools.ts) and [`src/runtime.ts`](../../src/runtime.ts) | P1 parser and runtime cases in [`tests/upstream.spec.ts`](../../tests/upstream.spec.ts) and [`tests/runtime.spec.ts`](../../tests/runtime.spec.ts); pixel-diff and long-OCR real-profile calls |
|
|
26
|
+
| P1-3 Dedicated Web presentation | **Delivered** | Browser plugin and dedicated cards in [`src/client/index.tsx`](../../src/client/index.tsx); presentation-only capability metadata in [`src/artifact-access.ts`](../../src/artifact-access.ts) | [`tests/client.spec.ts`](../../tests/client.spec.ts), safe preview tests in [`tests/artifact-access.spec.ts`](../../tests/artifact-access.spec.ts), Web visual/console QA |
|
|
27
|
+
| P1-4 Health checks | **Delivered** | Health/version runtime contracts in [`src/runtime.ts`](../../src/runtime.ts), same-origin Web actions in [`src/web.ts`](../../src/web.ts), and the Settings surface in [`src/client/index.tsx`](../../src/client/index.tsx); these administrative diagnostics are deliberately absent from the model tool registry | Health cases in [`tests/runtime.spec.ts`](../../tests/runtime.spec.ts), explicit-connection behavior in [`tests/web.spec.ts`](../../tests/web.spec.ts), and model-tool absence in [`tests/tools.spec.ts`](../../tests/tools.spec.ts) |
|
|
28
|
+
| P1-5 Settings | **Delivered** | Namespace/config in [`src/config.ts`](../../src/config.ts), prepare-before-swap manager in [`src/runtime-manager.ts`](../../src/runtime-manager.ts), private same-origin route in [`src/web.ts`](../../src/web.ts), section in [`src/client/index.tsx`](../../src/client/index.tsx) | [`tests/runtime-manager.spec.ts`](../../tests/runtime-manager.spec.ts), [`tests/web.spec.ts`](../../tests/web.spec.ts), [`tests/client.spec.ts`](../../tests/client.spec.ts), clean Web-profile save/restart QA |
|
|
29
|
+
| P1-6 Install and upgrade experience | **Delivered** | Bundle lifecycle in [`src/index.ts`](../../src/index.ts), content-addressed managed runtime in [`src/runtime-install.ts`](../../src/runtime-install.ts), generation manager in [`src/runtime-manager.ts`](../../src/runtime-manager.ts) | Runtime interruption/concurrency tests, package-layout test, clean-profile lifecycle, persisted Settings and failed-candidate retention checks |
|
|
30
|
+
|
|
31
|
+
## Cross-cutting requirements
|
|
32
|
+
|
|
33
|
+
| Area | Contract and evidence |
|
|
34
|
+
|---|---|
|
|
35
|
+
| Security | [`src/paths.ts`](../../src/paths.ts) confines reads and writes by realpath; [`src/runtime.ts`](../../src/runtime.ts) decodes and limits images before upload and structurally parses trace SVG before commit; [`src/upstream.ts`](../../src/upstream.ts) marks image text/instructions as untrusted in every direct and long-OCR visual-model prompt; [`src/tools.ts`](../../src/tools.ts) and [`src/skill.ts`](../../src/skill.ts) tell the text agent to treat derived output as evidence, not commands; [`src/artifact-access.ts`](../../src/artifact-access.ts) revalidates signed files on every read and sandboxes SVG; [`src/errors.ts`](../../src/errors.ts) redacts secrets. Path, symlink, format, size, XML/doctype, prompt guard, token-forgery, file-replacement, and CSP cases are automated. |
|
|
36
|
+
| Portability | Runtime and browser calls use argv vectors and Node filesystem/process APIs rather than POSIX shell composition. Managed preparation has `uv` and `venv`/pip paths; Python and Chrome discovery include platform-specific candidates. Package tests avoid machine-local dependency specifiers, and the checked-in fixture/profile flow is keyless. |
|
|
37
|
+
| Performance and cancellation | [`src/runtime.ts`](../../src/runtime.ts) applies one hard deadline, propagates `AbortSignal`, bounds per-session concurrency, rejects oversized decoded images before remote I/O, deduplicates repeated inputs within one glance operation, and keeps a one-entry content/configuration-keyed cache for the last successful identical glance in each live Session. [`src/index.ts`](../../src/index.ts) aborts plugin-owned calls before unregistering tools. Timeout, caller/plugin cancellation, cache-hit/miss, semaphore, and independent-session behavior are automated. |
|
|
38
|
+
| Observability | [`src/runtime.ts`](../../src/runtime.ts) logs bounded tool name, outcome, total/upstream duration, image count/bytes/pixels, cache hits, model, and error category while excluding base64, credentials, headers, and unbounded upstream output. |
|
|
39
|
+
| Model-context economy | The profile-global runtime publishes one small bootstrap, while [`src/exposure.ts`](../../src/exposure.ts) mounts the ten execution schemas only in an Agent that loads `vision-tools` and hides the bootstrap afterward. Other Agents remain unchanged. Health, connection, and version diagnostics stay exclusively on the Web Settings seam and never become model tools. Unit tests cover Agent isolation and restoration; every real-profile tool flow asserts the initial and activated schema sets. |
|
|
40
|
+
| Maintainability | Every tool calls one [`VisionToolkitRuntime`](../../src/runtime.ts); DSH-specific adaptation remains in [`src/tools.ts`](../../src/tools.ts), [`src/exposure.ts`](../../src/exposure.ts), [`src/index.ts`](../../src/index.ts), [`src/web.ts`](../../src/web.ts), and [`src/client/index.tsx`](../../src/client/index.tsx), while the pinned upstream remains the sole algorithm source. Runtime readiness, Skill/bootstrap publication, and Agent-scoped schemas belong to one lifecycle generation. |
|
|
41
|
+
| HTML screenshot isolation | The pinned screenshot guard uses a disposable Chrome profile, `--use-mock-keychain`, and `--incognito`, removes the profile after each call, and keeps network disabled. [`tests/html-screenshot-guard.spec.ts`](../../tests/html-screenshot-guard.spec.ts) captures the actual argv and cleanup behavior. |
|
|
42
|
+
| UI restoration acceptance | [`scripts/ui-restoration-example.ts`](../../scripts/ui-restoration-example.ts) performs reference → HTML render → pixel diff through the real runtime. [`examples/ui-restoration/README.md`](../../examples/ui-restoration/README.md) owns the reproducible procedure and checked-in evidence; [`tests/ui-restoration-example.spec.ts`](../../tests/ui-restoration-example.spec.ts) enforces it. |
|
|
43
|
+
|
|
44
|
+
## Error and lifecycle scenarios
|
|
45
|
+
|
|
46
|
+
| Scenario | Expected behavior | Evidence |
|
|
47
|
+
|---|---|---|
|
|
48
|
+
| Missing/invalid image, format, region, or path | Reject before upstream execution with an input, capacity, or path-safe error | [`tests/paths.spec.ts`](../../tests/paths.spec.ts), [`tests/runtime.spec.ts`](../../tests/runtime.spec.ts) |
|
|
49
|
+
| Missing Credential | Local tools remain usable; remote tools and explicit connection tests report a redacted configuration/service action | [`tests/runtime.spec.ts`](../../tests/runtime.spec.ts), [`tests/web.spec.ts`](../../tests/web.spec.ts) |
|
|
50
|
+
| 401/403, 429, timeout, malformed output, or cancellation | Return a stable actionable category, preserve bounded diagnostics, and stop the request/subprocess | [`tests/errors.spec.ts`](../../tests/errors.spec.ts), [`tests/runtime.spec.ts`](../../tests/runtime.spec.ts), [`tests/upstream.spec.ts`](../../tests/upstream.spec.ts) |
|
|
51
|
+
| Runtime unavailable during initial load | Register no Skill, activation bootstrap, or Agent-scoped tools; keep Web Settings available for repair | [`src/index.ts`](../../src/index.ts), lifecycle tests in [`tests/tools.spec.ts`](../../tests/tools.spec.ts) and [`tests/web.spec.ts`](../../tests/web.spec.ts) |
|
|
52
|
+
| Runtime candidate fails during live update | Preserve the current serving generation and stored usable configuration | [`tests/runtime-manager.spec.ts`](../../tests/runtime-manager.spec.ts), [`tests/web.spec.ts`](../../tests/web.spec.ts) |
|
|
53
|
+
| Concurrent Settings candidates complete out of order | The newer ticket wins; an obsolete slower candidate cannot activate | [`tests/runtime-manager.spec.ts`](../../tests/runtime-manager.spec.ts) |
|
|
54
|
+
| Disable, re-enable, or uninstall | Active plugin-owned calls are cancelled, Agent-scoped tools, bootstrap, and Skill disappear and return as one lifecycle unit, and uninstall removes the bundle layer | [`tests/tools.spec.ts`](../../tests/tools.spec.ts), [`tests/profile-install.e2e.spec.ts`](../../tests/profile-install.e2e.spec.ts) |
|
|
55
|
+
|
|
56
|
+
## P2 and P3 boundary
|
|
57
|
+
|
|
58
|
+
| Scope | Status | Decision |
|
|
59
|
+
|---|---|---|
|
|
60
|
+
| P2 stable `ctx.visionToolkit` service and capability discovery | **Deferred by design** | The product brief requires at least one independent plugin consumer before stabilizing this API. `VisionToolkitRuntime` remains package-internal, so P0/P1 can evolve without creating a false compatibility promise. |
|
|
61
|
+
| P2 provider ecosystem | **Deferred by design** | The package supports its pinned upstream and one configured OpenAI-compatible vision endpoint; it does not prebuild an unused provider registry. |
|
|
62
|
+
| P3 exploratory inputs and automation | **Out of scope** | Upload/drag-and-drop, camera/video/audio/document ingestion, interactive annotations, automatic clicking, remote clusters, model routing/voting, and cross-session caches are not part of this release contract. |
|
|
63
|
+
|
|
64
|
+
## Reproducible verification
|
|
65
|
+
|
|
66
|
+
Run package checks from `dsh-vision-toolkit/`:
|
|
67
|
+
|
|
68
|
+
```sh
|
|
69
|
+
pnpm run build
|
|
70
|
+
pnpm test
|
|
71
|
+
pnpm run example:ui-restoration
|
|
72
|
+
pnpm pack --dry-run
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
The release acceptance pass additionally runs the repository's plugin checker, installs the package into clean Web and Headless `DSH_HOME` directories, inspects `--dump-config`, exercises local and keyless mock-backed remote tool calls, verifies Settings persistence and live switching, checks dedicated Web cards with zero browser-console errors, disables/re-enables the row, and removes the package. These runtime checks complement the automated tests; they do not replace them.
|