@anionex/dsh-vision-toolkit 0.1.7 → 0.1.9

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (51) hide show
  1. package/README.i18n.yaml +2 -2
  2. package/README.md +34 -17
  3. package/README.zh.md +32 -7
  4. package/assets/community-group-qr.png +0 -0
  5. package/assets/vision-model-test.png +0 -0
  6. package/docs/requirements-traceability/README.i18n.yaml +2 -2
  7. package/docs/requirements-traceability/README.md +2 -2
  8. package/docs/requirements-traceability/README.zh.md +2 -2
  9. package/lib/client.js +361 -28
  10. package/lib/client.js.map +1 -1
  11. package/lib/config.js +14 -0
  12. package/lib/config.js.map +1 -1
  13. package/lib/image-input-variants.js +615 -0
  14. package/lib/image-input-variants.js.map +1 -0
  15. package/lib/index.js +8 -1
  16. package/lib/index.js.map +1 -1
  17. package/lib/paste-images.js +6 -0
  18. package/lib/paste-images.js.map +1 -1
  19. package/lib/runtime.js +37 -3
  20. package/lib/runtime.js.map +1 -1
  21. package/lib/types/client/index.d.ts +16 -5
  22. package/lib/types/client/index.d.ts.map +1 -1
  23. package/lib/types/client/paste-images.d.ts +69 -0
  24. package/lib/types/client/paste-images.d.ts.map +1 -1
  25. package/lib/types/config.d.ts +26 -0
  26. package/lib/types/config.d.ts.map +1 -1
  27. package/lib/types/image-input-variants.d.ts +153 -0
  28. package/lib/types/image-input-variants.d.ts.map +1 -0
  29. package/lib/types/index.d.ts.map +1 -1
  30. package/lib/types/paste-images.d.ts +35 -0
  31. package/lib/types/paste-images.d.ts.map +1 -1
  32. package/lib/types/runtime.d.ts +5 -2
  33. package/lib/types/runtime.d.ts.map +1 -1
  34. package/lib/types/web-request.d.ts +7 -0
  35. package/lib/types/web-request.d.ts.map +1 -1
  36. package/lib/types/web.d.ts +17 -2
  37. package/lib/types/web.d.ts.map +1 -1
  38. package/lib/web-request.js +11 -2
  39. package/lib/web-request.js.map +1 -1
  40. package/lib/web.js +88 -5
  41. package/lib/web.js.map +1 -1
  42. package/package.json +2 -1
  43. package/src/client/index.tsx +53 -14
  44. package/src/client/paste-images.tsx +331 -15
  45. package/src/config.ts +40 -0
  46. package/src/image-input-variants.ts +688 -0
  47. package/src/index.ts +18 -1
  48. package/src/paste-images.ts +39 -0
  49. package/src/runtime.ts +42 -3
  50. package/src/web-request.ts +12 -2
  51. package/src/web.ts +90 -4
package/README.i18n.yaml CHANGED
@@ -2,5 +2,5 @@
2
2
  # side as of the last confirmed-consistent state. Both languages carry equal authority;
3
3
  # after editing either side, bring the other along and re-record with:
4
4
  # pnpm run verify-translation-pairing --write dsh-vision-dark-theme/README.md
5
- README.md: ab7ac8d402f6da738a0d70cad3170c3dc0d970cb
6
- README.zh.md: 6d6da22d56babde78acffa0b7865b941476f0134
5
+ README.md: 026285db53ab41b5e70527060c692bf9f2785541
6
+ README.zh.md: df34e29906568cd13f235dad56abe652a659a620
package/README.md CHANGED
@@ -2,9 +2,11 @@
2
2
 
3
3
  # DSH Vision Toolkit
4
4
 
5
+ [![Recommended by dshfind](https://img.shields.io/badge/recommended%20by-dshfind-FFD700?style=flat-square)](https://dshfind.com/en/plugins/Anionex/dsh-vision-toolkit)
6
+ [![dshfind score: 94 — highest-rated plugin](https://img.shields.io/badge/dshfind%20score-94%20%7C%20highest--rated%20plugin-5B4CF0?style=flat-square)](https://dshfind.com/en/plugins/Anionex/dsh-vision-toolkit)
5
7
  [![X (Twitter)](https://img.shields.io/badge/-@anion__ex-000000?style=flat-square&logo=x&logoColor=white)](https://x.com/anion_ex)
6
- [![Release v0.1.7](https://img.shields.io/badge/release-v0.1.7-5B4CF0?style=flat-square)](https://github.com/Anionex/dsh-vision-toolkit/releases/tag/v0.1.7)
7
- [![Verified: 168 tests](https://img.shields.io/badge/verified-168%20tests-2EA44F?style=flat-square)](tests)
8
+ [![Release v0.1.9](https://img.shields.io/badge/release-v0.1.9-5B4CF0?style=flat-square)](https://github.com/Anionex/dsh-vision-toolkit/releases/tag/v0.1.9)
9
+ [![Verified: 230 tests](https://img.shields.io/badge/verified-230%20tests-2EA44F?style=flat-square)](tests)
8
10
  [![License: MIT](https://img.shields.io/badge/license-MIT-0B7285?style=flat-square)](LICENSE)
9
11
  [![Node.js](https://img.shields.io/badge/Node.js-%5E22.19%20%7C%20%3E%3D24-339933?style=flat-square&logo=nodedotjs&logoColor=white)](package.json)
10
12
  [![Python](https://img.shields.io/badge/Python-3.11%2B-3776AB?style=flat-square&logo=python&logoColor=white)](runtime/requirements.lock)
@@ -99,7 +101,7 @@ dsh --profile headless --dump-config | grep vision-toolkit
99
101
 
100
102
  Legacy profiles must use `nodeLinker: hoisted` and `autoInstallPeers: false` in their `pnpm-workspace.yaml`. An updated DSH launcher repairs these owned settings before `dsh plugin` runs; when using an older launcher, set them before installation so pnpm does not assemble a second Harness dependency graph inside the profile.
101
103
 
102
- Restart a running Web profile, open **Settings → Vision Toolkit**, select a DSH Credential for remote tools, and explicitly run **Test connection**. In a conversation, make an image available as a workspace path, invoke `/vision-tools`, and ask the Agent to call a specific `vision_*` tool. Local crop, trace, pixel, color, foreground, and HTML operations do not require a visual API credential.
104
+ Restart a running Web profile, open **Settings → Vision Toolkit**, select a DSH Credential for remote tools, run **Test API connection**, and then run **Test vision model** to verify one real image request. In a conversation, make an image available as a workspace path, invoke `/vision-tools`, and ask the Agent to call a specific `vision_*` tool. Local crop, trace, pixel, color, foreground, and HTML operations do not require a visual API credential.
103
105
 
104
106
  ## How it works
105
107
 
@@ -145,6 +147,14 @@ Runtime readiness is profile-wide, but the ten visual execution schemas are Agen
145
147
 
146
148
  Health checks, connection testing, and plugin/upstream version inspection are administrative Web Settings operations. `vision_toolkit_health` and `vision_toolkit_version` are not model tools and never enter an Agent's schema, including after visual-tool activation.
147
149
 
150
+ ## Image-input variants for text-only models
151
+
152
+ Text-only model routes get sibling model-selector entries named `<model> (Vision Toolkit)` under a matching provider group. A variant declares image input, so pasted images keep the native attachment flow — composer thumbnail, durable session image, and history rendering — and the plugin rewrites every image block into a Vision Toolkit description only on the wire to the model, before the request reaches the upstream route. The session log is untouched; replay and the UI keep the real image.
153
+
154
+ A variant is registered automatically for every model the host positively declares text-only (for example the DeepSeek chat family). Paste handling is automatic: when the current model is confirmed text-only and its variant exists, the browser integration switches the session to the variant by itself (a short notice names the new model) and the paste then keeps the native flow; no manual model change is needed. The host's verdict uses the exact model route the browser read from the live model catalog, with the selector label as fallback; unconfirmed or image-capable routes always keep the native flow, and a text-only model without a variant (for example when variants are disabled) keeps the paste-to-path takeover, which copies the image into the session workspace and inserts its path as text.
155
+
156
+ Description conversion needs the configured vision provider and its credential; when the runtime is not ready or a read fails, the wire block degrades to an explanatory note instead of failing the turn. Disable variants with `imageInputVariants.enabled: false`, restrict the wrapped routes with `imageInputVariants.providers`, or keep the paste-to-path behavior for text-only models with `imageInputVariants.autoSwitch: false`.
157
+
148
158
  ## Requirements
149
159
 
150
160
  - DeepSeek Harness with a Web or Headless profile and `pnpm` available to `dsh plugin`.
@@ -191,7 +201,7 @@ dsh plugin --profile web remove @dsh-external/dsh-vision-toolkit
191
201
  dsh plugin --profile web add @anionex/dsh-vision-toolkit
192
202
  ```
193
203
 
194
- After restarting, Settings → Vision should report plugin version **0.1.7**.
204
+ After restarting, Settings → Vision should report plugin version **0.1.9**.
195
205
 
196
206
  For a registry installation, update the dependency through the profile package manager:
197
207
 
@@ -233,6 +243,10 @@ The bundle defaults to the managed runtime. A profile patch can override the pro
233
243
  runtime:
234
244
  mode: managed
235
245
  allowedDirs: []
246
+ imageInputVariants:
247
+ enabled: true
248
+ providers: []
249
+ autoSwitch: true
236
250
  ```
237
251
 
238
252
  ### Configuration fields
@@ -254,6 +268,9 @@ The bundle defaults to the managed runtime. A profile patch can override the pro
254
268
  | `runtime.agentVisionToolkitPath` | unset | Required in `external` mode; exported exact snapshot or clean pinned Git checkout |
255
269
  | `runtime.python` | unset | Optional Python 3.11+ bootstrap/interpreter override |
256
270
  | `allowedDirs` | `[]` | Additional realpath-resolved input roots; the session workspace is always allowed |
271
+ | `imageInputVariants.enabled` | `true` | Register image-input variant entries for text-only model routes in the model selector |
272
+ | `imageInputVariants.providers` | `[]` | Restrict wrapped upstream routes by provider id; empty wraps every eligible route |
273
+ | `imageInputVariants.autoSwitch` | `true` | Automatically switch a text-only session to its image-input variant on paste, so the image keeps the native flow; `false` keeps the paste-to-path takeover for text-only models |
257
274
 
258
275
  ### Credentials
259
276
 
@@ -284,7 +301,7 @@ The Web profile registers a Vision Toolkit Settings section for the provider URL
284
301
 
285
302
  `Save and apply` validates the complete value, prepares the candidate Python/upstream runtime, commits the Settings revision, and only then atomically switches generations. A rejected candidate leaves the previous generation serving and is reported separately from a genuinely unavailable runtime. `Reload` always restores the authoritative saved value, even when its revision did not change, so a rejected browser draft is discarded. If initial startup cannot prepare a runtime, the Settings route remains available so a valid configuration can make the first generation operational. A stale browser revision receives a conflict instead of overwriting a newer save; reload before retrying. A read-only Settings provider allows inspection and health checks but disables saves.
286
303
 
287
- `Run health check` performs local checks only. `Test connection` is an explicit action that sends the configured Credential to `GET /models`; OpenAI uses Bearer authentication, while Anthropic uses `x-api-key` and `anthropic-version`. The check uploads no image and creates no completion. Plugin load and ordinary Settings reads never make that request.
304
+ `Run health check` performs local checks only. `Test API connection` is an explicit action that sends the configured Credential to `GET /models`; OpenAI uses Bearer authentication, while Anthropic uses `x-api-key` and `anthropic-version`. That lightweight probe uploads no image and creates no completion. `Test vision model` separately sends the bundled `assets/vision-model-test.png` through the same multimodal runtime path as `vision_glance`; it creates one real completion and is the authoritative check that the selected endpoint, credential, model, protocol, and upstream account can process images. The Vision model health card displays a dedicated `Verified`, `Not tested`, or `Test failed` tag, so an HTTP 200 response from `/models` is not presented as a successful image test. Plugin load and ordinary Settings reads never make either request.
288
305
 
289
306
  Health, connection testing, and plugin/upstream version inspection are administrative Web Settings capabilities rather than model-facing tools, so their schemas never occupy an agent request.
290
307
 
@@ -324,24 +341,16 @@ npm run example:ui-restoration:write
324
341
 
325
342
  The committed evidence records an initial `6.04%` difference across six non-zero worst regions and a final `0%` difference with no non-zero worst region. Check mode reproduces the tool path and verifies the committed assets; write mode intentionally refreshes the evidence.
326
343
 
327
- ## Security and execution model
344
+ ## Communication group
328
345
 
329
- - Inputs resolve against the session workspace and configured `allowedDirs`; realpath containment prevents traversal and symlink escape.
330
- - Pillow decodes every image before a remote request and verifies bytes, pixels, dimensions, and extension/content agreement. Unsupported or oversized images fail before upload.
331
- - Outputs use random staging files or directories inside the real managed destination, reject symbolic links, and commit only after format and contract validation.
332
- - Remote vision prompts explicitly classify text and instructions visible inside images as untrusted content. The native tool descriptions and bundled skill likewise tell the text agent to treat derived descriptions, labels, and OCR as visual evidence rather than executable instructions.
333
- - All upstream processes use argv vectors through `ctx.subprocess`, inherit caller cancellation, share one hard operation deadline, and terminate with the operation instead of continuing in the background. Plugin disposal aborts active calls before unregistering their tools.
334
- - One live Session retains only the most recent successful `vision_glance` result. An immediate repeat reuses it only when image content, query/OCR mode, region, endpoint, model, language, and Credential are unchanged; failures and other Sessions never share the entry.
335
- - Model-visible data is text, numbers, coordinates, structured JSON, and file descriptors. Tool calls/results remain reconstructable from the Session log; browser previews are presentation metadata only.
336
- - Metrics include tool name, total/upstream duration, bounded image counts/bytes/pixels, cache hits, model, and error category; they exclude base64, authentication headers, secrets, and unbounded upstream output.
346
+ <img width="254" height="328" alt="image" src="https://github.com/user-attachments/assets/63c25c69-c3ba-4c47-8dee-98d60fe3954d" />
337
347
 
338
- `vision_html_screenshot` accepts only authorized local `.html` or `.htm` files, disables network access in the pinned adapter, and launches a Chrome-family browser with `--headless=new`, `--use-mock-keychain`, `--incognito`, and a unique `--user-data-dir` under the system temporary directory. The profile is removed after every call, so headless rendering does not touch the user's daily Chrome profile or macOS login keychain.
339
348
 
340
349
  ## Troubleshooting
341
350
 
342
351
  | Symptom | Resolution |
343
352
  |---|---|
344
- | `Model "..." does not support image input. (attachment-error)` | The image used DSH's native model-attachment channel, so a text-only model rejected the turn before the Skill or Vision Toolkit could run. Use DSH Paste Input's attachment button, paste, or drop flow so the file is copied into the session workspace and represented by a path, then invoke `/vision-tools`. Restart the Web profile and reload the page after installing or upgrading either browser plugin. |
353
+ | `Model "..." does not support image input. (attachment-error)` | The image used DSH's native model-attachment channel, so a text-only model rejected the turn before the Skill or Vision Toolkit could run. With image-input variants enabled this is rare: pasting normally auto-switches the session to the `<model> (Vision Toolkit)` variant. If variants are disabled or auto-switch is off, use DSH Paste Input's attachment button, paste, or drop flow so the file is copied into the session workspace and represented by a path, then invoke `/vision-tools`. Restart the Web profile and reload the page after installing or upgrading either browser plugin. |
345
354
  | Credential reported missing | Paste the key into Web Settings **API key**, keep the advanced **Credential name** aligned with `provider.credential`, save, then rerun health. Headless deployments can provision the same reference in `$DSH_HOME/.credentials.yaml`. Local-only tools do not need it. |
346
355
  | Runtime preparation fails | Read the Settings runtime error, verify Python 3.11+, package-cache/network access, disk permissions, and the exact external pin. Save only after correcting the candidate; the active generation remains intact. |
347
356
  | Chrome is not found | Install Chrome, Chromium, or Edge or configure an environment where one is discoverable. Only `vision_html_screenshot` is unavailable. |
@@ -373,7 +382,7 @@ Update the upstream snapshot only through `pnpm run upstream:sync -- <checkout>`
373
382
 
374
383
  ## Project status and scope
375
384
 
376
- Version `0.1.4` is the current public npm release. P0 and P1 are product commitments in this package. P2 is a design threshold: no stable `ctx.visionToolkit` service, capability-discovery API, or provider ecosystem is published until at least one independent plugin consumes the internal capability shape. Web upload, drag-and-drop, camera/video/audio/document ingestion, interactive box editing, automatic GUI clicking, service clusters, model routing, model voting, and cross-session vision caches remain outside the current product.
385
+ Version `0.1.9` is the current public npm release. P0 and P1 are product commitments in this package. P2 is a design threshold: no stable `ctx.visionToolkit` service, capability-discovery API, or provider ecosystem is published until at least one independent plugin consumes the internal capability shape. Web upload, drag-and-drop, camera/video/audio/document ingestion, interactive box editing, automatic GUI clicking, service clusters, model routing, model voting, and cross-session vision caches remain outside the current product.
377
386
 
378
387
  ## Community and About
379
388
 
@@ -392,3 +401,11 @@ If you would like to follow my future work, [follow me on X](https://x.com/anion
392
401
  ## License
393
402
 
394
403
  The plugin is MIT-licensed. The packaged `agent-vision-toolkit` snapshot retains its upstream MIT license in `vendor/agent-vision-toolkit/LICENSE` and remains the sole implementation of its visual algorithms.
404
+
405
+ ## Join the Community Group
406
+
407
+ You are welcome to join the `agent-vision-toolkit` community group to exchange usage tips, share feedback, and suggest improvements. Scan the QR code below to join.
408
+
409
+ <p align="center">
410
+ <img src="assets/community-group-qr.png" alt="QR code for the agent-vision-toolkit community group" width="320">
411
+ </p>
package/README.zh.md CHANGED
@@ -2,9 +2,11 @@
2
2
 
3
3
  # DSH Vision Toolkit
4
4
 
5
+ [![由 dshfind 推荐](https://img.shields.io/badge/%E7%94%B1%20dshfind-%E6%8E%A8%E8%8D%90-FFD700?style=flat-square)](https://dshfind.com/zh/plugins/Anionex/dsh-vision-toolkit)
6
+ [![dshfind 评分:94——最高分插件](https://img.shields.io/badge/dshfind%20%E8%AF%84%E5%88%86-94%20%7C%20%E6%9C%80%E9%AB%98%E5%88%86%E6%8F%92%E4%BB%B6-5B4CF0?style=flat-square)](https://dshfind.com/zh/plugins/Anionex/dsh-vision-toolkit)
5
7
  [![X (Twitter)](https://img.shields.io/badge/-@anion__ex-000000?style=flat-square&logo=x&logoColor=white)](https://x.com/anion_ex)
6
- [![Release v0.1.7](https://img.shields.io/badge/release-v0.1.7-5B4CF0?style=flat-square)](https://github.com/Anionex/dsh-vision-toolkit/releases/tag/v0.1.7)
7
- [![Verified: 168 tests](https://img.shields.io/badge/verified-168%20tests-2EA44F?style=flat-square)](tests)
8
+ [![Release v0.1.9](https://img.shields.io/badge/release-v0.1.9-5B4CF0?style=flat-square)](https://github.com/Anionex/dsh-vision-toolkit/releases/tag/v0.1.9)
9
+ [![Verified: 230 tests](https://img.shields.io/badge/verified-230%20tests-2EA44F?style=flat-square)](tests)
8
10
  [![License: MIT](https://img.shields.io/badge/license-MIT-0B7285?style=flat-square)](LICENSE)
9
11
  [![Node.js](https://img.shields.io/badge/Node.js-%5E22.19%20%7C%20%3E%3D24-339933?style=flat-square&logo=nodedotjs&logoColor=white)](package.json)
10
12
  [![Python](https://img.shields.io/badge/Python-3.11%2B-3776AB?style=flat-square&logo=python&logoColor=white)](runtime/requirements.lock)
@@ -99,7 +101,7 @@ dsh --profile headless --dump-config | grep vision-toolkit
99
101
 
100
102
  旧 Profile 的 `pnpm-workspace.yaml` 必须使用 `nodeLinker: hoisted` 和 `autoInstallPeers: false`。更新后的 DSH launcher 会在 `dsh plugin` 运行前修复这两个自有设置;使用旧 launcher 时,应在安装前手动设置,避免 pnpm 在 Profile 内组装第二套 Harness 依赖图。
101
103
 
102
- 安装后重启正在运行的 Web Profile,打开 **设置 → 视觉工具**,为远程工具选择 DSH Credential,并显式执行**测试连接**。在会话中把图片放进工作区路径,调用 `/vision-tools`,再让 Agent 使用明确的 `vision_*` 工具。本地裁剪、SVG、像素、颜色、前景和 HTML 操作不需要视觉 API Credential。
104
+ 安装后重启正在运行的 Web Profile,打开 **设置 → 视觉工具**,为远程工具选择 DSH Credential,先执行**测试 API 连接**,再执行**测试视觉模型**以验证一次真实图片请求。在会话中把图片放进工作区路径,调用 `/vision-tools`,再让 Agent 使用明确的 `vision_*` 工具。本地裁剪、SVG、像素、颜色、前景和 HTML 操作不需要视觉 API Credential。
103
105
 
104
106
  ## 工作原理
105
107
 
@@ -145,6 +147,14 @@ flowchart LR
145
147
 
146
148
  健康检查、连接测试以及插件/上游版本检查属于 Web Settings 管理操作。`vision_toolkit_health` 和 `vision_toolkit_version` 不是模型工具,即使视觉执行工具已经激活,也永远不会进入 Agent schema。
147
149
 
150
+ ## 纯文本模型的图片输入变体
151
+
152
+ 纯文本模型路由会获得同名的兄弟模型条目:`<模型名> (Vision Toolkit)`,挂在对应的提供方分组下。变体声明支持图片输入,因此粘贴的图片走原生附件流程——输入框缩略图、会话持久化图片与历史渲染全部保留——插件只在发往模型的请求链路上把每个图片块改写成 Vision Toolkit 描述文本,再转交上游路由。会话日志不被改动;回放与 UI 看到的始终是真实图片。
153
+
154
+ 插件会自动为宿主明确声明为纯文本的每个模型注册变体(例如 DeepSeek 对话家族)。粘贴处理是全自动的:当当前模型被确认为纯文本、且它的变体已注册时,浏览器端集成会自动把会话切换到变体(会有一条简短提示说明新模型名),随后粘贴走原生流程,无需手动切换模型。宿主依据浏览器从实时模型目录读到的精确模型路由来裁决,模型选择器标签作为兜底;无法确认或支持图片的路由一律保持原生流程,而"纯文本但没有变体"的模型(例如变体被关闭时)继续走"粘贴转路径":图片被复制进会话工作区,输入框里插入的是它的路径文本。
155
+
156
+ 描述转换需要已配置的视觉提供方及其 Credential;当运行时未就绪或读取失败时,请求链路上的图片块降级为说明文本,而不是让整轮失败。用 `imageInputVariants.enabled: false` 关闭变体,用 `imageInputVariants.providers` 限制被包装的路由,或用 `imageInputVariants.autoSwitch: false` 让纯文本模型继续走"粘贴转路径"。
157
+
148
158
  ## 运行要求
149
159
 
150
160
  - 启用 Web 或 Headless Profile 的 DeepSeek Harness,并确保 `dsh plugin` 可以使用 `pnpm`。
@@ -191,7 +201,7 @@ dsh plugin --profile web remove @dsh-external/dsh-vision-toolkit
191
201
  dsh plugin --profile web add @anionex/dsh-vision-toolkit
192
202
  ```
193
203
 
194
- 重启后,Settings → 视觉工具 应显示插件版本 **0.1.7**。
204
+ 重启后,Settings → 视觉工具 应显示插件版本 **0.1.9**。
195
205
 
196
206
  通过注册表安装时,使用 Profile 的包管理命令更新依赖:
197
207
 
@@ -233,6 +243,10 @@ Bundle 默认使用 managed 运行时。Profile patch 可以覆盖提供方与
233
243
  runtime:
234
244
  mode: managed
235
245
  allowedDirs: []
246
+ imageInputVariants:
247
+ enabled: true
248
+ providers: []
249
+ autoSwitch: true
236
250
  ```
237
251
 
238
252
  ### 配置字段
@@ -254,6 +268,9 @@ Bundle 默认使用 managed 运行时。Profile patch 可以覆盖提供方与
254
268
  | `runtime.agentVisionToolkitPath` | 未设置 | `external` 模式必填;必须是精确导出快照或固定 commit 的干净 Git checkout |
255
269
  | `runtime.python` | 未设置 | 可选的 Python 3.11+ 引导程序/解释器覆盖值 |
256
270
  | `allowedDirs` | `[]` | 额外的 realpath 解析输入根目录;会话工作区始终允许 |
271
+ | `imageInputVariants.enabled` | `true` | 为纯文本模型路由在模型选择器中注册图片输入变体条目 |
272
+ | `imageInputVariants.providers` | `[]` | 按提供方 id 限制被包装的上游路由;为空时包装所有符合条件的路由 |
273
+ | `imageInputVariants.autoSwitch` | `true` | 粘贴时自动把纯文本会话切换到其图片输入变体,让图片保持原生流程;`false` 时纯文本模型继续走"粘贴转路径" |
257
274
 
258
275
  ### Credential
259
276
 
@@ -284,7 +301,7 @@ Web Profile 会注册 Vision Toolkit Settings 分区,可配置提供方 URL、
284
301
 
285
302
  “保存并应用”会验证完整配置,准备候选 Python/上游运行时,提交 Settings revision,最后才原子切换 generation。候选被拒绝时,之前的 generation 继续服务,页面也会把这种状态与运行时确实不可用区分开来。“重新加载”始终恢复后端已保存的权威值,即使 revision 没有变化也会丢弃被拒绝的浏览器草稿。初始启动无法准备运行时时,Settings 路由仍可用于提交有效配置并激活首个 generation。陈旧浏览器 revision 不会覆盖较新的保存结果,而是返回冲突;刷新后再重试。只读 Settings 提供方允许查看和健康检查,但禁用保存。
286
303
 
287
- “运行健康检查”只执行本地检查。“测试连接”是显式操作,会把已配置 Credential 发送到 `GET /models`;OpenAI 使用 Bearer 认证,Anthropic 使用 `x-api-key` 与 `anthropic-version`。该检查不会上传图片,也不会创建 completion。插件加载和普通 Settings 读取不会发送该请求。
304
+ “运行健康检查”只执行本地检查。“测试 API 连接”是显式操作,会把已配置 Credential 发送到 `GET /models`;OpenAI 使用 Bearer 认证,Anthropic 使用 `x-api-key` 与 `anthropic-version`。这个轻量测试不会上传图片,也不会创建 completion。“测试视觉模型”会另行把插件自带的 `assets/vision-model-test.png` 通过与 `vision_glance` 相同的多模态运行路径发送出去;它会创建一次真实 completion,并用于权威确认所选端点、Credential、模型、协议和上游账户确实能够处理图片。视觉模型检查卡会单独显示“已实测”“未测试”或“测试失败”Tag,避免把 `/models` 返回 HTTP 200 误认为图片调用成功。插件加载和普通 Settings 读取不会发送这两类请求。
288
305
 
289
306
  健康检查、连接测试以及插件/上游版本检查属于 Web Settings 管理能力,而不是模型工具,因此其 schema 永远不会占用 agent 请求上下文。
290
307
 
@@ -341,7 +358,7 @@ npm run example:ui-restoration:write
341
358
 
342
359
  | 症状 | 解决方法 |
343
360
  |---|---|
344
- | `Model "..." does not support image input. (attachment-error)` | 图片走了 DSH 的模型原生附件通道,纯文本模型会在 Skill 或 Vision Toolkit 运行前拒绝该轮。请使用 DSH Paste Input 的附件按钮、粘贴或拖放流程,让文件先复制到会话工作区并以路径形式进入消息,再调用 `/vision-tools`。安装或升级任一浏览器插件后,需要重启 Web Profile 并刷新页面。 |
361
+ | `Model "..." does not support image input. (attachment-error)` | 图片走了 DSH 的模型原生附件通道,纯文本模型会在 Skill 或 Vision Toolkit 运行前拒绝该轮。启用图片输入变体时这很少发生:粘贴会自动把会话切换到 `<模型名> (Vision Toolkit)` 变体。若变体被关闭或自动切换被禁用,请使用 DSH Paste Input 的附件按钮、粘贴或拖放流程,让文件先复制到会话工作区并以路径形式进入消息,再调用 `/vision-tools`。安装或升级任一浏览器插件后,需要重启 Web Profile 并刷新页面。 |
345
362
  | Credential 显示缺失 | 在 Web 设置页的 **API 密钥** 中粘贴密钥,确认高级设置中的 **凭据名称** 与 `provider.credential` 一致,保存后重新运行健康检查。Headless 部署可以在 `$DSH_HOME/.credentials.yaml` 中预置同名引用。本地工具不需要它。 |
346
363
  | 运行时准备失败 | 查看 Settings 中的运行时错误,检查 Python 3.11+、软件包缓存/网络、磁盘权限和精确 external 固定版本。修正候选后再保存;当前 generation 不受影响。 |
347
364
  | 找不到 Chrome | 安装 Chrome、Chromium 或 Edge,或让其中一个可被运行环境发现。只有 `vision_html_screenshot` 不可用。 |
@@ -373,7 +390,7 @@ pnpm pack --dry-run
373
390
 
374
391
  ## 项目状态与范围
375
392
 
376
- 版本 `0.1.4` 是当前公开 npm 发布。P0 和 P1 是本包的产品承诺。P2 是设计门槛:至少一个独立插件消费内部能力形态前,不发布稳定 `ctx.visionToolkit` 服务、能力发现 API 或提供方生态。Web 上传、拖拽、摄像头/视频/音频/文档输入、交互式标注框编辑、GUI 自动点击、远程服务集群、模型路由、模型投票和跨会话视觉缓存不属于当前产品范围。
393
+ 版本 `0.1.9` 是当前公开 npm 发布。P0 和 P1 是本包的产品承诺。P2 是设计门槛:至少一个独立插件消费内部能力形态前,不发布稳定 `ctx.visionToolkit` 服务、能力发现 API 或提供方生态。Web 上传、拖拽、摄像头/视频/音频/文档输入、交互式标注框编辑、GUI 自动点击、远程服务集群、模型路由、模型投票和跨会话视觉缓存不属于当前产品范围。
377
394
 
378
395
  ## 社区与关于
379
396
 
@@ -392,3 +409,11 @@ pnpm pack --dry-run
392
409
  ## 许可证
393
410
 
394
411
  插件采用 MIT 许可。打包的 `agent-vision-toolkit` 快照在 `vendor/agent-vision-toolkit/LICENSE` 保留上游 MIT 许可证,并继续作为视觉算法的唯一实现。
412
+
413
+ ## 加入交流群
414
+
415
+ 欢迎加入 `agent-vision-toolkit` 项目交流群,交流使用经验、反馈问题并提出建议。请扫描下方二维码入群。
416
+
417
+ <p align="center">
418
+ <img src="assets/community-group-qr.png" alt="agent-vision-toolkit 项目交流群二维码" width="320">
419
+ </p>
Binary file
Binary file
@@ -2,5 +2,5 @@
2
2
  # side as of the last confirmed-consistent state. Both languages carry equal authority;
3
3
  # after editing either side, bring the other along and re-record with:
4
4
  # pnpm run verify-translation-pairing --write dsh-vision-toolkit/docs/requirements-traceability/README.md
5
- README.md: 881cdb364929b3017d184d946193d77a8712c2e2
6
- README.zh.md: e5153d28f266762a19b2f35e8e88bfa08d979673
5
+ README.md: 00e1bd317061adf14f86b3da7c61f0f9d02b96ea
6
+ README.zh.md: 813fbd33c5ca38c37605ba2dedef4b09fae72dc0
@@ -24,7 +24,7 @@ This reference maps the DSH Vision Toolkit product brief's committed P0/P1 requi
24
24
  | P1-1 Artifact delivery | **Delivered** | Descriptor creation in [`src/artifacts.ts`](../../src/artifacts.ts), fenced atomic paths in [`src/paths.ts`](../../src/paths.ts), signed capability delivery in [`src/artifact-access.ts`](../../src/artifact-access.ts) | [`tests/artifacts.spec.ts`](../../tests/artifacts.spec.ts), [`tests/artifact-access.spec.ts`](../../tests/artifact-access.spec.ts), Artifact-producing runtime/profile tests |
25
25
  | P1-2 Extended tools | **Delivered** | `vision_pixel_diff`, `vision_long_screenshot_ocr`, `vision_extract_foreground`, `vision_dominant_colors`, and `vision_html_screenshot` in [`src/tools.ts`](../../src/tools.ts) and [`src/runtime.ts`](../../src/runtime.ts) | P1 parser and runtime cases in [`tests/upstream.spec.ts`](../../tests/upstream.spec.ts) and [`tests/runtime.spec.ts`](../../tests/runtime.spec.ts); pixel-diff and long-OCR real-profile calls |
26
26
  | P1-3 Dedicated Web presentation | **Delivered** | Browser plugin and dedicated cards in [`src/client/index.tsx`](../../src/client/index.tsx); presentation-only capability metadata in [`src/artifact-access.ts`](../../src/artifact-access.ts) | [`tests/client.spec.ts`](../../tests/client.spec.ts), safe preview tests in [`tests/artifact-access.spec.ts`](../../tests/artifact-access.spec.ts), Web visual/console QA |
27
- | P1-4 Health checks | **Delivered** | Health/version runtime contracts in [`src/runtime.ts`](../../src/runtime.ts), same-origin Web actions in [`src/web.ts`](../../src/web.ts), and the Settings surface in [`src/client/index.tsx`](../../src/client/index.tsx); these administrative diagnostics are deliberately absent from the model tool registry | Health cases in [`tests/runtime.spec.ts`](../../tests/runtime.spec.ts), explicit-connection behavior in [`tests/web.spec.ts`](../../tests/web.spec.ts), and model-tool absence in [`tests/tools.spec.ts`](../../tests/tools.spec.ts) |
27
+ | P1-4 Health checks | **Delivered** | Health/version runtime contracts in [`src/runtime.ts`](../../src/runtime.ts), same-origin Web actions in [`src/web.ts`](../../src/web.ts), and the Settings surface in [`src/client/index.tsx`](../../src/client/index.tsx); Settings distinguishes the lightweight authenticated `GET /models` probe from an explicit real multimodal request using the bundled diagnostic image, and these administrative diagnostics are deliberately absent from the model tool registry | Health and real-model success/failure cases in [`tests/runtime.spec.ts`](../../tests/runtime.spec.ts), explicit API/model-test behavior in [`tests/web.spec.ts`](../../tests/web.spec.ts) and [`tests/client.spec.ts`](../../tests/client.spec.ts), plus model-tool absence in [`tests/tools.spec.ts`](../../tests/tools.spec.ts) |
28
28
  | P1-5 Settings | **Delivered** | Namespace/config in [`src/config.ts`](../../src/config.ts), prepare-before-swap manager in [`src/runtime-manager.ts`](../../src/runtime-manager.ts), private same-origin route in [`src/web.ts`](../../src/web.ts), section in [`src/client/index.tsx`](../../src/client/index.tsx) | [`tests/runtime-manager.spec.ts`](../../tests/runtime-manager.spec.ts), [`tests/web.spec.ts`](../../tests/web.spec.ts), [`tests/client.spec.ts`](../../tests/client.spec.ts), clean Web-profile save/restart QA |
29
29
  | P1-6 Install and upgrade experience | **Delivered** | Bundle lifecycle in [`src/index.ts`](../../src/index.ts), content-addressed managed runtime in [`src/runtime-install.ts`](../../src/runtime-install.ts), generation manager in [`src/runtime-manager.ts`](../../src/runtime-manager.ts) | Runtime interruption/concurrency tests, package-layout test, clean-profile lifecycle, persisted Settings and failed-candidate retention checks |
30
30
 
@@ -46,7 +46,7 @@ This reference maps the DSH Vision Toolkit product brief's committed P0/P1 requi
46
46
  | Scenario | Expected behavior | Evidence |
47
47
  |---|---|---|
48
48
  | Missing/invalid image, format, region, or path | Reject before upstream execution with an input, capacity, or path-safe error | [`tests/paths.spec.ts`](../../tests/paths.spec.ts), [`tests/runtime.spec.ts`](../../tests/runtime.spec.ts) |
49
- | Missing Credential | Local tools remain usable; remote tools and explicit connection tests report a redacted configuration/service action | [`tests/runtime.spec.ts`](../../tests/runtime.spec.ts), [`tests/web.spec.ts`](../../tests/web.spec.ts) |
49
+ | Missing Credential | Local tools remain usable; remote tools plus explicit API-connection and real-model tests report a redacted configuration/service action | [`tests/runtime.spec.ts`](../../tests/runtime.spec.ts), [`tests/web.spec.ts`](../../tests/web.spec.ts) |
50
50
  | 401/403, 429, timeout, malformed output, or cancellation | Return a stable actionable category, preserve bounded diagnostics, and stop the request/subprocess | [`tests/errors.spec.ts`](../../tests/errors.spec.ts), [`tests/runtime.spec.ts`](../../tests/runtime.spec.ts), [`tests/upstream.spec.ts`](../../tests/upstream.spec.ts) |
51
51
  | Runtime unavailable during initial load | Register no Skill, activation bootstrap, or Agent-scoped tools; keep Web Settings available for repair | [`src/index.ts`](../../src/index.ts), lifecycle tests in [`tests/tools.spec.ts`](../../tests/tools.spec.ts) and [`tests/web.spec.ts`](../../tests/web.spec.ts) |
52
52
  | Runtime candidate fails during live update | Preserve the current serving generation and stored usable configuration | [`tests/runtime-manager.spec.ts`](../../tests/runtime-manager.spec.ts), [`tests/web.spec.ts`](../../tests/web.spec.ts) |
@@ -24,7 +24,7 @@
24
24
  | P1-1 产物交付 | **已交付** | [`src/artifacts.ts`](../../src/artifacts.ts) 中的产物描述创建、[`src/paths.ts`](../../src/paths.ts) 中受围栏保护的原子路径、[`src/artifact-access.ts`](../../src/artifact-access.ts) 中的签名能力交付 | [`tests/artifacts.spec.ts`](../../tests/artifacts.spec.ts)、[`tests/artifact-access.spec.ts`](../../tests/artifact-access.spec.ts)、会生成产物的运行时/Profile 测试 |
25
25
  | P1-2 扩展工具 | **已交付** | [`src/tools.ts`](../../src/tools.ts) 和 [`src/runtime.ts`](../../src/runtime.ts) 中的 `vision_pixel_diff`、`vision_long_screenshot_ocr`、`vision_extract_foreground`、`vision_dominant_colors`、`vision_html_screenshot` | [`tests/upstream.spec.ts`](../../tests/upstream.spec.ts) 和 [`tests/runtime.spec.ts`](../../tests/runtime.spec.ts) 中的 P1 解析器/运行时用例;真实 Profile pixel-diff 和长截图 OCR 调用 |
26
26
  | P1-3 专用 Web 展示 | **已交付** | [`src/client/index.tsx`](../../src/client/index.tsx) 中的浏览器插件与专用卡片;[`src/artifact-access.ts`](../../src/artifact-access.ts) 中仅供展示的能力元数据 | [`tests/client.spec.ts`](../../tests/client.spec.ts)、[`tests/artifact-access.spec.ts`](../../tests/artifact-access.spec.ts) 中的安全预览测试、Web 视觉/Console QA |
27
- | P1-4 健康检查 | **已交付** | [`src/runtime.ts`](../../src/runtime.ts) 中的健康检查/版本运行时契约、[`src/web.ts`](../../src/web.ts) 中的同源 Web 操作,以及 [`src/client/index.tsx`](../../src/client/index.tsx) 中的 Settings 界面;这些管理诊断能力有意不进入模型工具注册表 | [`tests/runtime.spec.ts`](../../tests/runtime.spec.ts) 中的健康检查用例、[`tests/web.spec.ts`](../../tests/web.spec.ts) 中的显式连接行为,以及 [`tests/tools.spec.ts`](../../tests/tools.spec.ts) 中的模型工具缺席断言 |
27
+ | P1-4 健康检查 | **已交付** | [`src/runtime.ts`](../../src/runtime.ts) 中的健康检查/版本运行时契约、[`src/web.ts`](../../src/web.ts) 中的同源 Web 操作,以及 [`src/client/index.tsx`](../../src/client/index.tsx) 中的 Settings 界面;Settings 明确区分携带凭据的轻量 `GET /models` 探测与使用自带诊断图片的显式真实多模态请求,这些管理诊断能力有意不进入模型工具注册表 | [`tests/runtime.spec.ts`](../../tests/runtime.spec.ts) 中的健康检查及真实模型成功/失败用例、[`tests/web.spec.ts`](../../tests/web.spec.ts) 与 [`tests/client.spec.ts`](../../tests/client.spec.ts) 中的显式 API/模型测试行为,以及 [`tests/tools.spec.ts`](../../tests/tools.spec.ts) 中的模型工具缺席断言 |
28
28
  | P1-5 Settings | **已交付** | [`src/config.ts`](../../src/config.ts) 中的 namespace/配置、[`src/runtime-manager.ts`](../../src/runtime-manager.ts) 中的 prepare-before-swap manager、[`src/web.ts`](../../src/web.ts) 中的同源私有路由、[`src/client/index.tsx`](../../src/client/index.tsx) 中的 Settings 分区 | [`tests/runtime-manager.spec.ts`](../../tests/runtime-manager.spec.ts)、[`tests/web.spec.ts`](../../tests/web.spec.ts)、[`tests/client.spec.ts`](../../tests/client.spec.ts)、干净 Web Profile 保存/重启 QA |
29
29
  | P1-6 安装与升级体验 | **已交付** | [`src/index.ts`](../../src/index.ts) 中的 Bundle 生命周期、[`src/runtime-install.ts`](../../src/runtime-install.ts) 中的内容寻址 managed 运行时、[`src/runtime-manager.ts`](../../src/runtime-manager.ts) 中的 generation manager | 运行时中断/并发测试、软件包布局测试、干净 Profile 生命周期、Settings 持久化和失败候选保留检查 |
30
30
 
@@ -46,7 +46,7 @@
46
46
  | 场景 | 预期行为 | 证据 |
47
47
  |---|---|---|
48
48
  | 图片缺失/无效、格式、区域或路径错误 | 在执行上游前以输入、容量或路径安全错误拒绝 | [`tests/paths.spec.ts`](../../tests/paths.spec.ts)、[`tests/runtime.spec.ts`](../../tests/runtime.spec.ts) |
49
- | Credential 缺失 | 本地工具保持可用;远程工具和显式连接测试返回脱敏且可执行下一步的配置/服务结果 | [`tests/runtime.spec.ts`](../../tests/runtime.spec.ts)、[`tests/web.spec.ts`](../../tests/web.spec.ts) |
49
+ | Credential 缺失 | 本地工具保持可用;远程工具以及显式 API 连接/真实模型测试返回脱敏且可执行下一步的配置/服务结果 | [`tests/runtime.spec.ts`](../../tests/runtime.spec.ts)、[`tests/web.spec.ts`](../../tests/web.spec.ts) |
50
50
  | 401/403、429、超时、畸形输出或取消 | 返回稳定且可执行下一步的类别,保留有界诊断信息,并停止请求/子进程 | [`tests/errors.spec.ts`](../../tests/errors.spec.ts)、[`tests/runtime.spec.ts`](../../tests/runtime.spec.ts)、[`tests/upstream.spec.ts`](../../tests/upstream.spec.ts) |
51
51
  | 初始加载时运行时不可用 | 不注册 skill、激活引导工具或 Agent 级工具;保留 Web Settings 以供修复 | [`src/index.ts`](../../src/index.ts)、[`tests/tools.spec.ts`](../../tests/tools.spec.ts) 与 [`tests/web.spec.ts`](../../tests/web.spec.ts) 中的生命周期测试 |
52
52
  | 实时更新时运行时候选失败 | 保留当前服务 generation 和已存储的可用配置 | [`tests/runtime-manager.spec.ts`](../../tests/runtime-manager.spec.ts)、[`tests/web.spec.ts`](../../tests/web.spec.ts) |