@anionex/dsh-vision-toolkit 0.1.8 → 0.1.10
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.i18n.yaml +2 -2
- package/README.md +43 -34
- package/README.zh.md +43 -21
- package/assets/community-group-qr.png +0 -0
- package/assets/vision-model-test.png +0 -0
- package/docs/requirements-traceability/README.i18n.yaml +2 -2
- package/docs/requirements-traceability/README.md +2 -2
- package/docs/requirements-traceability/README.zh.md +2 -2
- package/lib/client.js +77 -20
- package/lib/client.js.map +1 -1
- package/lib/config.js +20 -11
- package/lib/config.js.map +1 -1
- package/lib/defaults.js +6 -0
- package/lib/defaults.js.map +1 -0
- package/lib/image-input-variants.js +204 -23
- package/lib/image-input-variants.js.map +1 -1
- package/lib/runtime.js +45 -5
- package/lib/runtime.js.map +1 -1
- package/lib/types/client/index.d.ts +17 -6
- package/lib/types/client/index.d.ts.map +1 -1
- package/lib/types/config.d.ts +3 -0
- package/lib/types/config.d.ts.map +1 -1
- package/lib/types/defaults.d.ts +6 -0
- package/lib/types/defaults.d.ts.map +1 -0
- package/lib/types/image-input-variants.d.ts +2 -2
- package/lib/types/image-input-variants.d.ts.map +1 -1
- package/lib/types/runtime.d.ts +6 -3
- package/lib/types/runtime.d.ts.map +1 -1
- package/lib/types/web.d.ts.map +1 -1
- package/lib/web.js +14 -3
- package/lib/web.js.map +1 -1
- package/package.json +1 -1
- package/src/client/index.tsx +76 -22
- package/src/config.ts +30 -11
- package/src/defaults.ts +6 -0
- package/src/image-input-variants.ts +226 -25
- package/src/runtime.ts +50 -6
- package/src/web.ts +13 -2
package/README.i18n.yaml
CHANGED
|
@@ -2,5 +2,5 @@
|
|
|
2
2
|
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
|
3
3
|
# after editing either side, bring the other along and re-record with:
|
|
4
4
|
# pnpm run verify-translation-pairing --write dsh-vision-dark-theme/README.md
|
|
5
|
-
README.md:
|
|
6
|
-
README.zh.md:
|
|
5
|
+
README.md: 2a05d467a5c0cff82900d6074220392553f5d972
|
|
6
|
+
README.zh.md: 07cff26ce48a7f0df6791ac9723debaeb723d5f4
|
package/README.md
CHANGED
|
@@ -2,10 +2,11 @@
|
|
|
2
2
|
|
|
3
3
|
# DSH Vision Toolkit
|
|
4
4
|
|
|
5
|
-
[](https://dshfind.com/en/plugins/Anionex/dsh-vision-toolkit)
|
|
6
|
+
[](https://dshfind.com/en/plugins/Anionex/dsh-vision-toolkit)
|
|
6
7
|
[](https://x.com/anion_ex)
|
|
7
|
-
[](https://github.com/Anionex/dsh-vision-toolkit/releases/tag/v0.1.10)
|
|
9
|
+
[](tests)
|
|
9
10
|
[](LICENSE)
|
|
10
11
|
[](package.json)
|
|
11
12
|
[](runtime/requirements.lock)
|
|
@@ -100,7 +101,15 @@ dsh --profile headless --dump-config | grep vision-toolkit
|
|
|
100
101
|
|
|
101
102
|
Legacy profiles must use `nodeLinker: hoisted` and `autoInstallPeers: false` in their `pnpm-workspace.yaml`. An updated DSH launcher repairs these owned settings before `dsh plugin` runs; when using an older launcher, set them before installation so pnpm does not assemble a second Harness dependency graph inside the profile.
|
|
102
103
|
|
|
103
|
-
Restart a running Web profile, open **Settings → Vision Toolkit**,
|
|
104
|
+
Restart a running Web profile, open **Settings → Vision Toolkit**, and run **Test API connection** followed by **Test vision model**. New installations use the built-in free Moondream provider automatically, so no API key or DSH Credential is required. To use another provider, edit the endpoint/model/protocol and provide its DSH Credential. In a conversation, make an image available as a workspace path, invoke `/vision-tools`, and ask the Agent to call a specific `vision_*` tool. Local crop, trace, pixel, color, foreground, and HTML operations do not require a visual API credential.
|
|
105
|
+
|
|
106
|
+
## Community Group
|
|
107
|
+
|
|
108
|
+
Join the `agent-vision-toolkit` community group to exchange usage tips, share feedback, and suggest improvements.
|
|
109
|
+
|
|
110
|
+
<p align="center">
|
|
111
|
+
<img src="assets/community-group-qr.png" alt="QR code for the agent-vision-toolkit community group" width="260">
|
|
112
|
+
</p>
|
|
104
113
|
|
|
105
114
|
## How it works
|
|
106
115
|
|
|
@@ -148,18 +157,18 @@ Health checks, connection testing, and plugin/upstream version inspection are ad
|
|
|
148
157
|
|
|
149
158
|
## Image-input variants for text-only models
|
|
150
159
|
|
|
151
|
-
Text-only model routes get sibling model-selector entries named `<model> (Vision Toolkit)` under a matching provider group. A variant declares image input, so pasted images keep the native attachment flow — composer thumbnail, durable session image, and history rendering — and the plugin rewrites every image block into a Vision Toolkit description only on the wire to the model, before the request reaches the upstream route. The session log is untouched; replay and the UI keep the real image.
|
|
160
|
+
Text-only model routes get sibling model-selector entries named `<model> (Vision Toolkit)` under a matching provider group. A variant declares image input, so pasted images keep the native attachment flow — composer thumbnail, durable session image, and history rendering — and the plugin rewrites every image block into a Vision Toolkit description only on the wire to the model, before the request reaches the upstream route. The vision prompt is focus-hinted with the latest user or assistant intent, using the same role, instruction, image-text policy, and `[vision model description]` channel markers as `agent-vision-toolkit`; the model receives task-relevant evidence instead of a broad generic description. The session log is untouched; replay and the UI keep the real image.
|
|
152
161
|
|
|
153
162
|
A variant is registered automatically for every model the host positively declares text-only (for example the DeepSeek chat family). Paste handling is automatic: when the current model is confirmed text-only and its variant exists, the browser integration switches the session to the variant by itself (a short notice names the new model) and the paste then keeps the native flow; no manual model change is needed. The host's verdict uses the exact model route the browser read from the live model catalog, with the selector label as fallback; unconfirmed or image-capable routes always keep the native flow, and a text-only model without a variant (for example when variants are disabled) keeps the paste-to-path takeover, which copies the image into the session workspace and inserts its path as text.
|
|
154
163
|
|
|
155
|
-
Description conversion needs the configured vision provider and its credential; when the runtime is not ready or a read fails, the wire block degrades to
|
|
164
|
+
Description conversion needs the configured vision provider and its credential; when the runtime is not ready or a read fails, the wire block degrades to the upstream-compatible `[vision unavailable: ...]` note instead of failing the turn. The bridge does not treat injected context files as the current user intent, and it uses the latest assistant paragraph when a tool-fetched image is being described. Disable variants with `imageInputVariants.enabled: false`, restrict the wrapped routes with `imageInputVariants.providers`, or keep the paste-to-path behavior for text-only models with `imageInputVariants.autoSwitch: false`.
|
|
156
165
|
|
|
157
166
|
## Requirements
|
|
158
167
|
|
|
159
168
|
- DeepSeek Harness with a Web or Headless profile and `pnpm` available to `dsh plugin`.
|
|
160
169
|
- Python 3.11 or newer. Managed mode creates an isolated environment, so users do not install the upstream CLI or Python packages manually.
|
|
161
170
|
- Network access on the first managed-runtime activation unless the exact packages in `runtime/requirements.lock` are already available in the configured package cache.
|
|
162
|
-
-
|
|
171
|
+
- The built-in free Moondream provider is ready for `vision_glance`, `vision_ground`, `vision_detect`, and non-split-only long-screenshot OCR. A DSH Credential is required only when a custom OpenAI-compatible or Anthropic endpoint is configured. Local tools remain usable without either provider.
|
|
163
172
|
- Chrome, Chromium, or Edge only for `vision_html_screenshot`; all other tools remain available when no supported browser is installed.
|
|
164
173
|
- PNG, JPEG, GIF, or WebP inputs inside the session workspace or an explicitly configured `allowedDirs` root.
|
|
165
174
|
|
|
@@ -200,7 +209,7 @@ dsh plugin --profile web remove @dsh-external/dsh-vision-toolkit
|
|
|
200
209
|
dsh plugin --profile web add @anionex/dsh-vision-toolkit
|
|
201
210
|
```
|
|
202
211
|
|
|
203
|
-
After restarting, Settings → Vision should report plugin version **0.1.
|
|
212
|
+
After restarting, Settings → Vision should report plugin version **0.1.10**. The built-in free provider is selected automatically; custom providers still use the configured DSH Credential.
|
|
204
213
|
|
|
205
214
|
For a registry installation, update the dependency through the profile package manager:
|
|
206
215
|
|
|
@@ -228,16 +237,16 @@ The bundle defaults to the managed runtime. A profile patch can override the pro
|
|
|
228
237
|
- id: vision-toolkit
|
|
229
238
|
config:
|
|
230
239
|
provider:
|
|
231
|
-
baseUrl: https://
|
|
232
|
-
credential:
|
|
233
|
-
model:
|
|
240
|
+
baseUrl: https://vision.anionex.me/v1
|
|
241
|
+
credential: ANIONEX_FREE_VISION
|
|
242
|
+
model: moondream-3.1
|
|
234
243
|
protocol: openai
|
|
235
244
|
anthropicThinking: omit
|
|
236
245
|
userAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
|
|
237
246
|
language: zh
|
|
238
247
|
timeoutMs: 60000
|
|
239
|
-
maxImageBytes:
|
|
240
|
-
maxImagePixels:
|
|
248
|
+
maxImageBytes: 4194304
|
|
249
|
+
maxImagePixels: 20000000
|
|
241
250
|
concurrency: 4
|
|
242
251
|
runtime:
|
|
243
252
|
mode: managed
|
|
@@ -252,16 +261,16 @@ The bundle defaults to the managed runtime. A profile patch can override the pro
|
|
|
252
261
|
|
|
253
262
|
| Field | Default | Contract |
|
|
254
263
|
|---|---|---|
|
|
255
|
-
| `provider.baseUrl` | `https://
|
|
256
|
-
| `provider.credential` | `
|
|
257
|
-
| `provider.model` | `
|
|
264
|
+
| `provider.baseUrl` | `https://vision.anionex.me/v1` | Built-in free OpenAI-compatible endpoint; custom providers may use another base URL, normalized without trailing slashes |
|
|
265
|
+
| `provider.credential` | `ANIONEX_FREE_VISION` | Read-only built-in reference for the free service; custom providers use a DSH Credential reference, never a secret value |
|
|
266
|
+
| `provider.model` | `moondream-3.1` | Multimodal model name sent to remote tools |
|
|
258
267
|
| `provider.protocol` | `openai` | `openai` sends Chat Completions requests; `anthropic` sends native Messages requests |
|
|
259
268
|
| `provider.anthropicThinking` | `omit` | Anthropic thinking field. `omit` sends no thinking field and has the broadest compatibility. Use `disabled` or `adaptive` only when the selected model documents that mode; restore `omit` first if the provider returns HTTP 400. |
|
|
260
269
|
| `provider.userAgent` | browser-compatible default | User-Agent sent by vision requests and explicit connection tests; override it for provider or proxy compatibility |
|
|
261
270
|
| `language` | `zh` | Vision output language: `zh` or `en` |
|
|
262
271
|
| `timeoutMs` | `60000` | Whole-operation deadline, 1000-600000 ms; each tool may request a narrower override |
|
|
263
|
-
| `maxImageBytes` | `
|
|
264
|
-
| `maxImagePixels` | `
|
|
272
|
+
| `maxImageBytes` | `4194304` | Encoded-byte limit per input image; the built-in free service accepts up to 4 MiB |
|
|
273
|
+
| `maxImagePixels` | `20000000` | Decoded-pixel limit per input image; the built-in free service accepts up to 20,000,000 pixels |
|
|
265
274
|
| `concurrency` | `4` | In-flight operations per session, 1-16 |
|
|
266
275
|
| `runtime.mode` | `managed` | `managed` uses the packaged snapshot; `external` accepts only the exact pin |
|
|
267
276
|
| `runtime.agentVisionToolkitPath` | unset | Required in `external` mode; exported exact snapshot or clean pinned Git checkout |
|
|
@@ -273,10 +282,23 @@ The bundle defaults to the managed runtime. A profile patch can override the pro
|
|
|
273
282
|
|
|
274
283
|
### Credentials
|
|
275
284
|
|
|
276
|
-
The
|
|
285
|
+
The built-in free provider uses the fixed `ANIONEX_FREE_VISION` reference and does not accept or store a user API key. If you change the endpoint, model, or protocol to a custom provider, the write-only **API key** field unlocks; saving a non-empty value writes it under the advanced **Credential name** reference. Headless deployments can pre-provision that custom reference in `$DSH_HOME/.credentials.yaml`.
|
|
277
286
|
|
|
278
287
|
Settings store only the reference, never the value. The browser does not receive a stored value, and a successful save clears the field instead of echoing it. Remote operations resolve the reference once per call and inject the value only into that subprocess environment. The plugin excludes user `.env` files, checkout `.env` files, `PYTHONPATH`, `PYTHONHOME`, `VIRTUAL_ENV`, and user site-packages so ambient Python or upstream configuration cannot override the selected DSH provider. Logs, errors, tool results, Artifact metadata, and Settings responses never contain the secret.
|
|
279
288
|
|
|
289
|
+
### Built-in free service limits
|
|
290
|
+
|
|
291
|
+
The public service is shared and intended as a zero-configuration default, not an unlimited private endpoint. Limits are enforced by the proxy and returned as OpenAI-style errors with a reason code and readable message; rate-limit responses also include `Retry-After` and request-quota headers.
|
|
292
|
+
|
|
293
|
+
| Limit | Current value |
|
|
294
|
+
|---|---:|
|
|
295
|
+
| Per client | 30 requests per UTC day |
|
|
296
|
+
| Global service | 120 requests per UTC day |
|
|
297
|
+
| Burst | 6 requests per 60 seconds |
|
|
298
|
+
| Image bytes | 4 MiB per image |
|
|
299
|
+
| Decoded pixels | 20,000,000 per image |
|
|
300
|
+
| Output | 512 tokens maximum |
|
|
301
|
+
|
|
280
302
|
### Managed and external runtimes
|
|
281
303
|
|
|
282
304
|
Managed mode verifies `vendor/agent-vision-toolkit/UPSTREAM_MANIFEST.json`, prefers `uv`, falls back to `venv` plus pip, installs exact versions from `runtime/requirements.lock`, coordinates concurrent preparation with a heartbeat lock, and publishes a staged environment only after all probes pass.
|
|
@@ -300,7 +322,7 @@ The Web profile registers a Vision Toolkit Settings section for the provider URL
|
|
|
300
322
|
|
|
301
323
|
`Save and apply` validates the complete value, prepares the candidate Python/upstream runtime, commits the Settings revision, and only then atomically switches generations. A rejected candidate leaves the previous generation serving and is reported separately from a genuinely unavailable runtime. `Reload` always restores the authoritative saved value, even when its revision did not change, so a rejected browser draft is discarded. If initial startup cannot prepare a runtime, the Settings route remains available so a valid configuration can make the first generation operational. A stale browser revision receives a conflict instead of overwriting a newer save; reload before retrying. A read-only Settings provider allows inspection and health checks but disables saves.
|
|
302
324
|
|
|
303
|
-
`Run health check` performs local checks only. `Test connection` is an explicit action that sends the configured Credential to `GET /models`; OpenAI uses Bearer authentication, while Anthropic uses `x-api-key` and `anthropic-version`.
|
|
325
|
+
`Run health check` performs local checks only. `Test API connection` is an explicit action that sends the configured Credential to `GET /models`; OpenAI uses Bearer authentication, while Anthropic uses `x-api-key` and `anthropic-version`. That lightweight probe uploads no image and creates no completion. `Test vision model` separately sends the bundled `assets/vision-model-test.png` through the same multimodal runtime path as `vision_glance`; it creates one real completion and is the authoritative check that the selected endpoint, credential, model, protocol, and upstream account can process images. The Vision model health card displays a dedicated `Verified`, `Not tested`, or `Test failed` tag, so an HTTP 200 response from `/models` is not presented as a successful image test. Plugin load and ordinary Settings reads never make either request.
|
|
304
326
|
|
|
305
327
|
Health, connection testing, and plugin/upstream version inspection are administrative Web Settings capabilities rather than model-facing tools, so their schemas never occupy an agent request.
|
|
306
328
|
|
|
@@ -340,19 +362,6 @@ npm run example:ui-restoration:write
|
|
|
340
362
|
|
|
341
363
|
The committed evidence records an initial `6.04%` difference across six non-zero worst regions and a final `0%` difference with no non-zero worst region. Check mode reproduces the tool path and verifies the committed assets; write mode intentionally refreshes the evidence.
|
|
342
364
|
|
|
343
|
-
## Security and execution model
|
|
344
|
-
|
|
345
|
-
- Inputs resolve against the session workspace and configured `allowedDirs`; realpath containment prevents traversal and symlink escape.
|
|
346
|
-
- Pillow decodes every image before a remote request and verifies bytes, pixels, dimensions, and extension/content agreement. Unsupported or oversized images fail before upload.
|
|
347
|
-
- Outputs use random staging files or directories inside the real managed destination, reject symbolic links, and commit only after format and contract validation.
|
|
348
|
-
- Remote vision prompts explicitly classify text and instructions visible inside images as untrusted content. The native tool descriptions and bundled skill likewise tell the text agent to treat derived descriptions, labels, and OCR as visual evidence rather than executable instructions.
|
|
349
|
-
- All upstream processes use argv vectors through `ctx.subprocess`, inherit caller cancellation, share one hard operation deadline, and terminate with the operation instead of continuing in the background. Plugin disposal aborts active calls before unregistering their tools.
|
|
350
|
-
- One live Session retains only the most recent successful `vision_glance` result. An immediate repeat reuses it only when image content, query/OCR mode, region, endpoint, model, language, and Credential are unchanged; failures and other Sessions never share the entry.
|
|
351
|
-
- Model-visible data is text, numbers, coordinates, structured JSON, and file descriptors. Tool calls/results remain reconstructable from the Session log; browser previews are presentation metadata only.
|
|
352
|
-
- Metrics include tool name, total/upstream duration, bounded image counts/bytes/pixels, cache hits, model, and error category; they exclude base64, authentication headers, secrets, and unbounded upstream output.
|
|
353
|
-
|
|
354
|
-
`vision_html_screenshot` accepts only authorized local `.html` or `.htm` files, disables network access in the pinned adapter, and launches a Chrome-family browser with `--headless=new`, `--use-mock-keychain`, `--incognito`, and a unique `--user-data-dir` under the system temporary directory. The profile is removed after every call, so headless rendering does not touch the user's daily Chrome profile or macOS login keychain.
|
|
355
|
-
|
|
356
365
|
## Troubleshooting
|
|
357
366
|
|
|
358
367
|
| Symptom | Resolution |
|
|
@@ -389,7 +398,7 @@ Update the upstream snapshot only through `pnpm run upstream:sync -- <checkout>`
|
|
|
389
398
|
|
|
390
399
|
## Project status and scope
|
|
391
400
|
|
|
392
|
-
Version `0.1.
|
|
401
|
+
Version `0.1.10` is the current public npm release. P0 and P1 are product commitments in this package. P2 is a design threshold: no stable `ctx.visionToolkit` service, capability-discovery API, or provider ecosystem is published until at least one independent plugin consumes the internal capability shape. Web upload, drag-and-drop, camera/video/audio/document ingestion, interactive box editing, automatic GUI clicking, service clusters, model routing, model voting, and cross-session vision caches remain outside the current product.
|
|
393
402
|
|
|
394
403
|
## Community and About
|
|
395
404
|
|
package/README.zh.md
CHANGED
|
@@ -2,10 +2,11 @@
|
|
|
2
2
|
|
|
3
3
|
# DSH Vision Toolkit
|
|
4
4
|
|
|
5
|
-
[](https://dshfind.com/zh/plugins/Anionex/dsh-vision-toolkit)
|
|
6
|
+
[](https://dshfind.com/zh/plugins/Anionex/dsh-vision-toolkit)
|
|
6
7
|
[](https://x.com/anion_ex)
|
|
7
|
-
[](https://github.com/Anionex/dsh-vision-toolkit/releases/tag/v0.1.10)
|
|
9
|
+
[](tests)
|
|
9
10
|
[](LICENSE)
|
|
10
11
|
[](package.json)
|
|
11
12
|
[](runtime/requirements.lock)
|
|
@@ -100,7 +101,15 @@ dsh --profile headless --dump-config | grep vision-toolkit
|
|
|
100
101
|
|
|
101
102
|
旧 Profile 的 `pnpm-workspace.yaml` 必须使用 `nodeLinker: hoisted` 和 `autoInstallPeers: false`。更新后的 DSH launcher 会在 `dsh plugin` 运行前修复这两个自有设置;使用旧 launcher 时,应在安装前手动设置,避免 pnpm 在 Profile 内组装第二套 Harness 依赖图。
|
|
102
103
|
|
|
103
|
-
安装后重启正在运行的 Web Profile,打开 **设置 →
|
|
104
|
+
安装后重启正在运行的 Web Profile,打开 **设置 → 视觉工具**,先执行**测试 API 连接**,再执行**测试视觉模型**。新安装会自动使用内置免费 Moondream 提供方,不需要 API Key 或 DSH Credential。若要使用其他提供方,请修改端点、模型或协议,并配置对应的 DSH Credential。在会话中把图片放进工作区路径,调用 `/vision-tools`,再让 Agent 使用明确的 `vision_*` 工具。本地裁剪、SVG、像素、颜色、前景和 HTML 操作不需要视觉 API Credential。
|
|
105
|
+
|
|
106
|
+
## 加入交流群
|
|
107
|
+
|
|
108
|
+
欢迎加入 `agent-vision-toolkit` 项目交流群,交流使用经验、反馈问题并提出建议。
|
|
109
|
+
|
|
110
|
+
<p align="center">
|
|
111
|
+
<img src="assets/community-group-qr.png" alt="agent-vision-toolkit 项目交流群二维码" width="260">
|
|
112
|
+
</p>
|
|
104
113
|
|
|
105
114
|
## 工作原理
|
|
106
115
|
|
|
@@ -148,18 +157,18 @@ flowchart LR
|
|
|
148
157
|
|
|
149
158
|
## 纯文本模型的图片输入变体
|
|
150
159
|
|
|
151
|
-
纯文本模型路由会获得同名的兄弟模型条目:`<模型名> (Vision Toolkit)`,挂在对应的提供方分组下。变体声明支持图片输入,因此粘贴的图片走原生附件流程——输入框缩略图、会话持久化图片与历史渲染全部保留——插件只在发往模型的请求链路上把每个图片块改写成 Vision Toolkit
|
|
160
|
+
纯文本模型路由会获得同名的兄弟模型条目:`<模型名> (Vision Toolkit)`,挂在对应的提供方分组下。变体声明支持图片输入,因此粘贴的图片走原生附件流程——输入框缩略图、会话持久化图片与历史渲染全部保留——插件只在发往模型的请求链路上把每个图片块改写成 Vision Toolkit 描述文本,再转交上游路由。视觉提示会携带最新的用户或助手意图,并与 `agent-vision-toolkit` 对齐角色提示、描述要求、图片文字策略以及 `[vision model description]` 等通道标记;模型获得的是与当前任务相关的证据,而不是宽泛的通用描述。会话日志不被改动;回放与 UI 看到的始终是真实图片。
|
|
152
161
|
|
|
153
162
|
插件会自动为宿主明确声明为纯文本的每个模型注册变体(例如 DeepSeek 对话家族)。粘贴处理是全自动的:当当前模型被确认为纯文本、且它的变体已注册时,浏览器端集成会自动把会话切换到变体(会有一条简短提示说明新模型名),随后粘贴走原生流程,无需手动切换模型。宿主依据浏览器从实时模型目录读到的精确模型路由来裁决,模型选择器标签作为兜底;无法确认或支持图片的路由一律保持原生流程,而"纯文本但没有变体"的模型(例如变体被关闭时)继续走"粘贴转路径":图片被复制进会话工作区,输入框里插入的是它的路径文本。
|
|
154
163
|
|
|
155
|
-
描述转换需要已配置的视觉提供方及其 Credential
|
|
164
|
+
描述转换需要已配置的视觉提供方及其 Credential;当运行时未就绪或读取失败时,请求链路上的图片块降级为与上游兼容的 `[vision unavailable: ...]` 提示,而不是让整轮失败。桥接不会把注入的上下文文件当作当前用户意图;如果图片来自工具调用,则使用最新的助手段落作为关注提示。用 `imageInputVariants.enabled: false` 关闭变体,用 `imageInputVariants.providers` 限制被包装的路由,或用 `imageInputVariants.autoSwitch: false` 让纯文本模型继续走"粘贴转路径"。
|
|
156
165
|
|
|
157
166
|
## 运行要求
|
|
158
167
|
|
|
159
168
|
- 启用 Web 或 Headless Profile 的 DeepSeek Harness,并确保 `dsh plugin` 可以使用 `pnpm`。
|
|
160
169
|
- Python 3.11 或更高版本。Managed 模式会创建隔离环境,用户无需手工安装上游 CLI(命令行界面)或 Python 包。
|
|
161
170
|
- 首次启用 managed 运行时需要联网;如果配置的软件包缓存已有 `runtime/requirements.lock` 中的精确版本,则无需联网。
|
|
162
|
-
- `vision_glance`、`vision_ground`、`vision_detect` 和非仅切分长截图 OCR
|
|
171
|
+
- 内置免费 Moondream 提供方可直接用于 `vision_glance`、`vision_ground`、`vision_detect` 和非仅切分长截图 OCR。只有改用自定义 OpenAI 兼容或 Anthropic 端点时才需要 DSH Credential;本地工具不依赖任何远程提供方。
|
|
163
172
|
- 只有 `vision_html_screenshot` 需要 Chrome、Chromium 或 Edge;未安装受支持浏览器时,其他工具保持可用。
|
|
164
173
|
- 输入必须是会话工作区或显式 `allowedDirs` 根目录内的 PNG、JPEG、GIF 或 WebP。
|
|
165
174
|
|
|
@@ -200,7 +209,7 @@ dsh plugin --profile web remove @dsh-external/dsh-vision-toolkit
|
|
|
200
209
|
dsh plugin --profile web add @anionex/dsh-vision-toolkit
|
|
201
210
|
```
|
|
202
211
|
|
|
203
|
-
重启后,Settings → 视觉工具 应显示插件版本 **0.1.
|
|
212
|
+
重启后,Settings → 视觉工具 应显示插件版本 **0.1.10**。内置免费提供方会自动选中;自定义提供方仍使用配置的 DSH Credential。
|
|
204
213
|
|
|
205
214
|
通过注册表安装时,使用 Profile 的包管理命令更新依赖:
|
|
206
215
|
|
|
@@ -228,16 +237,16 @@ Bundle 默认使用 managed 运行时。Profile patch 可以覆盖提供方与
|
|
|
228
237
|
- id: vision-toolkit
|
|
229
238
|
config:
|
|
230
239
|
provider:
|
|
231
|
-
baseUrl: https://
|
|
232
|
-
credential:
|
|
233
|
-
model:
|
|
240
|
+
baseUrl: https://vision.anionex.me/v1
|
|
241
|
+
credential: ANIONEX_FREE_VISION
|
|
242
|
+
model: moondream-3.1
|
|
234
243
|
protocol: openai
|
|
235
244
|
anthropicThinking: omit
|
|
236
245
|
userAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
|
|
237
246
|
language: zh
|
|
238
247
|
timeoutMs: 60000
|
|
239
|
-
maxImageBytes:
|
|
240
|
-
maxImagePixels:
|
|
248
|
+
maxImageBytes: 4194304
|
|
249
|
+
maxImagePixels: 20000000
|
|
241
250
|
concurrency: 4
|
|
242
251
|
runtime:
|
|
243
252
|
mode: managed
|
|
@@ -252,16 +261,16 @@ Bundle 默认使用 managed 运行时。Profile patch 可以覆盖提供方与
|
|
|
252
261
|
|
|
253
262
|
| 字段 | 默认值 | 契约 |
|
|
254
263
|
|---|---|---|
|
|
255
|
-
| `provider.baseUrl` | `https://
|
|
256
|
-
| `provider.credential` | `
|
|
257
|
-
| `provider.model` | `
|
|
264
|
+
| `provider.baseUrl` | `https://vision.anionex.me/v1` | 内置免费 OpenAI 兼容端点;自定义提供方可改用其他基础 URL,使用时会去除结尾斜杠 |
|
|
265
|
+
| `provider.credential` | `ANIONEX_FREE_VISION` | 免费服务的只读内置引用;自定义提供方使用 DSH Credential 引用,而不是密钥值 |
|
|
266
|
+
| `provider.model` | `moondream-3.1` | 远程工具使用的多模态模型名 |
|
|
258
267
|
| `provider.protocol` | `openai` | `openai` 发送 Chat Completions 请求;`anthropic` 发送原生 Messages 请求 |
|
|
259
268
|
| `provider.anthropicThinking` | `omit` | Anthropic thinking 字段。`omit` 不发送 thinking 字段,兼容性最好;仅当所选模型明确支持时使用 `disabled` 或 `adaptive`,提供方返回 HTTP 400 时应先恢复 `omit`。 |
|
|
260
269
|
| `provider.userAgent` | 浏览器兼容默认值 | 视觉请求和显式连接测试发送的 User-Agent;可为提供方或代理兼容性覆盖 |
|
|
261
270
|
| `language` | `zh` | 视觉输出语言:`zh` 或 `en` |
|
|
262
271
|
| `timeoutMs` | `60000` | 完整操作截止时间,1000-600000 毫秒;每个工具可请求更窄的覆盖值 |
|
|
263
|
-
| `maxImageBytes` | `
|
|
264
|
-
| `maxImagePixels` | `
|
|
272
|
+
| `maxImageBytes` | `4194304` | 每张输入图片的编码字节上限;内置免费服务最多接受 4 MiB |
|
|
273
|
+
| `maxImagePixels` | `20000000` | 每张输入图片的解码像素上限;内置免费服务最多接受 20,000,000 像素 |
|
|
265
274
|
| `concurrency` | `4` | 每个会话内的并发操作数,1-16 |
|
|
266
275
|
| `runtime.mode` | `managed` | `managed` 使用打包快照;`external` 只接受精确固定版本 |
|
|
267
276
|
| `runtime.agentVisionToolkitPath` | 未设置 | `external` 模式必填;必须是精确导出快照或固定 commit 的干净 Git checkout |
|
|
@@ -273,10 +282,23 @@ Bundle 默认使用 managed 运行时。Profile patch 可以覆盖提供方与
|
|
|
273
282
|
|
|
274
283
|
### Credential
|
|
275
284
|
|
|
276
|
-
|
|
285
|
+
内置免费提供方使用固定的 `ANIONEX_FREE_VISION` 引用,不接受也不会保存用户 API Key。修改端点、模型或协议切换到自定义提供方后,只写的 **API 密钥** 输入框会自动解锁;填写后保存,会把密钥写入高级设置中的 **凭据名称** 引用。Headless 部署可以在 `$DSH_HOME/.credentials.yaml` 中预置该自定义引用。
|
|
277
286
|
|
|
278
287
|
Settings 只保存引用,不保存值。浏览器不会读取已保存的密钥,保存成功后输入框也会立即清空而不是回显。每次远程操作都会重新解析引用,并只把值注入对应子进程环境。插件排除用户 `.env`、checkout `.env`、`PYTHONPATH`、`PYTHONHOME`、`VIRTUAL_ENV` 和用户 site-packages,避免环境中的 Python 或上游配置覆盖选定的 DSH 提供方。日志、错误、工具结果、产物元数据和 Settings 响应都不包含密钥。
|
|
279
288
|
|
|
289
|
+
### 内置免费服务限制
|
|
290
|
+
|
|
291
|
+
公开服务是共享的零配置默认入口,不是无限量私有端点。限制由代理执行,并以 OpenAI 风格错误返回明确的原因代码和可读提示;限流响应还会携带 `Retry-After` 与请求额度响应头。
|
|
292
|
+
|
|
293
|
+
| 限制 | 当前值 |
|
|
294
|
+
|---|---:|
|
|
295
|
+
| 单客户端 | 每个 UTC 日 30 次 |
|
|
296
|
+
| 全局服务 | 每个 UTC 日 120 次 |
|
|
297
|
+
| 突发 | 60 秒内 6 次 |
|
|
298
|
+
| 图片字节 | 每张最多 4 MiB |
|
|
299
|
+
| 解码像素 | 每张最多 20,000,000 像素 |
|
|
300
|
+
| 输出 | 最多 512 tokens |
|
|
301
|
+
|
|
280
302
|
### Managed 与 external 运行时
|
|
281
303
|
|
|
282
304
|
Managed 模式会验证 `vendor/agent-vision-toolkit/UPSTREAM_MANIFEST.json`,优先使用 `uv`,回退到 `venv` 加 pip,按 `runtime/requirements.lock` 安装精确版本,通过 heartbeat 锁协调并发准备,并只在全部探针通过后发布 staging 环境。
|
|
@@ -300,7 +322,7 @@ Web Profile 会注册 Vision Toolkit Settings 分区,可配置提供方 URL、
|
|
|
300
322
|
|
|
301
323
|
“保存并应用”会验证完整配置,准备候选 Python/上游运行时,提交 Settings revision,最后才原子切换 generation。候选被拒绝时,之前的 generation 继续服务,页面也会把这种状态与运行时确实不可用区分开来。“重新加载”始终恢复后端已保存的权威值,即使 revision 没有变化也会丢弃被拒绝的浏览器草稿。初始启动无法准备运行时时,Settings 路由仍可用于提交有效配置并激活首个 generation。陈旧浏览器 revision 不会覆盖较新的保存结果,而是返回冲突;刷新后再重试。只读 Settings 提供方允许查看和健康检查,但禁用保存。
|
|
302
324
|
|
|
303
|
-
|
|
325
|
+
“运行健康检查”只执行本地检查。“测试 API 连接”是显式操作,会把已配置 Credential 发送到 `GET /models`;OpenAI 使用 Bearer 认证,Anthropic 使用 `x-api-key` 与 `anthropic-version`。这个轻量测试不会上传图片,也不会创建 completion。“测试视觉模型”会另行把插件自带的 `assets/vision-model-test.png` 通过与 `vision_glance` 相同的多模态运行路径发送出去;它会创建一次真实 completion,并用于权威确认所选端点、Credential、模型、协议和上游账户确实能够处理图片。视觉模型检查卡会单独显示“已实测”“未测试”或“测试失败”Tag,避免把 `/models` 返回 HTTP 200 误认为图片调用成功。插件加载和普通 Settings 读取不会发送这两类请求。
|
|
304
326
|
|
|
305
327
|
健康检查、连接测试以及插件/上游版本检查属于 Web Settings 管理能力,而不是模型工具,因此其 schema 永远不会占用 agent 请求上下文。
|
|
306
328
|
|
|
@@ -389,7 +411,7 @@ pnpm pack --dry-run
|
|
|
389
411
|
|
|
390
412
|
## 项目状态与范围
|
|
391
413
|
|
|
392
|
-
版本 `0.1.
|
|
414
|
+
版本 `0.1.10` 是当前公开 npm 发布。P0 和 P1 是本包的产品承诺。P2 是设计门槛:至少一个独立插件消费内部能力形态前,不发布稳定 `ctx.visionToolkit` 服务、能力发现 API 或提供方生态。Web 上传、拖拽、摄像头/视频/音频/文档输入、交互式标注框编辑、GUI 自动点击、远程服务集群、模型路由、模型投票和跨会话视觉缓存不属于当前产品范围。
|
|
393
415
|
|
|
394
416
|
## 社区与关于
|
|
395
417
|
|
|
Binary file
|
|
Binary file
|
|
@@ -2,5 +2,5 @@
|
|
|
2
2
|
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
|
3
3
|
# after editing either side, bring the other along and re-record with:
|
|
4
4
|
# pnpm run verify-translation-pairing --write dsh-vision-toolkit/docs/requirements-traceability/README.md
|
|
5
|
-
README.md:
|
|
6
|
-
README.zh.md:
|
|
5
|
+
README.md: 00e1bd317061adf14f86b3da7c61f0f9d02b96ea
|
|
6
|
+
README.zh.md: 813fbd33c5ca38c37605ba2dedef4b09fae72dc0
|
|
@@ -24,7 +24,7 @@ This reference maps the DSH Vision Toolkit product brief's committed P0/P1 requi
|
|
|
24
24
|
| P1-1 Artifact delivery | **Delivered** | Descriptor creation in [`src/artifacts.ts`](../../src/artifacts.ts), fenced atomic paths in [`src/paths.ts`](../../src/paths.ts), signed capability delivery in [`src/artifact-access.ts`](../../src/artifact-access.ts) | [`tests/artifacts.spec.ts`](../../tests/artifacts.spec.ts), [`tests/artifact-access.spec.ts`](../../tests/artifact-access.spec.ts), Artifact-producing runtime/profile tests |
|
|
25
25
|
| P1-2 Extended tools | **Delivered** | `vision_pixel_diff`, `vision_long_screenshot_ocr`, `vision_extract_foreground`, `vision_dominant_colors`, and `vision_html_screenshot` in [`src/tools.ts`](../../src/tools.ts) and [`src/runtime.ts`](../../src/runtime.ts) | P1 parser and runtime cases in [`tests/upstream.spec.ts`](../../tests/upstream.spec.ts) and [`tests/runtime.spec.ts`](../../tests/runtime.spec.ts); pixel-diff and long-OCR real-profile calls |
|
|
26
26
|
| P1-3 Dedicated Web presentation | **Delivered** | Browser plugin and dedicated cards in [`src/client/index.tsx`](../../src/client/index.tsx); presentation-only capability metadata in [`src/artifact-access.ts`](../../src/artifact-access.ts) | [`tests/client.spec.ts`](../../tests/client.spec.ts), safe preview tests in [`tests/artifact-access.spec.ts`](../../tests/artifact-access.spec.ts), Web visual/console QA |
|
|
27
|
-
| P1-4 Health checks | **Delivered** | Health/version runtime contracts in [`src/runtime.ts`](../../src/runtime.ts), same-origin Web actions in [`src/web.ts`](../../src/web.ts), and the Settings surface in [`src/client/index.tsx`](../../src/client/index.tsx); these administrative diagnostics are deliberately absent from the model tool registry | Health cases in [`tests/runtime.spec.ts`](../../tests/runtime.spec.ts), explicit-
|
|
27
|
+
| P1-4 Health checks | **Delivered** | Health/version runtime contracts in [`src/runtime.ts`](../../src/runtime.ts), same-origin Web actions in [`src/web.ts`](../../src/web.ts), and the Settings surface in [`src/client/index.tsx`](../../src/client/index.tsx); Settings distinguishes the lightweight authenticated `GET /models` probe from an explicit real multimodal request using the bundled diagnostic image, and these administrative diagnostics are deliberately absent from the model tool registry | Health and real-model success/failure cases in [`tests/runtime.spec.ts`](../../tests/runtime.spec.ts), explicit API/model-test behavior in [`tests/web.spec.ts`](../../tests/web.spec.ts) and [`tests/client.spec.ts`](../../tests/client.spec.ts), plus model-tool absence in [`tests/tools.spec.ts`](../../tests/tools.spec.ts) |
|
|
28
28
|
| P1-5 Settings | **Delivered** | Namespace/config in [`src/config.ts`](../../src/config.ts), prepare-before-swap manager in [`src/runtime-manager.ts`](../../src/runtime-manager.ts), private same-origin route in [`src/web.ts`](../../src/web.ts), section in [`src/client/index.tsx`](../../src/client/index.tsx) | [`tests/runtime-manager.spec.ts`](../../tests/runtime-manager.spec.ts), [`tests/web.spec.ts`](../../tests/web.spec.ts), [`tests/client.spec.ts`](../../tests/client.spec.ts), clean Web-profile save/restart QA |
|
|
29
29
|
| P1-6 Install and upgrade experience | **Delivered** | Bundle lifecycle in [`src/index.ts`](../../src/index.ts), content-addressed managed runtime in [`src/runtime-install.ts`](../../src/runtime-install.ts), generation manager in [`src/runtime-manager.ts`](../../src/runtime-manager.ts) | Runtime interruption/concurrency tests, package-layout test, clean-profile lifecycle, persisted Settings and failed-candidate retention checks |
|
|
30
30
|
|
|
@@ -46,7 +46,7 @@ This reference maps the DSH Vision Toolkit product brief's committed P0/P1 requi
|
|
|
46
46
|
| Scenario | Expected behavior | Evidence |
|
|
47
47
|
|---|---|---|
|
|
48
48
|
| Missing/invalid image, format, region, or path | Reject before upstream execution with an input, capacity, or path-safe error | [`tests/paths.spec.ts`](../../tests/paths.spec.ts), [`tests/runtime.spec.ts`](../../tests/runtime.spec.ts) |
|
|
49
|
-
| Missing Credential | Local tools remain usable; remote tools
|
|
49
|
+
| Missing Credential | Local tools remain usable; remote tools plus explicit API-connection and real-model tests report a redacted configuration/service action | [`tests/runtime.spec.ts`](../../tests/runtime.spec.ts), [`tests/web.spec.ts`](../../tests/web.spec.ts) |
|
|
50
50
|
| 401/403, 429, timeout, malformed output, or cancellation | Return a stable actionable category, preserve bounded diagnostics, and stop the request/subprocess | [`tests/errors.spec.ts`](../../tests/errors.spec.ts), [`tests/runtime.spec.ts`](../../tests/runtime.spec.ts), [`tests/upstream.spec.ts`](../../tests/upstream.spec.ts) |
|
|
51
51
|
| Runtime unavailable during initial load | Register no Skill, activation bootstrap, or Agent-scoped tools; keep Web Settings available for repair | [`src/index.ts`](../../src/index.ts), lifecycle tests in [`tests/tools.spec.ts`](../../tests/tools.spec.ts) and [`tests/web.spec.ts`](../../tests/web.spec.ts) |
|
|
52
52
|
| Runtime candidate fails during live update | Preserve the current serving generation and stored usable configuration | [`tests/runtime-manager.spec.ts`](../../tests/runtime-manager.spec.ts), [`tests/web.spec.ts`](../../tests/web.spec.ts) |
|
|
@@ -24,7 +24,7 @@
|
|
|
24
24
|
| P1-1 产物交付 | **已交付** | [`src/artifacts.ts`](../../src/artifacts.ts) 中的产物描述创建、[`src/paths.ts`](../../src/paths.ts) 中受围栏保护的原子路径、[`src/artifact-access.ts`](../../src/artifact-access.ts) 中的签名能力交付 | [`tests/artifacts.spec.ts`](../../tests/artifacts.spec.ts)、[`tests/artifact-access.spec.ts`](../../tests/artifact-access.spec.ts)、会生成产物的运行时/Profile 测试 |
|
|
25
25
|
| P1-2 扩展工具 | **已交付** | [`src/tools.ts`](../../src/tools.ts) 和 [`src/runtime.ts`](../../src/runtime.ts) 中的 `vision_pixel_diff`、`vision_long_screenshot_ocr`、`vision_extract_foreground`、`vision_dominant_colors`、`vision_html_screenshot` | [`tests/upstream.spec.ts`](../../tests/upstream.spec.ts) 和 [`tests/runtime.spec.ts`](../../tests/runtime.spec.ts) 中的 P1 解析器/运行时用例;真实 Profile pixel-diff 和长截图 OCR 调用 |
|
|
26
26
|
| P1-3 专用 Web 展示 | **已交付** | [`src/client/index.tsx`](../../src/client/index.tsx) 中的浏览器插件与专用卡片;[`src/artifact-access.ts`](../../src/artifact-access.ts) 中仅供展示的能力元数据 | [`tests/client.spec.ts`](../../tests/client.spec.ts)、[`tests/artifact-access.spec.ts`](../../tests/artifact-access.spec.ts) 中的安全预览测试、Web 视觉/Console QA |
|
|
27
|
-
| P1-4 健康检查 | **已交付** | [`src/runtime.ts`](../../src/runtime.ts) 中的健康检查/版本运行时契约、[`src/web.ts`](../../src/web.ts) 中的同源 Web 操作,以及 [`src/client/index.tsx`](../../src/client/index.tsx) 中的 Settings
|
|
27
|
+
| P1-4 健康检查 | **已交付** | [`src/runtime.ts`](../../src/runtime.ts) 中的健康检查/版本运行时契约、[`src/web.ts`](../../src/web.ts) 中的同源 Web 操作,以及 [`src/client/index.tsx`](../../src/client/index.tsx) 中的 Settings 界面;Settings 明确区分携带凭据的轻量 `GET /models` 探测与使用自带诊断图片的显式真实多模态请求,这些管理诊断能力有意不进入模型工具注册表 | [`tests/runtime.spec.ts`](../../tests/runtime.spec.ts) 中的健康检查及真实模型成功/失败用例、[`tests/web.spec.ts`](../../tests/web.spec.ts) 与 [`tests/client.spec.ts`](../../tests/client.spec.ts) 中的显式 API/模型测试行为,以及 [`tests/tools.spec.ts`](../../tests/tools.spec.ts) 中的模型工具缺席断言 |
|
|
28
28
|
| P1-5 Settings | **已交付** | [`src/config.ts`](../../src/config.ts) 中的 namespace/配置、[`src/runtime-manager.ts`](../../src/runtime-manager.ts) 中的 prepare-before-swap manager、[`src/web.ts`](../../src/web.ts) 中的同源私有路由、[`src/client/index.tsx`](../../src/client/index.tsx) 中的 Settings 分区 | [`tests/runtime-manager.spec.ts`](../../tests/runtime-manager.spec.ts)、[`tests/web.spec.ts`](../../tests/web.spec.ts)、[`tests/client.spec.ts`](../../tests/client.spec.ts)、干净 Web Profile 保存/重启 QA |
|
|
29
29
|
| P1-6 安装与升级体验 | **已交付** | [`src/index.ts`](../../src/index.ts) 中的 Bundle 生命周期、[`src/runtime-install.ts`](../../src/runtime-install.ts) 中的内容寻址 managed 运行时、[`src/runtime-manager.ts`](../../src/runtime-manager.ts) 中的 generation manager | 运行时中断/并发测试、软件包布局测试、干净 Profile 生命周期、Settings 持久化和失败候选保留检查 |
|
|
30
30
|
|
|
@@ -46,7 +46,7 @@
|
|
|
46
46
|
| 场景 | 预期行为 | 证据 |
|
|
47
47
|
|---|---|---|
|
|
48
48
|
| 图片缺失/无效、格式、区域或路径错误 | 在执行上游前以输入、容量或路径安全错误拒绝 | [`tests/paths.spec.ts`](../../tests/paths.spec.ts)、[`tests/runtime.spec.ts`](../../tests/runtime.spec.ts) |
|
|
49
|
-
| Credential 缺失 |
|
|
49
|
+
| Credential 缺失 | 本地工具保持可用;远程工具以及显式 API 连接/真实模型测试返回脱敏且可执行下一步的配置/服务结果 | [`tests/runtime.spec.ts`](../../tests/runtime.spec.ts)、[`tests/web.spec.ts`](../../tests/web.spec.ts) |
|
|
50
50
|
| 401/403、429、超时、畸形输出或取消 | 返回稳定且可执行下一步的类别,保留有界诊断信息,并停止请求/子进程 | [`tests/errors.spec.ts`](../../tests/errors.spec.ts)、[`tests/runtime.spec.ts`](../../tests/runtime.spec.ts)、[`tests/upstream.spec.ts`](../../tests/upstream.spec.ts) |
|
|
51
51
|
| 初始加载时运行时不可用 | 不注册 skill、激活引导工具或 Agent 级工具;保留 Web Settings 以供修复 | [`src/index.ts`](../../src/index.ts)、[`tests/tools.spec.ts`](../../tests/tools.spec.ts) 与 [`tests/web.spec.ts`](../../tests/web.spec.ts) 中的生命周期测试 |
|
|
52
52
|
| 实时更新时运行时候选失败 | 保留当前服务 generation 和已存储的可用配置 | [`tests/runtime-manager.spec.ts`](../../tests/runtime-manager.spec.ts)、[`tests/web.spec.ts`](../../tests/web.spec.ts) |
|