dsh-tool-vision 0.5.0 → 0.6.0

package/README.md

@@ -171,6 +171,51 @@ Example endpoints (`baseURL`):
 > - Settings changes hot-apply (no restart needed).
 > - Tested against DSH `0.1.0-rc.6`.
 
+## Pixel-level vision tools (v0.6.0, ported from dsh-vision-router)
+
+14 `vision_*` tools driven by the **same** configured endpoint as
+`inspect_image` (baseURL/apiKey/model) — no provider chain, no local models,
+no extra settings:
+
+| Tool | Purpose |
+|---|---|
+| `vision_describe` | Image Q&A / multi-image comparison (optional structured JSON) |
+| `vision_ground` | Locate a target and return its ORIGINAL-pixel bounding box |
+| `vision_detect` | Enumerate elements (buttons, inputs, icons…) with numbered boxes |
+| `vision_crop` | Crop a pixel region to a PNG artifact |
+| `vision_pixel_diff` | Per-pixel comparison: ratio, worst regions, heatmap, report |
+| `vision_colors` | Dominant-color quantization for palette matching |
+| `vision_ocr` | Verbatim text transcription (letters only — not scene analysis) |
+| `vision_long_screenshot_ocr` | Chunked long-screenshot transcription into Markdown |
+| `vision_trace` | Potrace vectorization into colored SVG (worker-thread, safe) |
+| `vision_extract_foreground` | Solid-background removal → transparent PNG |
+| `vision_html_screenshot` | Headless render of a local .html (network blocked) |
+| `vision_screenshot` | Desktop capture (privacy-gated: enable `desktopScreenshot` in settings; Win: PowerShell / macOS: screencapture / Linux: import/scrot) |
+| `vision_present` | Publish a generated image to the user via the host attachment store |
+| `vision_materialize` | Copy an attachment/local image into the workspace as a real path |
+
+Quality & safety details:
+
+- **Content-hash cache** keyed by endpoint+model+image+question (no stale
+  answers across model switches, failures are never cached).
+- **Uniform 4MP downscale** before every model call; oversized inputs are
+  rejected with a clear error (stat pre-check, 20MB cap on both file and
+  attachment paths).
+- **Rate-limit / 5xx auto-retry** with Retry-After-aware backoff; endpoint
+  **content-safety rejections** are surfaced as `VISION_CONTENT_FILTERED`
+  instead of a generic backend error.
+- **Long-OCR bounds**: 120s total budget, 40-chunk cap, cancellation checks,
+  stop-on-first-backend-failure.
+- **Path containment** for relative inputs; artifacts land in
+  `<workspace>/.dsh-tool-vision/`.
+
+Requires `sharp` / `potrace` / `puppeteer-core` (declared as optional
+dependencies: a failed platform install never blocks the plugin; missing ones
+degrade lazily with an install hint and never break other tools).
+
+`vision_screenshot` is privacy-sensitive and therefore **not registered by
+default** — set `desktopScreenshot: true` in the tool-vision settings to
+enable desktop capture.
 
 ## Limitations
 
@@ -183,4 +228,6 @@ Example endpoints (`baseURL`):
 
 ## License
 
-MIT
+MIT — bridge preview & integration: xing666173. Pixel vision tools ported
+from [dsh-vision-router](https://github.com/ysr666/dsh-vision-router)
+(© ysr666, MIT) with gratitude.

package/README.zh.md

@@ -113,6 +113,46 @@ DeepSeek 自家模型是纯文本的，而且 harness 的每次模型请求都**
 - **Moonshot（Kimi）**：`https://api.moonshot.cn/v1` —— `moonshot-v1-8k-vision-preview`
 - **Ollama 本地**：`http://localhost:11434/v1` —— `llama3.2-vision`（无需密钥）
 
+## 像素级视觉工具(v0.6.0,移植自 dsh-vision-router)
+
+14 个 `vision_*` 工具由**同一个** `inspect_image` 配置的端点驱动
+(baseURL/apiKey/model)——无 provider 链、无本地模型、零新增配置:
+
+| 工具 | 用途 |
+|---|---|
+| `vision_describe` | 看图问答 / 多图对比(可选结构化 JSON) |
+| `vision_ground` | 定位目标,返回原图像素坐标框 |
+| `vision_detect` | 枚举元素(按钮/输入框/图标…),带编号框 |
+| `vision_crop` | 按像素区域裁剪出 PNG 产物 |
+| `vision_pixel_diff` | 逐像素对比:差异比例、最差区域、热图、报告 |
+| `vision_colors` | 主色量化,还原 UI 调色板 |
+| `vision_ocr` | 逐字转写文字(只读字,不做场景识别) |
+| `vision_long_screenshot_ocr` | 长截图分块转写为 Markdown |
+| `vision_trace` | potrace 矢量化输出彩色 SVG(worker 线程,安全) |
+| `vision_extract_foreground` | 纯色背景抠图 → 透明 PNG |
+| `vision_html_screenshot` | 本地 HTML 无头渲染截图(禁网) |
+| `vision_screenshot` | 桌面截屏(隐私门控:需在设置中开启 `desktopScreenshot`;Win: PowerShell / macOS: screencapture / Linux: import/scrot) |
+| `vision_present` | 通过宿主附件库把生成的图片正式展示给用户 |
+| `vision_materialize` | 把附件/本地图片落盘为工作区真实路径 |
+
+质量与安全细节:
+
+- **内容哈希缓存**按 端点+模型+图片+问题 取键(切模型不吃旧答案,失败结果
+  不入缓存);
+- **统一 4MP 降采样**后再调用模型;超大输入 stat 预检直接拒绝(文件与
+  附件路径统一 20MB 上限);
+- **限流/5xx 自动重试**(感知 Retry-After 退避);端点**内容安全拒绝**明确
+  返回 `VISION_CONTENT_FILTERED`,不再误报后端不可用;
+- **长截图 OCR 边界**:120s 总预算、40 块上限、取消检查、首块失败即停;
+- **路径 containment**(相对输入禁止逃逸工作区);产物写入
+  `<工作区>/.dsh-tool-vision/`。
+
+依赖 `sharp` / `potrace` / `puppeteer-core`(声明为**可选依赖**:平台安装失败
+不会阻断插件安装;缺失时懒加载降级并给出安装提示,不影响其他工具)。
+
+`vision_screenshot` 属于隐私敏感能力,**默认不注册**——在 tool-vision 设置中
+开启 `desktopScreenshot: true` 后才会注册桌面截屏工具。
+
 ## 限制
 
 - 被桥接的图片以文本指引进入对话（转录而非像素）——文本模型无法做像素级上下文推理；视觉模型的描述通过 `inspect_image` 回传。
@@ -121,4 +161,6 @@ DeepSeek 自家模型是纯文本的，而且 harness 的每次模型请求都**
 
 ## License
 
-MIT
+MIT —— 桥接预览与整合:xing666173。像素级视觉工具移植自
+[dsh-vision-router](https://github.com/ysr666/dsh-vision-router)(© ysr666,MIT),
+在此致谢。

package/client.js

@@ -79,10 +79,12 @@ window.__ModuleLoader__.load({
       fieldBridgePreviewScanIntervalMs: "预览兜底扫描间隔（毫秒，0 = 关闭兜底）",
       fieldBridgePreviewHideHint: "图片加载成功后隐藏桥接提示文本",
       fieldBridgeAutoImage: "自动声明图片能力（纯文本模型也能粘贴图片）",
+      fieldDesktopScreenshot: "桌面截屏工具（vision_screenshot，默认关闭）",
       hintBridgePreview: "纯展示层：不影响模型侧文本与 inspect_image 调用。",
       hintBridgePreviewScanIntervalMs: "默认 2000ms；越小响应越快，越大越省资源。",
       hintBridgePreviewHideHint: "加载失败时保留文本（安全降级，绝不出现既无图又无字）。",
       hintBridgeAutoImage: "桥接开启时向宿主报告所有模型支持图片，绕过“模型不支持图片”的发送拦截；实际仍走桥接转文本。",
+      hintDesktopScreenshot: "隐私敏感能力：仅在开启后注册 vision_screenshot 桌面截屏工具，模型才可截取用户屏幕。",
       save: "保存",
       reset: "恢复默认",
       saved: "已保存",
@@ -116,10 +118,12 @@ window.__ModuleLoader__.load({
       fieldBridgePreviewScanIntervalMs: "Preview fallback scan interval (ms, 0 = disable)",
       fieldBridgePreviewHideHint: "Hide the bridged hint text once the image has loaded",
       fieldBridgeAutoImage: "Auto-declare image capability (paste images on text-only models)",
+      fieldDesktopScreenshot: "Desktop screenshot tool (vision_screenshot, off by default)",
       hintBridgePreview: "Pure display layer: the model-facing text and the inspect_image chain are untouched.",
       hintBridgePreviewScanIntervalMs: "Default 2000ms; lower is snappier, higher is cheaper.",
       hintBridgePreviewHideHint: "Text is kept on load failure (safe degradation, never no image AND no text).",
       hintBridgeAutoImage: "While the bridge is on, report image support for every model to bypass the host image-send gate; images still travel as bridged text hints.",
+      hintDesktopScreenshot: "Privacy-sensitive: the vision_screenshot desktop-capture tool is only registered (and visible to the model) when enabled.",
       save: "Save",
       reset: "Reset",
       saved: "Saved",
@@ -145,7 +149,8 @@ window.__ModuleLoader__.load({
       { key: "bridgePreview", label: "fieldBridgePreview", type: "checkbox" },
       { key: "bridgePreviewScanIntervalMs", label: "fieldBridgePreviewScanIntervalMs", type: "number" },
       { key: "bridgePreviewHideHint", label: "fieldBridgePreviewHideHint", type: "checkbox" },
-      { key: "bridgeAutoImage", label: "fieldBridgeAutoImage", type: "checkbox" }
+      { key: "bridgeAutoImage", label: "fieldBridgeAutoImage", type: "checkbox" },
+      { key: "desktopScreenshot", label: "fieldDesktopScreenshot", type: "checkbox" }
     ];
     var ZH_HINTS = {
       apiKey: "apiKeyHint",
@@ -158,7 +163,8 @@ window.__ModuleLoader__.load({
       bridgePreview: "hintBridgePreview",
       bridgePreviewScanIntervalMs: "hintBridgePreviewScanIntervalMs",
       bridgePreviewHideHint: "hintBridgePreviewHideHint",
-      bridgeAutoImage: "hintBridgeAutoImage"
+      bridgeAutoImage: "hintBridgeAutoImage",
+      desktopScreenshot: "hintDesktopScreenshot"
     };
 
     function labelOf(f, t) {

package/index.js

@@ -43,6 +43,7 @@ import z from "@deepseek-ai/schemastery";
 import { defineTool } from "@deepseek-ai/dsh-tools";
 import { installSettingsSection, settingsNamespace } from "@deepseek-ai/dsh-settings";
 import { ensureSettingsNamespaceExposed } from "./vendor/dsh-settings-expose.js";
+import { registerVisionTools, CONTENT_FILTER_RE } from "./lib/vision-tools.js";
 
 /** Cordis plugin name. */
 const name = "tool-vision";
@@ -110,6 +111,8 @@ const Config = z.object({
   bridgePreviewScanIntervalMs: z.number().default(2000),
   /** Hide the bridged hint text once the preview image has loaded (kept on failure — never "no image AND no text"). */
   bridgePreviewHideHint: z.boolean().default(true),
+  /** Privacy gate for vision_screenshot: desktop capture is only registered when explicitly enabled. */
+  desktopScreenshot: z.boolean().default(false),
   /**
    * Advertise image input capability for every model while the bridge is on.
    * The host admission gate (host-apiproxy `prompt`/`selectModel`) refuses
@@ -611,10 +614,27 @@ function apply(ctx, config) {
       const cfg = getConfig();
       const cwd = exec.agent?.session?.header?.cwd ?? process.cwd();
       const { url, note } = await toImageUrl(args.path, cwd, cfg);
-      const answer = await callVision(cfg, url, args.question, args.detail, exec.signal);
-      return note === url ? answer : `${answer}\n\n(image: ${note})`;
+      try {
+        const answer = await callVision(cfg, url, args.question, args.detail, exec.signal);
+        return note === url ? answer : `${answer}\n\n(image: ${note})`;
+      } catch (error) {
+        const raw = error && error.message ? String(error.message) : String(error);
+        if (CONTENT_FILTER_RE.test(raw)) {
+          throw new Error(
+            "inspect_image: 图片被视觉端点的内容安全策略拒绝(检测到敏感或不安全内容)。" +
+              "这不是网络或配置问题,请换一张图片或调整图片内容后再试。",
+          );
+        }
+        throw error;
+      }
     },
   }));
+
+  // ── pixel-level vision tools (ported from dsh-vision-router) ─────────────
+  // 14 vision_* tools driven by the SAME configured endpoint as inspect_image
+  // (baseURL/apiKey/model). No provider chain, no local models, no extra
+  // settings: everything comes from the existing tool-vision configuration.
+  registerVisionTools(ctx, getConfig);
 }
 
 export {