dsh-tool-vision 0.1.0 → 0.1.1

package/README.md

@@ -3,10 +3,17 @@
 External vision model for [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness).
 
 DeepSeek's own models are text-only, and dsh-llm has no multimodal content
-block yet. This plugin bridges the gap with a model-facing tool that sends an
-image (local file, or http(s) URL) to **any OpenAI-compatible**
-`/chat/completions` endpoint that supports `image_url` content parts, and
-returns the vision model's textual answer into the agent loop.
+block yet. This plugin bridges the gap in two ways:
+
+1. **`inspect_image` tool** — sends an image (local file, or http(s) URL) to
+   **any OpenAI-compatible** `/chat/completions` endpoint that supports
+   `image_url` content parts, and returns the vision model's textual answer
+   into the agent loop.
+2. **Image bridge (v0.1.1)** — when the active model is text-only, images
+   pasted into the conversation are intercepted on the `llm/stream`
+   waterfall, exported to a local file, and replaced with a text hint, so
+   the agent picks them up with `inspect_image`. Models listed in
+   `multimodalModels` receive image blocks directly instead.
 
 - Zero dependencies beyond the dsh SDK — works with any compatible endpoint:
   OpenAI GPT-4o, Qwen-VL (DashScope), GLM-4V (Zhipu), Moonshot, Gemini
@@ -47,6 +54,34 @@ Or load it from a local path without npm:
 | `timeoutMs` | `60000` | Per-request timeout. |
 | `maxImageBytes` | `10MB` | Largest accepted local image. |
 | `description` | default | Tool description shown to the model. |
+| `bridgeTextOnly` | `true` | Bridge pasted images to text hints for text-only models. |
+| `bridgeExportDir` | temp | Export dir for bridged images (`os.tmpdir()/dsh-vision-bridge`). |
+| `multimodalModels` | `[]` | Model ids that receive image blocks directly (e.g. `mimo-v2.5`). |
+
+## Image bridge setup
+
+1. In your model settings, declare image input on the text models you use
+   (pi-ai style), so the harness lets image messages through:
+   ```yaml
+   llm-pi-ai:
+     providers:
+       your-provider:
+         models:
+           - id: deepseek-v4-flash
+             input: [text, image]
+   ```
+2. List genuinely multimodal models in the plugin config so they receive
+   image blocks untouched:
+   ```yaml
+   - id: tool-vision
+     name: 'dsh-tool-vision'
+     config:
+       multimodalModels: ['mimo-v2.5', 'grok-4.5']
+   ```
+
+Then pasting an image while on a text-only model produces a hint like
+`[User sent an image, exported to: <path>. Inspect it with the inspect_image tool...]`
+and the agent inspects it through the configured vision endpoint.
 
 Key resolution order: `config.apiKey` → `process.env[apiKeyEnv]` →
 `process.env.OPENAI_API_KEY`.

package/README.zh.md

@@ -1,60 +1,115 @@
 # dsh-tool-vision
 
-给 DeepSeek Harness 外接**视觉模型**的工具插件。DeepSeek 自家模型是纯文本的，而 dsh-llm 目前还没有多模态内容块类型——本插件用一条 OpenAI 兼容接口补上这个缺口：
+External vision model for [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness).
 
-Agent 调用工具 `inspect_image`，把本地图片（或 http(s) 图片 URL）发给任意兼容 `image_url` 内容块的 `/chat/completions` 端点，把视觉模型的文字回答带回对话。
+DeepSeek's own models are text-only, and dsh-llm has no multimodal content
+block yet. This plugin bridges the gap in two ways:
 
-- 注册在**全局工具层**：进程内所有 Agent 都能调用 `inspect_image`
-- 支持任意 OpenAI 兼容视觉端点：OpenAI GPT-4o / Qwen-VL (DashScope) / GLM-4V (智谱) / Moonshot / Gemini 兼容端点 / Ollama 本地等
-- 本地图片自动转 base64 data URL，检查格式与大小上限
+1. **`inspect_image` tool** — sends an image (local file, or http(s) URL) to
+   **any OpenAI-compatible** `/chat/completions` endpoint that supports
+   `image_url` content parts, and returns the vision model's textual answer
+   into the agent loop.
+2. **Image bridge (v0.1.1)** — when the active model is text-only, images
+   pasted into the conversation are intercepted on the `llm/stream`
+   waterfall, exported to a local file, and replaced with a text hint, so
+   the agent picks them up with `inspect_image`. Models listed in
+   `multimodalModels` receive image blocks directly instead.
 
-## 配置
+- Zero dependencies beyond the dsh SDK — works with any compatible endpoint:
+  OpenAI GPT-4o, Qwen-VL (DashScope), GLM-4V (Zhipu), Moonshot, Gemini
+  compatible endpoints, local Ollama, etc.
+- Registered on the **global tools layer**: every agent in the process can
+  call `inspect_image`.
 
-| 字段 | 默认值 | 含义 |
-|---|---|---|
-| `baseURL` | `https://api.openai.com/v1` | OpenAI 兼容 API 基地址 |
-| `apiKey` | `''` | API 密钥（优先于环境变量） |
-| `apiKeyEnv` | `VISION_API_KEY` | 存放密钥的环境变量名 |
-| `model` | `gpt-4o-mini` | 视觉模型 id |
-| `maxTokens` | `1024` | 视觉调用最大输出 token |
-| `timeoutMs` | `60000` | 单次请求超时 |
-| `maxImageBytes` | `10MB` | 本地图片大小上限 |
-| `description` | 默认描述 | 工具描述（模型可见） |
+## Install
 
-密钥解析顺序：`config.apiKey` → `process.env[apiKeyEnv]` → `process.env.OPENAI_API_KEY`。
+Mount in a profile patch (`$DSH_HOME/profiles/<name>/cordis.patch.yml`):
 
-## 挂载
+```yaml
+- insert:
+    - id: tool-vision
+      name: 'dsh-tool-vision'     # after: pnpm add dsh-tool-vision in the profile
+      config:
+        baseURL: 'https://api.openai.com/v1'
+        apiKeyEnv: 'VISION_API_KEY'
+        model: 'gpt-4o-mini'
+```
 
-在 profile 的 `cordis.patch.yml` 里 insert：
+Or load it from a local path without npm:
 
 ```yaml
-- insert:
     - id: tool-vision
       name: './plugins/dsh-tool-vision/index.js'
-      config:
-        baseURL: 'https://dashscope.aliyuncs.com/compatible-mode/v1'
-        apiKeyEnv: 'DASHSCOPE_API_KEY'
-        model: 'qwen-vl-plus'
 ```
 
-## 工具：`inspect_image`
+## Config
 
-| 参数 | 必填 | 说明 |
+| Field | Default | Meaning |
 |---|---|---|
-| `path` | ✅ | 图片路径（绝对路径，或相对当前 workspace）或 http(s) URL |
-| `question` | – | 针对图片的具体问题；缺省为"详细描述" |
-| `detail` | – | `auto` / `low` / `high`，图片分辨率提示 |
+| `baseURL` | `https://api.openai.com/v1` | OpenAI-compatible API base URL. |
+| `apiKey` | `''` | API key (takes precedence over env). |
+| `apiKeyEnv` | `VISION_API_KEY` | Env var holding the key. |
+| `model` | `gpt-4o-mini` | Vision model id. |
+| `maxTokens` | `1024` | Max output tokens. |
+| `timeoutMs` | `60000` | Per-request timeout. |
+| `maxImageBytes` | `10MB` | Largest accepted local image. |
+| `description` | default | Tool description shown to the model. |
+| `bridgeTextOnly` | `true` | Bridge pasted images to text hints for text-only models. |
+| `bridgeExportDir` | temp | Export dir for bridged images (`os.tmpdir()/dsh-vision-bridge`). |
+| `multimodalModels` | `[]` | Model ids that receive image blocks directly (e.g. `mimo-v2.5`). |
+
+## Image bridge setup
+
+1. In your model settings, declare image input on the text models you use
+   (pi-ai style), so the harness lets image messages through:
+   ```yaml
+   llm-pi-ai:
+     providers:
+       your-provider:
+         models:
+           - id: deepseek-v4-flash
+             input: [text, image]
+   ```
+2. List genuinely multimodal models in the plugin config so they receive
+   image blocks untouched:
+   ```yaml
+   - id: tool-vision
+     name: 'dsh-tool-vision'
+     config:
+       multimodalModels: ['mimo-v2.5', 'grok-4.5']
+   ```
+
+Then pasting an image while on a text-only model produces a hint like
+`[User sent an image, exported to: <path>. Inspect it with the inspect_image tool...]`
+and the agent inspects it through the configured vision endpoint.
+
+Key resolution order: `config.apiKey` → `process.env[apiKeyEnv]` →
+`process.env.OPENAI_API_KEY`.
+
+## Tool: `inspect_image`
+
+| Arg | Required | Meaning |
+|---|---|---|
+| `path` | ✅ | Image path (absolute, or relative to the current workspace) or http(s) URL. |
+| `question` | – | Optional specific question about the image. |
+| `detail` | – | `auto` / `low` / `high` resolution hint. |
+
+Example endpoints (`baseURL`):
+
+- **OpenAI**: `https://api.openai.com/v1` — `gpt-4o`, `gpt-4o-mini`
+- **Alibaba DashScope (Qwen-VL)**: `https://dashscope.aliyuncs.com/compatible-mode/v1` — `qwen-vl-plus`, `qwen-vl-max`
+- **Zhipu (GLM-4V)**: `https://open.bigmodel.cn/api/paas/v4` — `glm-4v-flash` (free tier), `glm-4v-plus`
+- **Moonshot (Kimi)**: `https://api.moonshot.cn/v1` — `moonshot-v1-8k-vision-preview`
+- **Ollama local**: `http://localhost:11434/v1` — `llama3.2-vision` (no key)
 
-示例端点（`baseURL` 写法）：
+## Limitations
 
-- **OpenAI**：`https://api.openai.com/v1`，模型 `gpt-4o` / `gpt-4o-mini`
-- **阿里云百炼 (Qwen-VL)**：`https://dashscope.aliyuncs.com/compatible-mode/v1`，模型 `qwen-vl-plus` / `qwen-vl-max`
-- **智谱 (GLM-4V)**：`https://open.bigmodel.cn/api/paas/v4`，模型 `glm-4v-flash`（免费档）/ `glm-4v-plus`
-- **Moonshot (Kimi)**：`https://api.moonshot.cn/v1`，模型 `moonshot-v1-8k-vision-preview`
-- **Ollama 本地**：`http://localhost:11434/v1`，模型 `llama3.2-vision`（无需 key）
+- The image enters the conversation as a text description (a transcript, not
+  pixels) — pixel-precise tasks may be inaccurate.
+- Images are base64-transferred; mind privacy and size limits.
+- Independent of the dsh-llm routing/retry system; failures return clear
+  errors to the agent.
 
-## 局限
+## License
 
-- 图片内容以文字描述进入对话，Agent 得到的是"转述"而非像素——精确到像素的任务（如测量）可能不够准
-- 图片会在请求中 base64 传输，注意隐私与大小
-- 不依赖 dsh-llm 服务，因此不走模型路由/重试体系；失败会以明确的错误信息返回给 Agent
+MIT

package/index.js

@@ -1,24 +1,29 @@
 /**
  * dsh-tool-vision — external vision model for DeepSeek Harness.
  *
- * DeepSeek's own models are text-only, and dsh-llm has no multimodal content
- * block type yet. This plugin bridges the gap with a model-facing tool that
- * sends an image (local file, or http(s) URL) to any OpenAI-compatible
- * chat/completions endpoint that supports `image_url` content parts, and
- * returns the vision model's textual answer into the agent loop.
+ * Two capabilities:
  *
- * Registered on the GLOBAL tools layer, so every agent in the process can
- * call `inspect_image`.
+ * 1. `inspect_image` tool — sends an image (local file, or http(s) URL) to
+ *    any OpenAI-compatible chat/completions endpoint that supports
+ *    `image_url` content parts, and returns the vision model's text answer.
+ *
+ * 2. Image bridge — when the current model is text-only (e.g. DeepSeek),
+ *    pasted images in user messages are intercepted on the `llm/stream`
+ *    waterfall, exported to a local file, and replaced with a text hint
+ *    pointing at that path, so the agent can hand the image to
+ *    `inspect_image`. Models listed in `multimodalModels` are left
+ *    untouched (they receive image blocks directly).
  */
-import { readFile, stat } from "node:fs/promises";
-import { extname, isAbsolute, resolve as resolvePath } from "node:path";
+import { mkdir, readFile, stat, writeFile } from "node:fs/promises";
+import { extname, isAbsolute, join, resolve as resolvePath } from "node:path";
+import os from "node:os";
 import z from "@deepseek-ai/schemastery";
 import { defineTool } from "@deepseek-ai/dsh-tools";
 
 /** Cordis plugin name. */
 const name = "tool-vision";
-/** The tool registry this row contributes to. */
-const inject = ["tools"];
+/** The tool registry, the llm seam (waterfall), and the attachment store. */
+const inject = ["tools", "llm", "attachments"];
 
 const DEFAULT_DESCRIPTION =
   "Analyze an image using an external vision-capable model through an OpenAI-compatible API. " +
@@ -45,6 +50,12 @@ const Config = z.object({
   maxImageBytes: z.number().default(10 * 1024 * 1024),
   /** Tool description shown to the model; overrides the default. */
   description: z.string().default(DEFAULT_DESCRIPTION),
+  /** Bridge pasted images to text hints when the current model is text-only. */
+  bridgeTextOnly: z.boolean().default(true),
+  /** Export directory for bridged images; empty = system temp. */
+  bridgeExportDir: z.string().default(""),
+  /** Model ids that receive image blocks directly (never bridged). */
+  multimodalModels: z.array(z.string()).default([]),
 });
 
 const MIME_BY_EXT = {
@@ -59,6 +70,73 @@ const MIME_BY_EXT = {
   ".ico": "image/x-icon",
 };
 
+const EXT_BY_MEDIA = {
+  "image/png": ".png",
+  "image/jpeg": ".jpg",
+  "image/webp": ".webp",
+  "image/gif": ".gif",
+};
+
+/** True when any message carries an image content block. */
+function hasImageBlock(messages) {
+  return (messages ?? []).some((m) =>
+    Array.isArray(m?.content) && m.content.some((b) => b?.type === "image"),
+  );
+}
+
+/** Export one attachment to disk; returns the file path (cached per process). */
+const exportedPaths = new Map();
+async function exportImage(attachment, ctx, dir) {
+  const cached = exportedPaths.get(attachment.attachmentId);
+  if (cached) return cached;
+  const { data } = await ctx.attachments.readImage(attachment);
+  const ext = EXT_BY_MEDIA[attachment.mediaType] ?? ".img";
+  const safeName = attachment.name
+    ? attachment.name
+        .replace(/\.[^.]+$/, "")
+        .replace(/[^\w\-]+/g, "_")
+        .replace(/^_+|_+$/g, "")
+        .slice(0, 40)
+    : "";
+  const base = (safeName ? `${safeName}_` : "") + attachment.attachmentId.slice(0, 12);
+  const path = join(dir, `${base}${ext}`);
+  await writeFile(path, data);
+  exportedPaths.set(attachment.attachmentId, path);
+  return path;
+}
+
+/**
+ * Replace image content blocks with text hints pointing at exported files.
+ * Exported for unit testing; `ctx` only needs `attachments`.
+ */
+async function bridgeMessages(messages, ctx, dir) {
+  const next = [];
+  for (const message of messages) {
+    const content = message?.content;
+    if (!Array.isArray(content) || !content.some((b) => b?.type === "image")) {
+      next.push(message);
+      continue;
+    }
+    const blocks = [];
+    for (const block of content) {
+      if (block?.type !== "image") {
+        blocks.push(block);
+        continue;
+      }
+      const path = await exportImage(block.attachment, ctx, dir);
+      const name = block.attachment.name ? ` (${block.attachment.name})` : "";
+      blocks.push({
+        type: "text",
+        text:
+          `[User sent an image${name}, exported to: ${path}. ` +
+          `Inspect it with the inspect_image tool to see its content.]`,
+      });
+    }
+    next.push({ ...message, content: blocks });
+  }
+  return next;
+}
+
 function resolveApiKey(config) {
   if (config.apiKey) return config.apiKey;
   if (config.apiKeyEnv) {
@@ -146,6 +224,23 @@ async function callVision(config, imageUrl, question, detail, signal) {
 }
 
 function apply(ctx, config) {
+  // ── image bridge: text-only models get exported paths instead of pixels ──
+  if (config.bridgeTextOnly) {
+    const exportDir = config.bridgeExportDir || join(os.tmpdir(), "dsh-vision-bridge");
+    mkdir(exportDir, { recursive: true }).catch(() => {});
+    ctx.on("llm/stream", async function* (options, next) {
+      let effective = options;
+      try {
+        if (options?.messages && hasImageBlock(options.messages) && !config.multimodalModels.includes(options.model)) {
+          effective = { ...options, messages: await bridgeMessages(options.messages, ctx, exportDir) };
+        }
+      } catch (error) {
+        ctx.logger.warn(`[tool-vision] image bridge failed: ${String(error)}`);
+      }
+      yield* next(effective);
+    });
+  }
+
   ctx.tools.register(defineTool({
     name: "inspect_image",
     description: config.description,
@@ -178,4 +273,14 @@ function apply(ctx, config) {
   }));
 }
 
-export { Config, DEFAULT_DESCRIPTION, apply, inject, name };
+export {
+  Config,
+  DEFAULT_DESCRIPTION,
+  EXT_BY_MEDIA,
+  apply,
+  bridgeMessages,
+  exportImage,
+  hasImageBlock,
+  inject,
+  name,
+};

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-tool-vision",
-  "version": "0.1.0",
+  "version": "0.1.1",
   "description": "External vision model for DeepSeek Harness: inspect_image sends local images or http(s) image URLs to any OpenAI-compatible chat/completions endpoint and returns the vision model's text answer.",
   "type": "module",
   "main": "index.js",
@@ -42,3 +42,4 @@
   }
 }
 
+