dsh-tool-vision 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Scorp1o117
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,80 @@
+# dsh-tool-vision
+
+External vision model for [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness).
+
+DeepSeek's own models are text-only, and dsh-llm has no multimodal content
+block yet. This plugin bridges the gap with a model-facing tool that sends an
+image (local file, or http(s) URL) to **any OpenAI-compatible**
+`/chat/completions` endpoint that supports `image_url` content parts, and
+returns the vision model's textual answer into the agent loop.
+
+- Zero dependencies beyond the dsh SDK — works with any compatible endpoint:
+  OpenAI GPT-4o, Qwen-VL (DashScope), GLM-4V (Zhipu), Moonshot, Gemini
+  compatible endpoints, local Ollama, etc.
+- Registered on the **global tools layer**: every agent in the process can
+  call `inspect_image`.
+
+## Install
+
+Mount in a profile patch (`$DSH_HOME/profiles/<name>/cordis.patch.yml`):
+
+```yaml
+- insert:
+    - id: tool-vision
+      name: 'dsh-tool-vision'     # after: pnpm add dsh-tool-vision in the profile
+      config:
+        baseURL: 'https://api.openai.com/v1'
+        apiKeyEnv: 'VISION_API_KEY'
+        model: 'gpt-4o-mini'
+```
+
+Or load it from a local path without npm:
+
+```yaml
+    - id: tool-vision
+      name: './plugins/dsh-tool-vision/index.js'
+```
+
+## Config
+
+| Field | Default | Meaning |
+|---|---|---|
+| `baseURL` | `https://api.openai.com/v1` | OpenAI-compatible API base URL. |
+| `apiKey` | `''` | API key (takes precedence over env). |
+| `apiKeyEnv` | `VISION_API_KEY` | Env var holding the key. |
+| `model` | `gpt-4o-mini` | Vision model id. |
+| `maxTokens` | `1024` | Max output tokens. |
+| `timeoutMs` | `60000` | Per-request timeout. |
+| `maxImageBytes` | `10MB` | Largest accepted local image. |
+| `description` | default | Tool description shown to the model. |
+
+Key resolution order: `config.apiKey` → `process.env[apiKeyEnv]` →
+`process.env.OPENAI_API_KEY`.
+
+## Tool: `inspect_image`
+
+| Arg | Required | Meaning |
+|---|---|---|
+| `path` | ✅ | Image path (absolute, or relative to the current workspace) or http(s) URL. |
+| `question` | – | Optional specific question about the image. |
+| `detail` | – | `auto` / `low` / `high` resolution hint. |
+
+Example endpoints (`baseURL`):
+
+- **OpenAI**: `https://api.openai.com/v1` — `gpt-4o`, `gpt-4o-mini`
+- **Alibaba DashScope (Qwen-VL)**: `https://dashscope.aliyuncs.com/compatible-mode/v1` — `qwen-vl-plus`, `qwen-vl-max`
+- **Zhipu (GLM-4V)**: `https://open.bigmodel.cn/api/paas/v4` — `glm-4v-flash` (free tier), `glm-4v-plus`
+- **Moonshot (Kimi)**: `https://api.moonshot.cn/v1` — `moonshot-v1-8k-vision-preview`
+- **Ollama local**: `http://localhost:11434/v1` — `llama3.2-vision` (no key)
+
+## Limitations
+
+- The image enters the conversation as a text description (a transcript, not
+  pixels) — pixel-precise tasks may be inaccurate.
+- Images are base64-transferred; mind privacy and size limits.
+- Independent of the dsh-llm routing/retry system; failures return clear
+  errors to the agent.
+
+## License
+
+MIT

package/README.zh.md

@@ -0,0 +1,60 @@
+# dsh-tool-vision
+
+给 DeepSeek Harness 外接**视觉模型**的工具插件。DeepSeek 自家模型是纯文本的，而 dsh-llm 目前还没有多模态内容块类型——本插件用一条 OpenAI 兼容接口补上这个缺口：
+
+Agent 调用工具 `inspect_image`，把本地图片（或 http(s) 图片 URL）发给任意兼容 `image_url` 内容块的 `/chat/completions` 端点，把视觉模型的文字回答带回对话。
+
+- 注册在**全局工具层**：进程内所有 Agent 都能调用 `inspect_image`
+- 支持任意 OpenAI 兼容视觉端点：OpenAI GPT-4o / Qwen-VL (DashScope) / GLM-4V (智谱) / Moonshot / Gemini 兼容端点 / Ollama 本地等
+- 本地图片自动转 base64 data URL，检查格式与大小上限
+
+## 配置
+
+| 字段 | 默认值 | 含义 |
+|---|---|---|
+| `baseURL` | `https://api.openai.com/v1` | OpenAI 兼容 API 基地址 |
+| `apiKey` | `''` | API 密钥（优先于环境变量） |
+| `apiKeyEnv` | `VISION_API_KEY` | 存放密钥的环境变量名 |
+| `model` | `gpt-4o-mini` | 视觉模型 id |
+| `maxTokens` | `1024` | 视觉调用最大输出 token |
+| `timeoutMs` | `60000` | 单次请求超时 |
+| `maxImageBytes` | `10MB` | 本地图片大小上限 |
+| `description` | 默认描述 | 工具描述（模型可见） |
+
+密钥解析顺序：`config.apiKey` → `process.env[apiKeyEnv]` → `process.env.OPENAI_API_KEY`。
+
+## 挂载
+
+在 profile 的 `cordis.patch.yml` 里 insert：
+
+```yaml
+- insert:
+    - id: tool-vision
+      name: './plugins/dsh-tool-vision/index.js'
+      config:
+        baseURL: 'https://dashscope.aliyuncs.com/compatible-mode/v1'
+        apiKeyEnv: 'DASHSCOPE_API_KEY'
+        model: 'qwen-vl-plus'
+```
+
+## 工具：`inspect_image`
+
+| 参数 | 必填 | 说明 |
+|---|---|---|
+| `path` | ✅ | 图片路径（绝对路径，或相对当前 workspace）或 http(s) URL |
+| `question` | – | 针对图片的具体问题；缺省为"详细描述" |
+| `detail` | – | `auto` / `low` / `high`，图片分辨率提示 |
+
+示例端点（`baseURL` 写法）：
+
+- **OpenAI**：`https://api.openai.com/v1`，模型 `gpt-4o` / `gpt-4o-mini`
+- **阿里云百炼 (Qwen-VL)**：`https://dashscope.aliyuncs.com/compatible-mode/v1`，模型 `qwen-vl-plus` / `qwen-vl-max`
+- **智谱 (GLM-4V)**：`https://open.bigmodel.cn/api/paas/v4`，模型 `glm-4v-flash`（免费档）/ `glm-4v-plus`
+- **Moonshot (Kimi)**：`https://api.moonshot.cn/v1`，模型 `moonshot-v1-8k-vision-preview`
+- **Ollama 本地**：`http://localhost:11434/v1`，模型 `llama3.2-vision`（无需 key）
+
+## 局限
+
+- 图片内容以文字描述进入对话，Agent 得到的是"转述"而非像素——精确到像素的任务（如测量）可能不够准
+- 图片会在请求中 base64 传输，注意隐私与大小
+- 不依赖 dsh-llm 服务，因此不走模型路由/重试体系；失败会以明确的错误信息返回给 Agent

package/index.js

@@ -0,0 +1,181 @@
+/**
+ * dsh-tool-vision — external vision model for DeepSeek Harness.
+ *
+ * DeepSeek's own models are text-only, and dsh-llm has no multimodal content
+ * block type yet. This plugin bridges the gap with a model-facing tool that
+ * sends an image (local file, or http(s) URL) to any OpenAI-compatible
+ * chat/completions endpoint that supports `image_url` content parts, and
+ * returns the vision model's textual answer into the agent loop.
+ *
+ * Registered on the GLOBAL tools layer, so every agent in the process can
+ * call `inspect_image`.
+ */
+import { readFile, stat } from "node:fs/promises";
+import { extname, isAbsolute, resolve as resolvePath } from "node:path";
+import z from "@deepseek-ai/schemastery";
+import { defineTool } from "@deepseek-ai/dsh-tools";
+
+/** Cordis plugin name. */
+const name = "tool-vision";
+/** The tool registry this row contributes to. */
+const inject = ["tools"];
+
+const DEFAULT_DESCRIPTION =
+  "Analyze an image using an external vision-capable model through an OpenAI-compatible API. " +
+  "Provide the path to a local image file (absolute, or relative to the current workspace) or an http(s) URL, " +
+  "optionally with a specific question. Returns the vision model's textual description or answer. " +
+  "Use this whenever you need to read, describe, or extract information from image content, " +
+  "since the main model is text-only.";
+
+/** Runtime schema for the tool-vision row. */
+const Config = z.object({
+  /** Base URL of an OpenAI-compatible API, e.g. https://api.openai.com/v1 or https://dashscope.aliyuncs.com/compatible-mode/v1 */
+  baseURL: z.string().default("https://api.openai.com/v1"),
+  /** API key; takes precedence over apiKeyEnv. */
+  apiKey: z.string().default(""),
+  /** Environment variable holding the API key. */
+  apiKeyEnv: z.string().default("VISION_API_KEY"),
+  /** Vision model id served by the endpoint. */
+  model: z.string().default("gpt-4o-mini"),
+  /** Max output tokens for the vision call. */
+  maxTokens: z.number().default(1024),
+  /** Per-request timeout in milliseconds. */
+  timeoutMs: z.number().default(60000),
+  /** Largest local image accepted, in bytes. */
+  maxImageBytes: z.number().default(10 * 1024 * 1024),
+  /** Tool description shown to the model; overrides the default. */
+  description: z.string().default(DEFAULT_DESCRIPTION),
+});
+
+const MIME_BY_EXT = {
+  ".png": "image/png",
+  ".jpg": "image/jpeg",
+  ".jpeg": "image/jpeg",
+  ".gif": "image/gif",
+  ".webp": "image/webp",
+  ".bmp": "image/bmp",
+  ".avif": "image/avif",
+  ".svg": "image/svg+xml",
+  ".ico": "image/x-icon",
+};
+
+function resolveApiKey(config) {
+  if (config.apiKey) return config.apiKey;
+  if (config.apiKeyEnv) {
+    const fromEnv = process.env[config.apiKeyEnv];
+    if (fromEnv) return fromEnv;
+  }
+  return process.env.OPENAI_API_KEY ?? "";
+}
+
+/** Turn a tool argument into an image_url payload: local file -> data URL, http(s) -> as-is. */
+async function toImageUrl(target, cwd, config) {
+  if (/^https?:\/\//i.test(target)) return { url: target, note: target };
+  const abs = isAbsolute(target) ? target : resolvePath(cwd, target);
+  const info = await stat(abs).catch(() => null);
+  if (!info) throw new Error(`image not found: ${abs}`);
+  if (info.size > config.maxImageBytes) {
+    throw new Error(
+      `image too large: ${abs} (${info.size} bytes, limit ${config.maxImageBytes})`,
+    );
+  }
+  const mime = MIME_BY_EXT[extname(abs).toLowerCase()];
+  if (!mime) {
+    throw new Error(
+      `unsupported image extension: ${abs} (supported: ${Object.keys(MIME_BY_EXT).join(", ")})`,
+    );
+  }
+  const data = await readFile(abs);
+  return { url: `data:${mime};base64,${data.toString("base64")}`, note: abs };
+}
+
+/** One OpenAI-compatible chat/completions call with an image_url content part. */
+async function callVision(config, imageUrl, question, detail, signal) {
+  const key = resolveApiKey(config);
+  if (!key) {
+    throw new Error(
+      `vision API key missing: set the plugin config (apiKey / apiKeyEnv) or the OPENAI_API_KEY environment variable`,
+    );
+  }
+  const base = config.baseURL.endsWith("/") ? config.baseURL : `${config.baseURL}/`;
+  const endpoint = new URL("chat/completions", base);
+  const controller = new AbortController();
+  const timer = setTimeout(
+    () => controller.abort(new Error(`vision request timed out after ${config.timeoutMs}ms`)),
+    config.timeoutMs,
+  );
+  const onSignalAbort = () => controller.abort(signal?.reason);
+  if (signal) {
+    if (signal.aborted) controller.abort();
+    else signal.addEventListener("abort", onSignalAbort, { once: true });
+  }
+  const content = [
+    { type: "text", text: question || "Describe this image in detail, including all key visual elements, text, and context you can see." },
+    { type: "image_url", image_url: detail ? { url: imageUrl, detail } : { url: imageUrl } },
+  ];
+  try {
+    const response = await fetch(endpoint, {
+      method: "POST",
+      headers: {
+        "Content-Type": "application/json",
+        Authorization: `Bearer ${key}`,
+      },
+      body: JSON.stringify({
+        model: config.model,
+        messages: [{ role: "user", content }],
+        max_tokens: config.maxTokens,
+      }),
+      signal: controller.signal,
+    });
+    const body = await response.json().catch(() => null);
+    if (!response.ok) {
+      const detailText = body?.error?.message ?? response.statusText;
+      throw new Error(
+        `vision endpoint returned ${response.status}: ${detailText} (endpoint ${endpoint})`,
+      );
+    }
+    const answer = body?.choices?.[0]?.message?.content;
+    if (typeof answer !== "string" || !answer.trim()) {
+      throw new Error("vision endpoint returned an empty response");
+    }
+    return answer.trim();
+  } finally {
+    clearTimeout(timer);
+    signal?.removeEventListener("abort", onSignalAbort);
+  }
+}
+
+function apply(ctx, config) {
+  ctx.tools.register(defineTool({
+    name: "inspect_image",
+    description: config.description,
+    parameters: {
+      path: {
+        type: "string",
+        required: true,
+        description: "Path to the image file (absolute, or relative to the current workspace) or an http(s) URL.",
+      },
+      question: {
+        type: "string",
+        description: "Optional specific question about the image. Omit for a general detailed description.",
+      },
+      detail: {
+        type: "string",
+        enum: ["auto", "low", "high"],
+        description: "Optional image resolution hint for the vision API (auto by default).",
+      },
+    },
+    output: {
+      schema: { type: "string" },
+      render: (_args, value) => [{ type: "text", text: value }],
+    },
+    async execute(args, exec) {
+      const cwd = exec.agent?.session?.header?.cwd ?? process.cwd();
+      const { url, note } = await toImageUrl(args.path, cwd, config);
+      const answer = await callVision(config, url, args.question, args.detail, exec.signal);
+      return note === url ? answer : `${answer}\n\n(image: ${note})`;
+    },
+  }));
+}
+
+export { Config, DEFAULT_DESCRIPTION, apply, inject, name };

package/package.json

@@ -0,0 +1,44 @@
+{
+  "name": "dsh-tool-vision",
+  "version": "0.1.0",
+  "description": "External vision model for DeepSeek Harness: inspect_image sends local images or http(s) image URLs to any OpenAI-compatible chat/completions endpoint and returns the vision model's text answer.",
+  "type": "module",
+  "main": "index.js",
+  "exports": {
+    ".": "./index.js"
+  },
+  "files": [
+    "index.js",
+    "README.md",
+    "README.zh.md",
+    "LICENSE"
+  ],
+  "keywords": [
+    "dsh",
+    "dsh-plugin",
+    "deepseek-harness",
+    "vision",
+    "multimodal",
+    "image",
+    "openai-compatible",
+    "qwen-vl",
+    "glm-4v",
+    "ollama"
+  ],
+  "license": "MIT",
+  "engines": {
+    "node": ">=20.18"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/Scorp1o117/dsh-tool-vision.git"
+  },
+  "bugs": {
+    "url": "https://github.com/Scorp1o117/dsh-tool-vision/issues"
+  },
+  "dependencies": {
+    "@deepseek-ai/dsh-tools": "^0.1.0-rc.6",
+    "@deepseek-ai/schemastery": "^3.18.1"
+  }
+}
+