@yandy0725/pi-vision-tools 0.1.0

package/README.md

@@ -0,0 +1,130 @@
+# pi-vision-tools
+
+Lets non-multimodal models analyze images by delegating to a configured vision model. A single `describe_image` tool + `/vision` command.
+
+## Features
+
+- **One tool** (`describe_image`) that sends an image + prompt to a vision-capable model and returns the text result to the calling model
+- **Calling model controls cost/quality per call**: `compress` (on/off), `reasoning` (off through xhigh), and the prompt itself — no preconfiguration needed
+- **Auto enable/disable** by calling model modality: if the current model already has image input, the tool disables itself; otherwise it's on
+- **Footer indicator** (`👁 provider/model`) visible when the tool is active and a vision model is configured
+- **No `/reload` required**: config changes take effect immediately
+
+## How it works
+
+```
+calling model  →  describe_image  →  vision model  →  text back to calling model
+ (no vision)       (image+prompt)      (sees image)
+```
+
+1. The calling model invokes `describe_image` with an image and a prompt
+2. The tool decodes the image, optionally compresses it with sharp, then calls the configured vision model
+3. The vision model's text answer is returned as the tool result
+
+## Install
+
+```bash
+pi install npm:@yandy0725/pi-vision-tools
+```
+
+Or add to `~/.pi/agent/settings.json`:
+
+```json
+{
+  "packages": ["npm:@yandy0725/pi-vision-tools"]
+}
+```
+
+## Configuration
+
+### `/vision` command
+
+| Command | What it does |
+|---------|-------------|
+| `/vision` or `/vision status` | Show current config: provider/model, enabled state, effective on/off, whether the calling model has vision |
+| `/vision config provider <p>` | Set the vision model provider (e.g. `openai`, `anthropic`) |
+| `/vision config model <m>` | Set the vision model ID (e.g. `gpt-4o`, `claude-sonnet-4-20250514`) |
+| `/vision config default-reasoning <level>` | Set default reasoning depth: `off`, `minimal`, `low`, `medium`, `high`, `xhigh` |
+| `/vision on` | Force-enable the tool (even if the calling model has vision) |
+| `/vision off` | Force-disable the tool |
+| `/vision auto` | Auto mode: tool enabled only when the calling model lacks image input (default) |
+
+Config is persisted to `~/.pi/agent/vision-tools.json` and takes effect immediately — no `/reload` needed.
+
+### Optional: sharp
+
+Install sharp for automatic image compression before sending:
+
+```bash
+npm install sharp
+```
+
+sharp is optional. Without it, images are sent as-is (no error). Compression downsamples the longest edge to ≤1568px, removes alpha, and converts to JPEG.
+
+#### Environment variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `PI_VISION_MAX_DIM` | `1568` | Longest-edge pixel limit (1–10000) |
+| `PI_VISION_JPEG_QUALITY` | `85` | JPEG quality (1–100) |
+
+Set `compress: false` on any call to skip compression for pixel-perfect needs (reading coordinates, inspecting tiny UI elements).
+
+## Tool reference
+
+```
+describe_image(image_path: string, prompt: string, compress?: boolean, reasoning?: string)
+```
+
+| Parameter | Required | Default | Description |
+|-----------|:--------:|---------|-------------|
+| `image_path` | yes | — | File path, `data:` URL, or raw base64 (>100 chars) |
+| `prompt` | yes | — | Instruction for the vision model |
+| `compress` | no | `true` | Compress before sending; set `false` for pixel-perfect |
+| `reasoning` | no | `off` | Reasoning effort: `off`, `minimal`, `low`, `medium`, `high`, `xhigh` |
+
+### Example tool call
+
+```json
+{
+  "image_path": "/home/user/screenshot.png",
+  "prompt": "Describe what you see in this screenshot.",
+  "compress": true,
+  "reasoning": "high"
+}
+```
+
+### Example prompts
+
+| Goal | prompt |
+|------|--------|
+| Describe an image | `"Describe this image in detail."` |
+| Read coordinates/position | `"What are the pixel coordinates of the submit button?"` |
+| Extract text (OCR) | `"Extract all visible text from this image."` |
+| Find UI bugs | `"Inspect this screenshot for layout, alignment, or text overflow issues."` |
+| Explain a diagram | `"Explain this architecture diagram step by step."` |
+| Analyze an error | `"What does this error message mean and how can it be fixed?"` |
+
+### Reasoning levels
+
+| Level | When to use |
+|-------|-------------|
+| `off` | Simple description, text extraction, basic Q&A |
+| `minimal` | Quick glance, "what is this?" |
+| `low` | Slightly more thought, moderate detail |
+| `medium` | Detailed description, UI inspection |
+| `high` | Complex analysis, architecture diagrams, code screenshots |
+| `xhigh` | Deep reasoning, bug hunting, multi-step visual puzzles |
+
+## Image formats
+
+Supported: PNG, JPEG, GIF, WebP, BMP.
+
+Input can be:
+- A file path (`/path/to/image.png`, `./relative.png`, `~`-prefixed)
+- A `data:` URL (`data:image/png;base64,...`)
+- Raw base64 (string >100 characters, auto-detected)
+
+## License
+
+MIT

package/README.zh.md

@@ -0,0 +1,130 @@
+# pi-vision-tools
+
+让非多模态模型通过委托给配置好的视觉模型来分析图像。提供一个 `describe_image` 工具 + `/vision` 命令。
+
+## 功能
+
+- **一个工具**（`describe_image`），将图像 + 提示词发送给支持视觉的模型，并将文本结果返回给调用方模型
+- **调用方模型按需控制成本/质量**：`compress`（开关）、`reasoning`（从 off 到 xhigh）以及提示词本身——无需预配置
+- **按调用方模型模态自动开关**：如果当前模型已支持图像输入，工具自动禁用；否则启用
+- **页脚指示器**（`👁 provider/model`），在工具激活且视觉模型已配置时可见
+- **无需 `/reload`**：配置更改即时生效
+
+## 工作原理
+
+```
+调用方模型  →  describe_image  →  视觉模型  →  文本返回给调用方模型
+（无视觉能力）     （图像+提示词）      （能看图像）
+```
+
+1. 调用方模型调用 `describe_image`，传入图像和提示词
+2. 工具解码图像，可选使用 sharp 压缩，然后调用配置好的视觉模型
+3. 视觉模型的文本回答作为工具结果返回
+
+## 安装
+
+```bash
+pi install npm:@yandy0725/pi-vision-tools
+```
+
+或在 `~/.pi/agent/settings.json` 中添加：
+
+```json
+{
+  "packages": ["npm:@yandy0725/pi-vision-tools"]
+}
+```
+
+## 配置
+
+### `/vision` 命令
+
+| 命令 | 功能 |
+|---------|------|
+| `/vision` 或 `/vision status` | 显示当前配置：provider/model、启用状态、生效开关、调用方模型是否支持视觉 |
+| `/vision config provider <p>` | 设置视觉模型提供商（如 `openai`、`anthropic`） |
+| `/vision config model <m>` | 设置视觉模型 ID（如 `gpt-4o`、`claude-sonnet-4-20250514`） |
+| `/vision config default-reasoning <level>` | 设置默认推理深度：`off`、`minimal`、`low`、`medium`、`high`、`xhigh` |
+| `/vision on` | 强制启用工具（即使调用方模型支持视觉） |
+| `/vision off` | 强制禁用工具 |
+| `/vision auto` | 自动模式：仅当调用方模型不支持图像输入时启用（默认） |
+
+配置持久化到 `~/.pi/agent/vision-tools.json`，即时生效——无需 `/reload`。
+
+### 可选：sharp
+
+安装 sharp 以在发送前自动压缩图像：
+
+```bash
+npm install sharp
+```
+
+sharp 是可选的。没有它，图像将原样发送（不报错）。压缩会将最长边降采样至 ≤1568px，去除 alpha 通道，并转换为 JPEG 格式。
+
+#### 环境变量
+
+| 变量 | 默认值 | 说明 |
+|----------|---------|------|
+| `PI_VISION_MAX_DIM` | `1568` | 最长边像素上限（1–10000） |
+| `PI_VISION_JPEG_QUALITY` | `85` | JPEG 质量（1–100） |
+
+在任何调用中设置 `compress: false` 可跳过压缩，用于需要像素级精度的场景（读取坐标、检查微小 UI 元素）。
+
+## 工具参考
+
+```
+describe_image(image_path: string, prompt: string, compress?: boolean, reasoning?: string)
+```
+
+| 参数 | 必填 | 默认值 | 说明 |
+|-----------|:--:|---------|------|
+| `image_path` | 是 | — | 文件路径、`data:` URL 或原始 base64（>100 字符） |
+| `prompt` | 是 | — | 给视觉模型的指令 |
+| `compress` | 否 | `true` | 发送前是否压缩；设为 `false` 以获取像素级精度 |
+| `reasoning` | 否 | `off` | 推理力度：`off`、`minimal`、`low`、`medium`、`high`、`xhigh` |
+
+### 示例工具调用
+
+```json
+{
+  "image_path": "/home/user/screenshot.png",
+  "prompt": "请描述这张截图中看到的内容。",
+  "compress": true,
+  "reasoning": "high"
+}
+```
+
+### 示例提示词
+
+| 用途 | 提示词 |
+|------|--------|
+| 描述图像 | `"请详细描述这张图片。"` |
+| 读取坐标/位置 | `"提交按钮的像素坐标是多少？"` |
+| 提取文字（OCR） | `"提取这张图片中所有可见的文字。"` |
+| 查找 UI 缺陷 | `"检查这张截图中是否存在布局、对齐或文字溢出问题。"` |
+| 解释图表 | `"请逐步解释这个架构图。"` |
+| 分析错误 | `"这个错误信息是什么意思，如何修复？"` |
+
+### 推理级别
+
+| 级别 | 适用场景 |
+|-------|------|
+| `off` | 简单描述、文字提取、基础问答 |
+| `minimal` | 快速浏览，"这是什么？" |
+| `low` | 略多思考、中等细节 |
+| `medium` | 详细描述、UI 检查 |
+| `high` | 复杂分析、架构图、代码截图 |
+| `xhigh` | 深度推理、bug 排查、多步骤视觉谜题 |
+
+## 支持的图像格式
+
+支持：PNG、JPEG、GIF、WebP、BMP。
+
+输入可以是：
+- 文件路径（`/path/to/image.png`、`./relative.png`，支持 `~` 前缀）
+- `data:` URL（`data:image/png;base64,...`）
+- 原始 base64（字符串 >100 字符，自动检测）
+
+## 许可证
+
+MIT

package/index.ts

@@ -0,0 +1,218 @@
+import { StringEnum } from "@earendil-works/pi-ai";
+import { complete } from "@earendil-works/pi-ai/compat";
+import { type ExtensionAPI, type ExtensionContext, getAgentDir } from "@earendil-works/pi-coding-agent";
+import { Text } from "@earendil-works/pi-tui";
+import { Type } from "typebox";
+import { compressImage, readCompressionSettings } from "./src/compress.js";
+import { loadConfig, saveConfig, type VisionConfig } from "./src/config.js";
+import { type DecodedImage, decodeImage } from "./src/image.js";
+import { effectiveReasoning, reasoningToOptions, type VisionReasoning } from "./src/reasoning.js";
+import { callingModelHasVision, effectiveEnabled, footerLabel } from "./src/state.js";
+import { callVision, resolveVisionModel } from "./src/vision.js";
+
+const TOOL_NAME = "describe_image";
+const STATUS_KEY = "pi-vision";
+
+export default function (pi: ExtensionAPI) {
+	let config: VisionConfig = { enabled: "auto" };
+	let enabled = false;
+
+	const refresh = (ctx: ExtensionContext) => {
+		enabled = effectiveEnabled(config, ctx.model);
+		const active = pi.getActiveTools();
+		if (enabled && !active.includes(TOOL_NAME)) {
+			pi.setActiveTools([...active, TOOL_NAME]);
+		} else if (!enabled && active.includes(TOOL_NAME)) {
+			pi.setActiveTools(active.filter((t) => t !== TOOL_NAME));
+		}
+		if (ctx.hasUI) {
+			const label = footerLabel(config, enabled);
+			ctx.ui.setStatus(STATUS_KEY, label);
+		}
+	};
+
+	pi.on("session_start", async (_event, ctx) => {
+		config = await loadConfig(getAgentDir());
+		refresh(ctx);
+	});
+	pi.on("model_select", async (_event, ctx) => {
+		refresh(ctx);
+	});
+
+	pi.registerTool({
+		name: TOOL_NAME,
+		label: "Describe Image",
+		description:
+			"Analyze an image by delegating to a vision-capable model. Lets non-multimodal models understand images. " +
+			"`image_path` is a file path, data: URL, or raw base64 (>100 chars). " +
+			"`compress` (default true) downscales/strips to speed up; set false for pixel-perfect needs. " +
+			"`reasoning` controls the vision model's thinking effort (off/minimal/low/medium/high/xhigh).",
+		promptSnippet: "describe_image: delegate image analysis to a vision model (non-multimodal models).",
+		promptGuidelines: [
+			"Use describe_image when you need to understand an image you cannot see (the calling model lacks vision).",
+			"Set compress:false when you need pixel-perfect accuracy (reading coordinates, tiny UI elements).",
+			"Set reasoning:'high'/'xhigh' for complex visual analysis (architecture diagrams, bug hunting).",
+		],
+		parameters: Type.Object({
+			image_path: Type.String({ description: "File path, data: URL, or raw base64 (>100 chars)." }),
+			prompt: Type.String({
+				description: "Instruction for the vision model, e.g. 'describe', 'extract text', 'find the bug'.",
+			}),
+			compress: Type.Optional(
+				Type.Boolean({ default: true, description: "Compress image before sending (default true)." }),
+			),
+			reasoning: Type.Optional(
+				StringEnum(["off", "minimal", "low", "medium", "high", "xhigh"] as const, {
+					description: "Vision model reasoning effort. Default off.",
+				}),
+			),
+		}),
+		renderCall(args, theme) {
+			const p = args as { image_path?: string; prompt?: string };
+			const target = p.image_path ? (p.image_path.length > 40 ? `${p.image_path.slice(0, 37)}...` : p.image_path) : "...";
+			return new Text(
+				theme.fg("toolTitle", theme.bold("describe_image ")) +
+					theme.fg("accent", target) +
+					theme.fg("dim", ` · ${p.prompt?.slice(0, 30) ?? ""}`),
+				0,
+				0,
+			);
+		},
+		renderResult(result, { expanded }, theme) {
+			const text = result.content?.[0];
+			const body = text?.type === "text" ? text.text : "";
+			const lines = body.split("\n");
+			if (!expanded) {
+				const preview = lines.slice(0, 6);
+				if (lines.length > 6) preview.push(theme.fg("dim", `... ${lines.length - 6} more lines · ctrl+o to expand`));
+				return new Text(preview.join("\n"), 0, 0);
+			}
+			return new Text(body, 0, 0);
+		},
+		async execute(_toolCallId, params, signal, onUpdate, ctx) {
+			if (!enabled) {
+				return {
+					content: [{ type: "text", text: "describe_image is disabled. Run /vision on to enable." }],
+					details: { error: "disabled" },
+					isError: true,
+				};
+			}
+			const p = params as { image_path: string; prompt: string; compress?: boolean; reasoning?: VisionReasoning };
+
+			const resolved = resolveVisionModel(ctx.modelRegistry, config);
+			if (!resolved.ok) {
+				return { content: [{ type: "text", text: resolved.error }], details: { error: resolved.error }, isError: true };
+			}
+
+			const auth = await ctx.modelRegistry.getApiKeyAndHeaders(resolved.model);
+			if (!auth.ok || !auth.apiKey) {
+				const msg = auth.ok ? `No API key for ${config.provider}/${config.model}` : auth.error;
+				return { content: [{ type: "text", text: msg }], details: { error: msg }, isError: true };
+			}
+
+			onUpdate?.({ content: [{ type: "text", text: "Decoding image..." }], details: {} });
+
+			let image: DecodedImage;
+			try {
+				image = await decodeImage(p.image_path);
+			} catch (e) {
+				const msg = e instanceof Error ? e.message : String(e);
+				return { content: [{ type: "text", text: `Image decode failed: ${msg}` }], details: { error: msg }, isError: true };
+			}
+
+			const doCompress = p.compress !== false;
+			let compressed = false;
+			let mimeType = image.mimeType;
+			if (doCompress) {
+				onUpdate?.({ content: [{ type: "text", text: "Compressing..." }], details: {} });
+				const out = await compressImage(image, readCompressionSettings());
+				compressed = out !== image;
+				mimeType = out.mimeType;
+				image = out;
+			}
+
+			onUpdate?.({ content: [{ type: "text", text: "Analyzing image..." }], details: {} });
+
+			const reasoningLevel = effectiveReasoning(p.reasoning, config.defaultReasoning);
+			const reasoning = reasoningToOptions(reasoningLevel);
+			const result = await callVision(
+				{
+					model: resolved.model,
+					auth: { apiKey: auth.apiKey, headers: auth.headers },
+					prompt: p.prompt,
+					images: [image],
+					reasoning,
+					signal: signal ?? undefined,
+				},
+				complete,
+			);
+
+			if (result.errorMessage) {
+				return {
+					content: [{ type: "text", text: `Vision model error: ${result.errorMessage}` }],
+					details: { error: result.errorMessage, model: `${config.provider}/${config.model}` },
+					isError: true,
+				};
+			}
+
+			return {
+				content: [{ type: "text", text: result.text }],
+				details: {
+					model: `${config.provider}/${config.model}`,
+					usage: result.usage,
+					compressed,
+					mimeType,
+					reasoning: reasoningLevel,
+				},
+			};
+		},
+	});
+
+	pi.registerCommand("vision", {
+		description: "Configure the vision model for describe_image (/vision config | on | off | status)",
+		handler: async (args, ctx) => {
+			const parts = args.trim().split(/\s+/).filter(Boolean);
+			const sub = parts[0];
+
+			const notifyConfig = () => {
+				const target = config.provider && config.model ? `${config.provider}/${config.model}` : "(unconfigured)";
+				const visionCap = callingModelHasVision(ctx.model) ? "yes" : "no";
+				ctx.ui.notify(
+					`vision: ${target}\nenabled: ${config.enabled} (effective: ${enabled ? "on" : "off"})\ncalling model has vision: ${visionCap}`,
+					"info",
+				);
+			};
+
+			if (!sub || sub === "status") {
+				notifyConfig();
+				return;
+			}
+
+			if (sub === "on" || sub === "off" || sub === "auto") {
+				config = { ...config, enabled: sub as VisionConfig["enabled"] };
+				await saveConfig(getAgentDir(), config);
+				refresh(ctx);
+				ctx.ui.notify(`vision ${sub}`, "info");
+				return;
+			}
+
+			if (sub === "config") {
+				const key = parts[1];
+				const val = parts[2];
+				if (key === "provider" && val) config = { ...config, provider: val };
+				else if (key === "model" && val) config = { ...config, model: val };
+				else if (key === "default-reasoning" && val) config = { ...config, defaultReasoning: val as VisionReasoning };
+				else {
+					ctx.ui.notify("Usage: /vision config provider <p> | model <m> | default-reasoning <level>", "warning");
+					return;
+				}
+				await saveConfig(getAgentDir(), config);
+				refresh(ctx);
+				ctx.ui.notify(`vision ${key} = ${val}`, "info");
+				return;
+			}
+
+			ctx.ui.notify("Usage: /vision [config provider <p> | config model <m> | on | off | auto | status]", "warning");
+		},
+	});
+}

package/package.json

@@ -0,0 +1,45 @@
+{
+	"name": "@yandy0725/pi-vision-tools",
+	"publishConfig": {
+		"access": "public"
+	},
+	"version": "0.1.0",
+	"description": "pi package adding a describe_image tool that lets non-multimodal models delegate image analysis to a vision model",
+	"license": "MIT",
+	"repository": {
+		"type": "git",
+		"url": "https://github.com/yandy/pi-packages",
+		"directory": "pi-vision-tools"
+	},
+	"type": "module",
+	"keywords": [
+		"pi-package"
+	],
+	"files": [
+		"index.ts",
+		"src/"
+	],
+	"scripts": {
+		"test": "vitest run",
+		"test:watch": "vitest",
+		"typecheck": "tsc --noEmit",
+		"lint": "biome lint .",
+		"format": "biome format --write .",
+		"check": "biome check ."
+	},
+	"pi": {
+		"extensions": [
+			"./index.ts"
+		]
+	},
+	"peerDependencies": {
+		"@earendil-works/pi-coding-agent": ">=0.74.0"
+	},
+	"dependencies": {
+		"@earendil-works/pi-ai": "^0.80.2"
+	},
+	"devDependencies": {
+		"@earendil-works/pi-tui": "^0.79.9",
+		"typebox": "^1.1.38"
+	}
+}

package/src/compress.ts

@@ -0,0 +1,67 @@
+import type { DecodedImage } from "./image.js";
+
+export interface CompressionSettings {
+	maxDim: number;
+	jpegQuality: number;
+}
+
+const DEFAULT_MAX_DIM = 1568;
+const DEFAULT_JPEG_QUALITY = 85;
+
+function parseIntOrDefault(v: string | undefined, dflt: number, min: number, max: number): number {
+	if (v == null) return dflt;
+	const n = Number.parseInt(v, 10);
+	if (!Number.isFinite(n) || n < min || n > max) return dflt;
+	return n;
+}
+
+export function readCompressionSettings(env: NodeJS.ProcessEnv = process.env): CompressionSettings {
+	return {
+		maxDim: parseIntOrDefault(env.PI_VISION_MAX_DIM, DEFAULT_MAX_DIM, 1, 10_000),
+		jpegQuality: parseIntOrDefault(env.PI_VISION_JPEG_QUALITY, DEFAULT_JPEG_QUALITY, 1, 100),
+	};
+}
+
+// Minimal structural type for the bits of sharp we use.
+interface SharpPipeline {
+	resize(opts: { width?: number; height?: number; withoutEnlargement: boolean; fit: string }): SharpPipeline;
+	removeAlpha(): SharpPipeline;
+	jpeg(opts: { quality: number }): SharpPipeline;
+	toBuffer(): Promise<Buffer>;
+}
+type SharpModule = (data: Buffer) => SharpPipeline;
+
+const defaultSharpLoader = async (): Promise<SharpModule | null> => {
+	try {
+		// @ts-ignore sharp is an optional dependency, may not be installed
+		const mod = (await import("sharp")) as { default?: SharpModule } & SharpModule;
+		return mod.default ?? (mod as unknown as SharpModule);
+	} catch {
+		return null;
+	}
+};
+
+export async function compressImage(
+	image: DecodedImage,
+	settings: CompressionSettings,
+	_sharpLoader: () => Promise<SharpModule | null> = defaultSharpLoader,
+): Promise<DecodedImage> {
+	let sharp: SharpModule | null;
+	try {
+		sharp = await _sharpLoader();
+	} catch {
+		return image;
+	}
+	if (!sharp) return image;
+
+	try {
+		const buf = await sharp(image.data)
+			.resize({ width: settings.maxDim, height: settings.maxDim, withoutEnlargement: true, fit: "inside" })
+			.removeAlpha()
+			.jpeg({ quality: settings.jpegQuality })
+			.toBuffer();
+		return { data: buf, mimeType: "image/jpeg" };
+	} catch {
+		return image;
+	}
+}

package/src/config.ts

@@ -0,0 +1,79 @@
+import { mkdir, readFile, rename, writeFile } from "node:fs/promises";
+import { join } from "node:path";
+import type { ThinkingLevel } from "@earendil-works/pi-ai";
+
+export type VisionEnabledState = "auto" | "on" | "off";
+
+export interface VisionConfig {
+	provider?: string;
+	model?: string;
+	enabled: VisionEnabledState;
+	defaultReasoning?: ThinkingLevel | "off";
+}
+
+export const DEFAULT_CONFIG: VisionConfig = { enabled: "auto" };
+
+const ENABLED_STATES = ["auto", "on", "off"] as const;
+const REASONING_LEVELS = ["off", "minimal", "low", "medium", "high", "xhigh"] as const;
+
+export function configPath(agentDir: string): string {
+	return join(agentDir, "vision-tools.json");
+}
+
+export function parseConfig(raw: unknown): VisionConfig {
+	if (raw == null || typeof raw !== "object") return { ...DEFAULT_CONFIG };
+	const obj = raw as Record<string, unknown>;
+
+	const cfg: VisionConfig = { enabled: "auto" };
+
+	if (obj.provider !== undefined) {
+		if (typeof obj.provider !== "string" || obj.provider.length === 0) {
+			throw new Error("vision-tools config: provider must be a non-empty string");
+		}
+		cfg.provider = obj.provider;
+	}
+
+	if (obj.model !== undefined) {
+		if (typeof obj.model !== "string" || obj.model.length === 0) {
+			throw new Error("vision-tools config: model must be a non-empty string");
+		}
+		cfg.model = obj.model;
+	}
+
+	if (obj.enabled !== undefined) {
+		if (typeof obj.enabled !== "string" || !ENABLED_STATES.includes(obj.enabled as VisionEnabledState)) {
+			throw new Error(`vision-tools config: enabled must be one of ${ENABLED_STATES.join(", ")}`);
+		}
+		cfg.enabled = obj.enabled as VisionEnabledState;
+	}
+
+	if (obj.defaultReasoning !== undefined) {
+		if (
+			typeof obj.defaultReasoning !== "string" ||
+			!REASONING_LEVELS.includes(obj.defaultReasoning as (typeof REASONING_LEVELS)[number])
+		) {
+			throw new Error(`vision-tools config: defaultReasoning must be one of ${REASONING_LEVELS.join(", ")}`);
+		}
+		cfg.defaultReasoning = obj.defaultReasoning as ThinkingLevel | "off";
+	}
+
+	return cfg;
+}
+
+export async function loadConfig(agentDir: string): Promise<VisionConfig> {
+	try {
+		const text = await readFile(configPath(agentDir), "utf8");
+		return parseConfig(JSON.parse(text));
+	} catch {
+		return { ...DEFAULT_CONFIG };
+	}
+}
+
+export async function saveConfig(agentDir: string, config: VisionConfig): Promise<void> {
+	await mkdir(agentDir, { recursive: true });
+	const text = JSON.stringify(config);
+	const target = configPath(agentDir);
+	const tmp = `${target}.tmp`;
+	await writeFile(tmp, text, "utf8");
+	await rename(tmp, target);
+}

package/src/image.ts

@@ -0,0 +1,77 @@
+import { readFile } from "node:fs/promises";
+import { extname } from "node:path";
+
+export interface DecodedImage {
+	data: Buffer;
+	mimeType: string;
+}
+
+export interface ImageReadOptions {
+	readFile?: (path: string) => Promise<Buffer>;
+}
+
+const EXT_TO_MIME: Record<string, string> = {
+	".png": "image/png",
+	".jpg": "image/jpeg",
+	".jpeg": "image/jpeg",
+	".gif": "image/gif",
+	".webp": "image/webp",
+	".bmp": "image/bmp",
+};
+
+const SUPPORTED_MIMES = new Set(Object.values(EXT_TO_MIME));
+
+const DATA_URL_RE = /^data:([^;]+)?;base64,(.*)$/s;
+
+function looksLikePath(s: string): boolean {
+	if (s.startsWith("/")) return true;
+	if (s.startsWith("./") || s.startsWith("../")) return true;
+	if (s.startsWith("~")) return true;
+	// has a dot-extension and is short enough not to be base64
+	const ext = extname(s).toLowerCase();
+	if (ext && EXT_TO_MIME[ext] && s.length <= 100) return true;
+	return false;
+}
+
+export async function decodeImage(imagePath: string, opts?: ImageReadOptions): Promise<DecodedImage> {
+	const src = imagePath?.trim();
+	if (!src) throw new Error("image_path is required");
+
+	// 1. data URL
+	const m = DATA_URL_RE.exec(src);
+	if (m) {
+		const mime = (m[1] || "").toLowerCase();
+		if (!SUPPORTED_MIMES.has(mime)) {
+			throw new Error(`Unsupported image mime type: ${mime || "(missing)"}`);
+		}
+		const data = Buffer.from(m[2], "base64");
+		if (data.length === 0) throw new Error("Empty image data in data URL");
+		return { data, mimeType: mime };
+	}
+
+	// 2. file path
+	if (looksLikePath(src)) {
+		const ext = extname(src).toLowerCase();
+		const mime = EXT_TO_MIME[ext];
+		if (!mime) throw new Error(`Unsupported image extension: ${ext || "(none)"}`);
+		const read = opts?.readFile ?? readFile;
+		let data: Buffer;
+		try {
+			data = await read(src);
+		} catch (e) {
+			const msg = e instanceof Error ? e.message : String(e);
+			throw new Error(`Failed to read image file ${src}: ${msg}`);
+		}
+		if (data.length === 0) throw new Error(`Empty image file: ${src}`);
+		return { data, mimeType: mime };
+	}
+
+	// 3. raw base64 (>100 chars)
+	if (src.length > 100) {
+		const data = Buffer.from(src, "base64");
+		if (data.length === 0) throw new Error("Invalid base64 image data");
+		return { data, mimeType: "image/png" };
+	}
+
+	throw new Error("image_path must be a file path, a data: URL, or raw base64 (>100 chars)");
+}

package/src/reasoning.ts

@@ -0,0 +1,22 @@
+import type { ThinkingLevel } from "@earendil-works/pi-ai";
+
+export type VisionReasoning = "off" | ThinkingLevel;
+
+export interface ReasoningOptions {
+	reasoningEffort?: ThinkingLevel;
+}
+
+export function reasoningToOptions(level: VisionReasoning | undefined): ReasoningOptions {
+	if (!level || level === "off") return {};
+	return { reasoningEffort: level };
+}
+
+/**
+ * Resolve the effective reasoning level: explicit param wins, else config default, else "off".
+ */
+export function effectiveReasoning(
+	param: VisionReasoning | undefined,
+	configDefault: VisionReasoning | undefined,
+): VisionReasoning {
+	return param ?? configDefault ?? "off";
+}

package/src/state.ts

@@ -0,0 +1,20 @@
+import type { Model } from "@earendil-works/pi-ai";
+import type { VisionConfig } from "./config.js";
+
+// biome-ignore lint/suspicious/noExplicitAny: generic Model type parameter
+export function callingModelHasVision(model: Model<any> | undefined): boolean {
+	return !!model && Array.isArray(model.input) && model.input.includes("image");
+}
+
+// biome-ignore lint/suspicious/noExplicitAny: generic Model type parameter
+export function effectiveEnabled(config: VisionConfig, model: Model<any> | undefined): boolean {
+	if (config.enabled === "on") return true;
+	if (config.enabled === "off") return false;
+	return !callingModelHasVision(model);
+}
+
+export function footerLabel(config: VisionConfig, enabled: boolean): string | undefined {
+	if (!enabled) return undefined;
+	if (!config.provider || !config.model) return undefined;
+	return `👁 ${config.provider}/${config.model}`;
+}

package/src/vision.ts

@@ -0,0 +1,90 @@
+import type { AssistantMessage, Context, Model, UserMessage } from "@earendil-works/pi-ai";
+import type { VisionConfig } from "./config.js";
+import type { DecodedImage } from "./image.js";
+import type { ReasoningOptions } from "./reasoning.js";
+
+export type CompleteFn = (
+	model: Model<any>,
+	context: Context,
+	options?: Record<string, unknown>,
+) => Promise<AssistantMessage>;
+
+export interface VisionCallInput {
+	model: Model<any>;
+	auth: { apiKey?: string; headers?: Record<string, string> };
+	prompt: string;
+	images: DecodedImage[];
+	reasoning: ReasoningOptions;
+	signal?: AbortSignal;
+}
+
+export interface VisionCallResult {
+	text: string;
+	usage?: { input?: number; output?: number };
+	errorMessage?: string;
+	stopReason?: string;
+}
+
+export type ResolveResult = { ok: true; model: Model<any> } | { ok: false; error: string };
+
+interface ModelLookup {
+	find(provider: string, id: string): Model<any> | undefined;
+}
+
+export function resolveVisionModel(registry: ModelLookup, config: VisionConfig): ResolveResult {
+	if (!config.provider || !config.model) {
+		return {
+			ok: false,
+			error: "Vision model not configured. Run: /vision config provider <p> ; /vision config model <m>",
+		};
+	}
+	const model = registry.find(config.provider, config.model);
+	if (!model) {
+		return { ok: false, error: `Vision model not found: ${config.provider}/${config.model}` };
+	}
+	if (!Array.isArray(model.input) || !model.input.includes("image")) {
+		return { ok: false, error: `Model ${config.provider}/${config.model} does not support image input` };
+	}
+	return { ok: true, model };
+}
+
+export async function callVision(input: VisionCallInput, completeFn: CompleteFn): Promise<VisionCallResult> {
+	const userMessage: UserMessage = {
+		role: "user",
+		content: [
+			{ type: "text", text: input.prompt },
+			...input.images.map((img) => ({
+				type: "image" as const,
+				data: img.data.toString("base64"),
+				mimeType: img.mimeType,
+			})),
+		],
+		timestamp: Date.now(),
+	};
+
+	const context: Context = { messages: [userMessage] };
+
+	const options: Record<string, unknown> = {
+		apiKey: input.auth.apiKey,
+		headers: input.auth.headers,
+		...input.reasoning,
+	};
+	if (input.signal) options.signal = input.signal;
+
+	try {
+		const res = await completeFn(input.model, context, options);
+		const text = res.content
+			.filter((c): c is { type: "text"; text: string } => c.type === "text")
+			.map((c) => c.text)
+			.join("\n");
+		return {
+			text,
+			usage: { input: res.usage?.input, output: res.usage?.output },
+			stopReason: res.stopReason,
+			errorMessage: res.errorMessage,
+		};
+	} catch (e) {
+		const msg = e instanceof Error ? e.message : String(e);
+		return { text: "", errorMessage: msg };
+	}
+}