@letta-ai/image-understanding 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Letta, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/MOD.md

@@ -0,0 +1,84 @@
+---
+name: "@letta-ai/image-understanding"
+description: "Image-understanding tool, slash commands, and optional auto-captioning for text-only agents."
+---
+
+# Image understanding mod semantics
+
+## When to use
+
+Use this package when the current model cannot inspect images directly but needs to understand a screenshot, diagram, photo, UI error, OCR text, or accessibility context.
+
+The package does not make the main model natively multimodal. It sends the image to a separate configured vision backend and returns text.
+
+## Capabilities
+
+This package registers:
+
+- Tool: `image_understand`
+- Commands:
+  - `/image-understand`
+  - `/image-understanding-status`
+- Optional `turn_start` auto-captioning when `IMAGE_UNDERSTANDING_AUTO_CAPTION=1`
+
+## Providers
+
+Supported providers:
+
+- `openai-compatible` (default), using chat completions with image input
+- `ollama`, using `/api/generate` with `images`
+
+Configuration is controlled by environment variables, especially:
+
+- `IMAGE_UNDERSTANDING_PROVIDER`
+- `IMAGE_UNDERSTANDING_API_KEY` or `OPENAI_API_KEY`
+- `IMAGE_UNDERSTANDING_MODEL`
+- `IMAGE_UNDERSTANDING_BASE_URL`
+- `IMAGE_UNDERSTANDING_ALLOW_CLOUD`
+- `IMAGE_UNDERSTANDING_ALLOW_URLS`
+- `IMAGE_UNDERSTANDING_REQUIRE_LOCAL`
+- `IMAGE_UNDERSTANDING_AUTO_CAPTION`
+- `IMAGE_UNDERSTANDING_AUTO_MODE`
+
+## Tool behavior
+
+`image_understand` supports:
+
+- `action: "status"` to inspect provider configuration
+- `path_or_url` for local image paths, workspace-relative paths, `~/` paths, or HTTP(S) image URLs
+- `question` for targeted image questions
+- `mode` for built-in prompts: `describe`, `ocr`, `ui_debug`, `diagram`, `accessibility`
+- `detail` for OpenAI-compatible detail preference (`low`, `high`, `auto`)
+
+Agent-initiated tool calls require approval because image bytes may be read from local disk or sent to a provider.
+
+## Auto-captioning behavior
+
+Auto-captioning is opt-in. When enabled, the mod listens for `turn_start`, finds image content parts or markdown image links in user messages, runs image understanding, and appends text descriptions to the user turn before the main model sees it.
+
+For sensitive images, prefer:
+
+```bash
+IMAGE_UNDERSTANDING_PROVIDER=ollama
+IMAGE_UNDERSTANDING_ALLOW_CLOUD=0
+IMAGE_UNDERSTANDING_REQUIRE_LOCAL=1
+IMAGE_UNDERSTANDING_AUTO_CAPTION=1
+```
+
+## Safety
+
+- Slash commands are direct user actions and run immediately.
+- Auto-captioning runs automatically once enabled.
+- `IMAGE_UNDERSTANDING_ALLOW_CLOUD=0` blocks non-local providers.
+- `IMAGE_UNDERSTANDING_ALLOW_URLS=0` blocks remote image fetching.
+- `IMAGE_UNDERSTANDING_REQUIRE_LOCAL=1` requires the Ollama provider.
+- Local and fetched images over 20 MB are rejected.
+- Supported local extensions: `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`.
+
+## Adaptation notes for agents
+
+- Use `/image-understanding-status` or tool `action: "status"` before debugging provider setup.
+- Prefer `mode: "ui_debug"` for screenshots and error states.
+- Prefer `mode: "ocr"` when the goal is text extraction.
+- Prefer `mode: "diagram"` for architecture or technical diagrams.
+- Do not enable auto-captioning casually; it is intentionally opt-in.

package/README.md

@@ -0,0 +1,204 @@
+# Image Understanding
+
+A Letta Code mod package that gives text-only/non-vision agents image understanding by routing images through a separate vision backend and returning text the main model can reason over.
+
+This does not make the main model natively multimodal. It adds a trusted bridge:
+
+1. The user or agent provides an image path or URL.
+2. The mod sends that image to a configured vision backend.
+3. The backend returns a text description, OCR extraction, UI-debug analysis, diagram explanation, or accessibility description.
+4. The text-only agent uses that returned text like any other context.
+
+Original source: <https://tangled.org/cameron.stream/image-understanding>
+
+## Install
+
+```bash
+letta install npm:@letta-ai/image-understanding
+```
+
+Run `/reload` in active sessions after installing.
+
+## Features
+
+- Agent tool: `image_understand`
+- Slash commands:
+  - `/image-understand`
+  - `/image-understanding-status`
+- Optional `turn_start` auto-captioning for image-bearing user turns
+- Prompt modes:
+  - `describe`
+  - `ocr`
+  - `ui_debug`
+  - `diagram`
+  - `accessibility`
+
+## Quick start
+
+### OpenAI-compatible provider
+
+```bash
+export OPENAI_API_KEY=...
+# optional
+export IMAGE_UNDERSTANDING_PROVIDER=openai-compatible
+export IMAGE_UNDERSTANDING_MODEL=gpt-4o-mini
+```
+
+Then reload and test:
+
+```text
+/reload
+/image-understanding-status
+/image-understand ~/Desktop/screenshot.png what error is shown?
+```
+
+### Local Ollama provider
+
+Use this when you want image bytes to stay local.
+
+```bash
+ollama pull llava:latest
+export IMAGE_UNDERSTANDING_PROVIDER=ollama
+export IMAGE_UNDERSTANDING_MODEL=llava:latest
+export IMAGE_UNDERSTANDING_BASE_URL=http://localhost:11434
+export IMAGE_UNDERSTANDING_ALLOW_CLOUD=0
+```
+
+Then reload and test:
+
+```text
+/reload
+/image-understanding-status
+/image-understand ~/Desktop/screenshot.png summarize this screenshot
+```
+
+Any Ollama model that supports image input should work. Other possible models include `llama3.2-vision` or Qwen/VL variants if they are available in your Ollama installation.
+
+## Configuration
+
+| Variable | Default | Description |
+| --- | --- | --- |
+| `IMAGE_UNDERSTANDING_PROVIDER` | `openai-compatible` | Provider backend. Supported: `openai-compatible`, `ollama`. Alias: `openai`. |
+| `IMAGE_UNDERSTANDING_API_KEY` | unset | API key for OpenAI-compatible backends. Overrides `OPENAI_API_KEY`. |
+| `OPENAI_API_KEY` | unset | Fallback API key for OpenAI-compatible backends. |
+| `IMAGE_UNDERSTANDING_MODEL` | `gpt-4o-mini` or `llava:latest` | Vision model name. Default depends on provider. |
+| `IMAGE_UNDERSTANDING_BASE_URL` | OpenAI or Ollama URL | Base URL. OpenAI-compatible default: `https://api.openai.com/v1`; Ollama default: `http://localhost:11434`. |
+| `IMAGE_UNDERSTANDING_MAX_TOKENS` | `1200` | Max tokens for OpenAI-compatible responses. |
+| `IMAGE_UNDERSTANDING_ALLOW_CLOUD` | `1` | Set `0` to block non-local providers. |
+| `IMAGE_UNDERSTANDING_ALLOW_URLS` | `1` | Set `0` to block fetching remote image URLs. |
+| `IMAGE_UNDERSTANDING_REQUIRE_LOCAL` | `0` | Set `1` to require local provider use. Currently this requires `provider=ollama`. |
+| `IMAGE_UNDERSTANDING_AUTO_CAPTION` | `0` | Set `1` to enable automatic image caption injection on `turn_start`. |
+| `IMAGE_UNDERSTANDING_AUTO_MODE` | `describe` | Mode used for auto-captioning. Supports `describe`, `ocr`, `ui_debug`, `diagram`, `accessibility`. |
+
+## Tool usage
+
+Status check:
+
+```json
+{ "action": "status" }
+```
+
+General image description:
+
+```json
+{
+  "path_or_url": "~/Desktop/screenshot.png"
+}
+```
+
+Targeted question:
+
+```json
+{
+  "path_or_url": "~/Desktop/screenshot.png",
+  "question": "What error is shown in this screenshot?"
+}
+```
+
+Use a built-in mode:
+
+```json
+{
+  "path_or_url": "~/Desktop/screenshot.png",
+  "mode": "ui_debug"
+}
+```
+
+## Slash commands
+
+```text
+/image-understanding-status
+/image-understand ~/Desktop/screenshot.png what error is shown?
+```
+
+Use quotes for paths containing spaces:
+
+```text
+/image-understand "~/Desktop/error screenshot.png" summarize the UI state
+```
+
+## Auto-captioning
+
+Auto-captioning is off by default. Enable it only when you want the mod to automatically inspect images before the main model sees the user turn.
+
+```bash
+export IMAGE_UNDERSTANDING_AUTO_CAPTION=1
+export IMAGE_UNDERSTANDING_AUTO_MODE=ui_debug
+```
+
+For private local-only auto-captioning:
+
+```bash
+export IMAGE_UNDERSTANDING_PROVIDER=ollama
+export IMAGE_UNDERSTANDING_ALLOW_CLOUD=0
+export IMAGE_UNDERSTANDING_REQUIRE_LOCAL=1
+export IMAGE_UNDERSTANDING_AUTO_CAPTION=1
+```
+
+## Privacy and security
+
+This is trusted local code. It can read local image files you ask it to process and can send image bytes to the configured backend.
+
+Important behavior:
+
+- Agent-initiated `image_understand` tool calls require approval.
+- Slash commands are direct user actions and run immediately.
+- Auto-captioning is opt-in and runs automatically once enabled.
+- `IMAGE_UNDERSTANDING_ALLOW_CLOUD=0` blocks non-local providers.
+- `IMAGE_UNDERSTANDING_ALLOW_URLS=0` blocks fetching remote image URLs.
+- `IMAGE_UNDERSTANDING_REQUIRE_LOCAL=1` requires local provider use.
+
+For sensitive screenshots, prefer Ollama/local provider mode.
+
+## Supported inputs
+
+Local file extensions:
+
+- `.png`
+- `.jpg`
+- `.jpeg`
+- `.webp`
+- `.gif`
+
+Remote inputs:
+
+- `http://...`
+- `https://...`
+
+Limits:
+
+- Local and fetched images over 20 MB are rejected.
+- URL responses must have an `image/*` content type.
+- Unsupported local extensions are rejected unless the image is provided via HTTP(S).
+
+## Safety
+
+If a mod breaks startup or command handling, recover with:
+
+```bash
+letta --no-mods
+# or
+LETTA_DISABLE_MODS=1 letta
+```
+
+See [`MOD.md`](./MOD.md) for the agent-facing behavioral contract.

package/mods/index.mjs

@@ -0,0 +1,452 @@
+import { readFile } from "node:fs/promises";
+import { statSync } from "node:fs";
+import path from "node:path";
+
+const DEFAULT_PROVIDER = "openai-compatible";
+const DEFAULT_OPENAI_MODEL = "gpt-4o-mini";
+const DEFAULT_OPENAI_BASE_URL = "https://api.openai.com/v1";
+const DEFAULT_OLLAMA_MODEL = "llava:latest";
+const DEFAULT_OLLAMA_BASE_URL = "http://localhost:11434";
+const MAX_IMAGE_BYTES = 20 * 1024 * 1024;
+
+const MIME_BY_EXT = new Map([
+  [".png", "image/png"],
+  [".jpg", "image/jpeg"],
+  [".jpeg", "image/jpeg"],
+  [".webp", "image/webp"],
+  [".gif", "image/gif"],
+]);
+
+const MODE_PROMPTS = {
+  describe:
+    "Describe this image clearly. Include important visual details, context, visible objects, people, UI state, errors, diagrams, and any visible text.",
+  ocr:
+    "Extract all visible text from this image. Preserve line breaks and reading order where possible. If text is unclear, mark uncertain words with [?].",
+  ui_debug:
+    "Analyze this screenshot for UI/debugging. Extract visible text, identify errors or broken states, describe relevant controls/layout, and suggest likely next debugging steps.",
+  diagram:
+    "Explain this diagram or technical visual. Identify nodes, arrows, labels, structure, relationships, and the main takeaway.",
+  accessibility:
+    "Write an accessibility-focused description of this image for someone who cannot see it. Include layout, salient visual details, and visible text.",
+};
+
+function envBool(name, defaultValue) {
+  const raw = process.env[name];
+  if (raw === undefined || raw === "") return defaultValue;
+  return !["0", "false", "no", "off"].includes(String(raw).toLowerCase());
+}
+
+function normalizeProvider(value) {
+  const provider = String(value || DEFAULT_PROVIDER).toLowerCase();
+  if (["openai", "openai-compatible", "openai_compatible"].includes(provider)) return "openai-compatible";
+  if (provider === "ollama") return "ollama";
+  return provider;
+}
+
+function getConfig() {
+  const provider = normalizeProvider(process.env.IMAGE_UNDERSTANDING_PROVIDER);
+  const openaiApiKey = process.env.IMAGE_UNDERSTANDING_API_KEY || process.env.OPENAI_API_KEY || "";
+  return {
+    provider,
+    apiKey: openaiApiKey,
+    model:
+      process.env.IMAGE_UNDERSTANDING_MODEL ||
+      (provider === "ollama" ? DEFAULT_OLLAMA_MODEL : DEFAULT_OPENAI_MODEL),
+    baseUrl: (
+      process.env.IMAGE_UNDERSTANDING_BASE_URL ||
+      (provider === "ollama" ? DEFAULT_OLLAMA_BASE_URL : DEFAULT_OPENAI_BASE_URL)
+    ).replace(/\/$/, ""),
+    maxTokens: Number.parseInt(process.env.IMAGE_UNDERSTANDING_MAX_TOKENS || "1200", 10),
+    allowCloud: envBool("IMAGE_UNDERSTANDING_ALLOW_CLOUD", true),
+    allowUrls: envBool("IMAGE_UNDERSTANDING_ALLOW_URLS", true),
+    requireLocal: envBool("IMAGE_UNDERSTANDING_REQUIRE_LOCAL", false),
+    autoCaption: envBool("IMAGE_UNDERSTANDING_AUTO_CAPTION", false),
+    autoMode: process.env.IMAGE_UNDERSTANDING_AUTO_MODE || "describe",
+  };
+}
+
+function isHttpUrl(value) {
+  try {
+    const url = new URL(value);
+    return url.protocol === "http:" || url.protocol === "https:";
+  } catch {
+    return false;
+  }
+}
+
+function resolveLocalPath(input, cwd) {
+  const expanded = input.startsWith("~/")
+    ? path.join(process.env.HOME || "", input.slice(2))
+    : input;
+  return path.isAbsolute(expanded) ? expanded : path.resolve(cwd || process.cwd(), expanded);
+}
+
+function promptFor({ question, mode }) {
+  const trimmed = typeof question === "string" ? question.trim() : "";
+  if (trimmed) return trimmed;
+  return MODE_PROMPTS[mode] || MODE_PROMPTS.describe;
+}
+
+async function localImage(pathOrUrl, cwd) {
+  const resolved = resolveLocalPath(pathOrUrl, cwd);
+  const stat = statSync(resolved);
+  if (!stat.isFile()) throw new Error(`Not a file: ${resolved}`);
+  if (stat.size > MAX_IMAGE_BYTES) {
+    throw new Error(`Image is too large (${stat.size} bytes). Limit is ${MAX_IMAGE_BYTES} bytes.`);
+  }
+
+  const ext = path.extname(resolved).toLowerCase();
+  const mime = MIME_BY_EXT.get(ext);
+  if (!mime) {
+    throw new Error(`Unsupported image extension "${ext}". Use png, jpg, jpeg, webp, gif, or an http(s) URL.`);
+  }
+
+  const data = await readFile(resolved);
+  return { bytes: data, base64: data.toString("base64"), mime, source: resolved, isUrl: false };
+}
+
+async function urlImage(pathOrUrl, signal) {
+  const response = await fetch(pathOrUrl, { signal });
+  if (!response.ok) {
+    throw new Error(`Failed to fetch image URL: HTTP ${response.status} ${response.statusText}`);
+  }
+  const contentLength = Number.parseInt(response.headers.get("content-length") || "0", 10);
+  if (contentLength > MAX_IMAGE_BYTES) {
+    throw new Error(`Image URL is too large (${contentLength} bytes). Limit is ${MAX_IMAGE_BYTES} bytes.`);
+  }
+  const arrayBuffer = await response.arrayBuffer();
+  if (arrayBuffer.byteLength > MAX_IMAGE_BYTES) {
+    throw new Error(`Image URL is too large (${arrayBuffer.byteLength} bytes). Limit is ${MAX_IMAGE_BYTES} bytes.`);
+  }
+  const contentType = response.headers.get("content-type")?.split(";")[0]?.trim() || "image/png";
+  if (!contentType.startsWith("image/")) {
+    throw new Error(`URL did not return an image content-type: ${contentType}`);
+  }
+  const bytes = Buffer.from(arrayBuffer);
+  return { bytes, base64: bytes.toString("base64"), mime: contentType, source: pathOrUrl, isUrl: true };
+}
+
+async function loadImage(pathOrUrl, ctx) {
+  return isHttpUrl(pathOrUrl) ? await urlImage(pathOrUrl, ctx.signal) : await localImage(pathOrUrl, ctx.cwd);
+}
+
+function enforceSafety({ pathOrUrl, config }) {
+  if (config.requireLocal && config.provider !== "ollama") {
+    return "IMAGE_UNDERSTANDING_REQUIRE_LOCAL=1 requires IMAGE_UNDERSTANDING_PROVIDER=ollama.";
+  }
+  if (!config.allowCloud && config.provider !== "ollama") {
+    return "IMAGE_UNDERSTANDING_ALLOW_CLOUD=0 blocks non-local vision providers. Set IMAGE_UNDERSTANDING_PROVIDER=ollama or allow cloud use.";
+  }
+  if (!config.allowUrls && isHttpUrl(pathOrUrl)) {
+    return "IMAGE_UNDERSTANDING_ALLOW_URLS=0 blocks fetching image URLs. Use a local file path or allow URLs.";
+  }
+  return null;
+}
+
+function parseJson(text) {
+  try {
+    return text ? JSON.parse(text) : null;
+  } catch {
+    return text;
+  }
+}
+
+function errorContent(message, config) {
+  const setup = config.provider === "ollama"
+    ? `Ollama setup example: IMAGE_UNDERSTANDING_PROVIDER=ollama IMAGE_UNDERSTANDING_MODEL=${DEFAULT_OLLAMA_MODEL} IMAGE_UNDERSTANDING_BASE_URL=${DEFAULT_OLLAMA_BASE_URL}`
+    : "OpenAI-compatible setup example: set OPENAI_API_KEY or IMAGE_UNDERSTANDING_API_KEY, optionally IMAGE_UNDERSTANDING_MODEL and IMAGE_UNDERSTANDING_BASE_URL.";
+  return `${message}\n${setup}`;
+}
+
+async function askOpenAiCompatible({ image, prompt, detail }, config, ctx) {
+  if (!config.apiKey) {
+    return { status: "error", content: errorContent("Missing API key for openai-compatible image understanding.", config) };
+  }
+
+  const imageUrl = { url: `data:${image.mime};base64,${image.base64}` };
+  if (detail) imageUrl.detail = detail;
+
+  const response = await fetch(`${config.baseUrl}/chat/completions`, {
+    method: "POST",
+    headers: {
+      Authorization: `Bearer ${config.apiKey}`,
+      "Content-Type": "application/json",
+    },
+    body: JSON.stringify({
+      model: config.model,
+      messages: [
+        {
+          role: "user",
+          content: [
+            { type: "text", text: prompt },
+            { type: "image_url", image_url: imageUrl },
+          ],
+        },
+      ],
+      max_tokens: Number.isFinite(config.maxTokens) ? config.maxTokens : 1200,
+    }),
+    signal: ctx.signal,
+  });
+
+  const body = parseJson(await response.text());
+  if (!response.ok) {
+    const rendered = typeof body === "string" ? body : JSON.stringify(body);
+    const hint = response.status === 400
+      ? " The configured endpoint/model may not support image input. Use a vision-capable model or provider=ollama with a vision model."
+      : "";
+    return { status: "error", content: errorContent(`Vision request failed: HTTP ${response.status} ${response.statusText}: ${rendered}.${hint}`, config) };
+  }
+
+  const answer = body?.choices?.[0]?.message?.content;
+  if (!answer) return { status: "error", content: `Vision response had no answer: ${JSON.stringify(body)}` };
+  return answer;
+}
+
+async function askOllama({ image, prompt }, config, ctx) {
+  const response = await fetch(`${config.baseUrl}/api/generate`, {
+    method: "POST",
+    headers: { "Content-Type": "application/json" },
+    body: JSON.stringify({
+      model: config.model,
+      prompt,
+      images: [image.base64],
+      stream: false,
+    }),
+    signal: ctx.signal,
+  });
+
+  const body = parseJson(await response.text());
+  if (!response.ok) {
+    const rendered = typeof body === "string" ? body : JSON.stringify(body);
+    const hint = response.status === 404
+      ? ` Is Ollama running and is the model pulled? Try: ollama pull ${config.model}`
+      : "";
+    return { status: "error", content: errorContent(`Ollama vision request failed: HTTP ${response.status} ${response.statusText}: ${rendered}.${hint}`, config) };
+  }
+
+  const answer = body?.response;
+  if (!answer) return { status: "error", content: `Ollama response had no answer: ${JSON.stringify(body)}` };
+  return answer;
+}
+
+async function askVision({ pathOrUrl, question, detail, mode }, ctx) {
+  const config = getConfig();
+  if (!["openai-compatible", "ollama"].includes(config.provider)) {
+    return { status: "error", content: `Unsupported IMAGE_UNDERSTANDING_PROVIDER=${config.provider}. Supported providers: openai-compatible, ollama.` };
+  }
+
+  const safetyError = enforceSafety({ pathOrUrl, config });
+  if (safetyError) return { status: "error", content: safetyError };
+
+  const image = await loadImage(pathOrUrl, ctx);
+  const prompt = promptFor({ question, mode });
+
+  if (config.provider === "ollama") {
+    return await askOllama({ image, prompt }, config, ctx);
+  }
+  return await askOpenAiCompatible({ image, prompt, detail }, config, ctx);
+}
+
+async function providerStatus() {
+  const config = getConfig();
+  const lines = [
+    `provider: ${config.provider}`,
+    `model: ${config.model}`,
+    `base_url: ${config.baseUrl}`,
+    `api_key: ${config.provider === "ollama" ? "not required" : (config.apiKey ? "present" : "missing")}`,
+    `allow_cloud: ${config.allowCloud ? "yes" : "no"}`,
+    `allow_urls: ${config.allowUrls ? "yes" : "no"}`,
+    `require_local: ${config.requireLocal ? "yes" : "no"}`,
+    `auto_caption: ${config.autoCaption ? "yes" : "no"}`,
+    `auto_mode: ${config.autoMode}`,
+    `supported_providers: openai-compatible, ollama`,
+    `modes: ${Object.keys(MODE_PROMPTS).join(", ")}`,
+  ];
+
+  if (config.provider === "ollama") {
+    try {
+      const response = await fetch(`${config.baseUrl}/api/tags`, { signal: AbortSignal.timeout(3_000) });
+      lines.push(`ollama_reachable: ${response.ok ? "yes" : `no (${response.status})`}`);
+      if (response.ok) {
+        const body = await response.json();
+        const models = Array.isArray(body.models) ? body.models.map((m) => m.name).slice(0, 20) : [];
+        lines.push(`ollama_models: ${models.length ? models.join(", ") : "none reported"}`);
+      }
+    } catch (error) {
+      lines.push(`ollama_reachable: no (${error instanceof Error ? error.message : String(error)})`);
+    }
+  }
+
+  return lines.join("\n");
+}
+
+function imageRefFromPart(part) {
+  if (!part || typeof part !== "object") return null;
+  if (typeof part.path === "string" && part.path) return part.path;
+  if (typeof part.file_path === "string" && part.file_path) return part.file_path;
+  if (typeof part.url === "string" && isHttpUrl(part.url)) return part.url;
+  if (typeof part.image_url === "string" && isHttpUrl(part.image_url)) return part.image_url;
+  if (part.image_url && typeof part.image_url === "object" && typeof part.image_url.url === "string") return part.image_url.url;
+  if (part.source && typeof part.source === "object") {
+    if (typeof part.source.path === "string") return part.source.path;
+    if (typeof part.source.url === "string") return part.source.url;
+  }
+  return null;
+}
+
+function extractImageRefsFromContent(content) {
+  const refs = [];
+  if (typeof content === "string") {
+    const markdownImage = /!\[[^\]]*\]\(([^)]+)\)/g;
+    for (const match of content.matchAll(markdownImage)) {
+      const ref = match[1]?.trim();
+      if (ref) refs.push(ref);
+    }
+    return refs;
+  }
+  if (!Array.isArray(content)) return refs;
+  for (const part of content) {
+    const type = String(part?.type || "").toLowerCase();
+    const looksImage = type.includes("image") || Boolean(part?.image_url);
+    if (!looksImage) continue;
+    const ref = imageRefFromPart(part);
+    if (ref) refs.push(ref);
+  }
+  return refs;
+}
+
+function appendTextContent(content, text) {
+  if (typeof content === "string") return `${content}\n\n${text}`;
+  if (Array.isArray(content)) return [...content, { type: "text", text }];
+  return text;
+}
+
+async function autoCaptionTurn(event, ctx) {
+  const config = getConfig();
+  if (!config.autoCaption) return;
+  const input = Array.isArray(event.input) ? event.input : [];
+  const descriptions = [];
+
+  for (const item of input) {
+    if (!item || item.type === "approval" || item.role !== "user") continue;
+    const refs = extractImageRefsFromContent(item.content);
+    for (const ref of refs.slice(0, 4)) {
+      try {
+        const result = await askVision({ pathOrUrl: ref, mode: config.autoMode }, ctx);
+        const text = typeof result === "string" ? result : result.content || JSON.stringify(result);
+        descriptions.push(`Image ${descriptions.length + 1} (${ref}):\n${text}`);
+      } catch (error) {
+        descriptions.push(`Image ${descriptions.length + 1} (${ref}): auto-caption failed: ${error instanceof Error ? error.message : String(error)}`);
+      }
+    }
+  }
+
+  if (descriptions.length === 0) return;
+  const note = `[Auto image understanding (${config.provider}/${config.model})]\n${descriptions.join("\n\n")}`;
+  event.input = input.map((item) => {
+    if (!item || item.type === "approval" || item.role !== "user") return item;
+    const refs = extractImageRefsFromContent(item.content);
+    if (refs.length === 0) return item;
+    return { ...item, content: appendTextContent(item.content, note) };
+  });
+  return { input: event.input };
+}
+
+function parseCommandArgs(args) {
+  const raw = String(args || "").trim();
+  if (!raw) return { pathOrUrl: "", question: "" };
+  if (raw.startsWith('"') || raw.startsWith("'")) {
+    const quote = raw[0];
+    const end = raw.indexOf(quote, 1);
+    if (end > 0) {
+      return { pathOrUrl: raw.slice(1, end), question: raw.slice(end + 1).trim() };
+    }
+  }
+  const [pathOrUrl, ...rest] = raw.split(/\s+/);
+  return { pathOrUrl, question: rest.join(" ").trim() };
+}
+
+export default function activate(letta) {
+  const disposers = [];
+
+  if (letta.capabilities.tools) {
+    disposers.push(letta.tools.register({
+      name: "image_understand",
+      description: "Use a vision model to answer questions about a local image path or image URL when the current model cannot inspect images directly. Supports OpenAI-compatible vision and local Ollama vision backends. Useful for screenshots, UI errors, diagrams, photos, OCR, and visual debugging.",
+      parameters: {
+        type: "object",
+        properties: {
+          action: {
+            type: "string",
+            enum: ["understand", "status"],
+            description: "Use status to inspect provider configuration. Defaults to understand.",
+          },
+          path_or_url: {
+            type: "string",
+            description: "Local image path, path relative to the current workspace, ~/ path, or http(s) image URL. Required for action=understand.",
+          },
+          question: {
+            type: "string",
+            description: "Optional specific question to ask about the image. If omitted, the selected mode prompt is used.",
+          },
+          mode: {
+            type: "string",
+            enum: ["describe", "ocr", "ui_debug", "diagram", "accessibility"],
+            description: "Prompt mode to use when question is omitted. Defaults to describe.",
+          },
+          detail: {
+            type: "string",
+            enum: ["low", "high", "auto"],
+            description: "Optional OpenAI image detail preference. Ignored by Ollama.",
+          },
+        },
+        additionalProperties: false,
+      },
+      requiresApproval: true,
+      parallelSafe: true,
+      async run(ctx) {
+        const action = String(ctx.args.action || "understand");
+        if (action === "status") return await providerStatus();
+        const pathOrUrl = String(ctx.args.path_or_url || "").trim();
+        if (!pathOrUrl) return { status: "error", content: "path_or_url is required for image understanding. Use action=status to inspect configuration." };
+        const question = typeof ctx.args.question === "string" ? ctx.args.question : "";
+        const detail = typeof ctx.args.detail === "string" ? ctx.args.detail : undefined;
+        const mode = typeof ctx.args.mode === "string" ? ctx.args.mode : "describe";
+        return await askVision({ pathOrUrl, question, detail, mode }, ctx);
+      },
+    }));
+  }
+
+  if (letta.capabilities.events?.turns) {
+    disposers.push(letta.events.on("turn_start", autoCaptionTurn));
+  }
+
+  if (letta.capabilities.commands) {
+    disposers.push(letta.commands.register({
+      id: "image-understanding-status",
+      description: "Show image-understanding provider configuration and diagnostics",
+      async run() {
+        return { type: "output", output: await providerStatus() };
+      },
+    }));
+
+    disposers.push(letta.commands.register({
+      id: "image-understand",
+      description: "Inspect an image with the configured vision backend",
+      args: "<path-or-url> [question]",
+      async run(ctx) {
+        const { pathOrUrl, question } = parseCommandArgs(ctx.args);
+        if (!pathOrUrl) {
+          return { type: "output", output: "Usage: /image-understand <path-or-url> [question]" };
+        }
+        const result = await askVision({ pathOrUrl, question, mode: "describe" }, ctx);
+        const output = typeof result === "string" ? result : result.content || JSON.stringify(result, null, 2);
+        return { type: "output", output };
+      },
+    }));
+  }
+
+  return () => {
+    for (const dispose of disposers.reverse()) dispose();
+  };
+}

package/package.json

@@ -0,0 +1,41 @@
+{
+  "name": "@letta-ai/image-understanding",
+  "version": "0.1.0",
+  "description": "Letta Code mod package that adds image understanding for text-only agents using a separate vision backend.",
+  "author": "just-cameron",
+  "license": "MIT",
+  "type": "module",
+  "keywords": [
+    "letta-package",
+    "letta-mod",
+    "letta-code",
+    "image-understanding",
+    "vision",
+    "ollama"
+  ],
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/letta-ai/mods.git",
+    "directory": "packages/image-understanding"
+  },
+  "files": [
+    "README.md",
+    "MOD.md",
+    "LICENSE",
+    "mods"
+  ],
+  "letta": {
+    "manifestVersion": 1,
+    "mods": [
+      "./mods/index.mjs"
+    ],
+    "capabilities": [
+      "tools",
+      "commands",
+      "events.turns"
+    ],
+    "engines": {
+      "lettaCodeCli": ">=0.27.14"
+    }
+  }
+}