pi-vision-handoff 0.2.0 → 0.2.1

package/README.md

@@ -26,8 +26,8 @@ The `pi-umans-provider` extension quietly solved this for GLM 5.1: a hardcoded "
 
 - **Pick any vision-capable model** from your registry via an interactive picker — OpenAI, Anthropic, Google, Ollama, or any custom provider pi knows about.
 - Your choice **persists** to `~/.pi/agent/extensions/pi-vision-handoff.json`.
-- For any model that doesn't declare image input (or any model you explicitly target), `pi-vision-handoff` describes the image with your chosen vision model **before** the request is sent, then **swaps the image block for the description** in the actual provider payload.
-- Works across all three request formats pi uses — OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages — detected by block shape.
+- For any model that doesn't declare image input (or any model you explicitly target), `pi-vision-handoff` describes the image with your chosen vision model and **inserts the description before the kept image** in the stored `read`-tool result — **before** pi-ai can strip the image for non-vision models. The image stays for kitty rendering; pi-ai later strips only the image block, leaving the description text for the model.
+- Works across all four image-block shapes pi uses — the three provider-transformed formats (OpenAI Chat Completions, OpenAI Responses, Anthropic Messages) plus pi-ai's internal `{ type: "image", data, mimeType }` emitted by the `read` tool — detected by shape.
 - Descriptions are **cached per image hash** (LRU), so the swap is instant by the time the request fires.
 
 No `settings.json` touched. No per-provider glue. Pick a describer once and every text-only model you own can suddenly see.
@@ -35,12 +35,13 @@ No `settings.json` touched. No per-provider glue. Pick a describer once and ever
 ## Features
 
 - **🎮 Interactive picker** — `/vision-handoff` opens a TUI listing every model, vision-capable ones first (👁), to choose your describer.
+- **🖼️ Read-tool hijack** — `read`-tool images are intercepted at `tool_result` time, before pi-ai can strip them for non-vision models. The kept image still renders inline (kitty); the description is inserted as a text block the text-only model consumes.
 - **🔌 Provider-agnostic** — uses pi's own model execution machinery (`@earendil-works/pi-ai`'s `complete()`), so it works with any provider/configured model, including custom provider extensions.
 - **🧠 Automatic targets** — by default, handoff applies to *every* model that lacks native vision. Opt out with `/vision-handoff auto off`.
 - **🗂️ Explicit overrides** — force handoff for specific models (e.g. a weak vision model) with `/vision-handoff add`.
 - **⚡ Cache-warmed** — `before_agent_start` describes attached images the moment you submit, so the request is rarely delayed.
 - **🛡️ Graceful degradation** — no API key? Describer unreachable? Aborted? The image is replaced with a clean `[Image: description unavailable]` placeholder instead of crashing your turn.
-- **🔧 Tunable** — cap description length and cache size in the config file.
+- **🔧 Tunable** — cap description length (`maxDescriptionLines`; unbounded by default, so `read`'s native `ctrl+o` collapse handles compactness) and cache size, in the config file.
 
 ## Usage
 
@@ -71,6 +72,7 @@ Created automatically at `~/.pi/agent/extensions/pi-vision-handoff.json` on firs
   "handoffModels": ["ollama/llava:13b"],
   "maxTokens": 1024,
   "cacheMax": 50,
+  "maxDescriptionLines": 0,
   "prompt": "Describe this image exhaustively…",
   "userPromptPrefix": "The user's request about this image: "
 }
@@ -84,6 +86,7 @@ Created automatically at `~/.pi/agent/extensions/pi-vision-handoff.json` on firs
 | `handoffModels` | `[]` | Extra `provider/id` refs that should also receive handoff. |
 | `maxTokens` | `1024` | Cap on a single description's output. |
 | `cacheMax` | `50` | Max described images kept in the in-memory cache per session. |
+| `maxDescriptionLines` | `0` | Cap on description lines (`0` = unbounded). Default keeps the full description so the `read` tool's native collapse (`ctrl+o`) handles compactness and the model gets complete context; setting `> 0` applies a lossy head-cap to both the TUI render and the model. |
 | `prompt` | _(built-in)_ | Override the describer system prompt. |
 | `userPromptPrefix` | _(built-in)_ | Override the prefix prepended to your original prompt. |
 
@@ -129,31 +132,47 @@ pi -e ./vision-handoff.ts
 
 ## How It Works
 
-```
-You submit a prompt + image, on a text-only model
-  → before_agent_start
-      • is this model a handoff target? (non-vision, or in handoffModels)
-      • for each attached image → describeImage() with your chosen vision model
-          - complete() via pi-ai (resolves API key/headers/baseUrl for you)
-          - cached by sha256(mime + base64)
-      • fire-and-forget — warms the cache
-  → before_provider_request
-      • walk the provider payload's messages[]
-      • for every image block (detected by shape) → swap for a text block:
-          "[Image: <cached description>]"
-      • returns the modified payload to pi
-
-Result: the text-only model receives a vivid text description in place of
-the image, and your turn proceeds normally.
-```
+Read-tool images (the common case — paste a screenshot, `read` an image file):
+
+    read tool returns an image block
+      → tool_result (toolName === "read")
+          • fires BEFORE pi-ai's transformMessages can strip the image for
+            non-vision models (the strip would leave only a "(tool image
+            omitted)" placeholder — too late to describe)
+          • is this model a handoff target? (non-vision, or in handoffModels)
+          • for each {type:"image",...} block → describeImage() with your
+            chosen vision model
+              - complete() via pi-ai (resolves API key/headers/baseUrl)
+              - cached by sha256(mime + base64)
+          • AUGMENTS the stored result: inserts "[Image: <description>]" as a
+            text block BEFORE the kept image, returns {content} to pi
+          • the image stays in content → kitty renders it inline
+          • at provider time, pi-ai strips ONLY the image block for non-vision
+            models — the inserted description text passes through to the model
+
+User-attached images (pasted into the prompt, not read via a tool):
+
+    → before_agent_start
+        • warms the description cache for attached images (fire-and-forget)
+    → before_provider_request
+        • walks the provider payload, swaps image blocks for "[Image: <desc>]"
+        • NOTE: for non-vision models, pi-ai strips image blocks BEFORE this
+          hook fires — so user-attached images on text-only models are not yet
+          described. Use the `read` tool on the image file instead, which routes
+          through the tool_result path above.
+
+Result: the text-only model receives a vivid text description alongside the
+(now-stripped) image reference, and your turn proceeds normally — while the
+terminal still renders the image inline via kitty.
 
 ### Image-block formats handled
 
-| API | Image block shape | Replacement |
-|-----|-------------------|-------------|
-| OpenAI Chat Completions | `{ type: "image_url", image_url: { url: "data:…" } }` | `{ type: "text", text }` |
-| OpenAI Responses | `{ type: "input_image", image_url: "data:…" }` | `{ type: "input_text", text }` |
-| Anthropic Messages | `{ type: "image", source: { type: "base64", media_type, data } }` | `{ type: "text", text }` |
+| Hook | Image block shape | Replacement |
+|------|-------------------|-------------|
+| `tool_result` (read) | `{ type: "image", data, mimeType }` (pi-ai internal) | description text block inserted **before** the kept image |
+| `before_provider_request` | `{ type: "image_url", image_url: { url: "data:…" } }` (OpenAI Chat Completions) | `{ type: "text", text }` |
+| `before_provider_request` | `{ type: "input_image", image_url: "data:…" }` (OpenAI Responses) | `{ type: "input_text", text }` |
+| `before_provider_request` | `{ type: "image", source: { type: "base64", media_type, data } }` (Anthropic Messages) | `{ type: "text", text }` |
 
 The describer call itself goes through pi's normal model machinery (`complete()`), **not** the agent event loop — so it never re-triggers `before_provider_request` (no recursion).
 
@@ -171,7 +190,7 @@ The describer call itself goes through pi's normal model machinery (`complete()`
 
 ```bash
 pnpm install
-pnpm test          # Vitest unit tests (31 passing)
+pnpm test          # Vitest unit tests (50 passing)
 pnpm typecheck     # TypeScript validation
 pnpm lint:dead     # Dead code detection (knip)
 ```
@@ -186,7 +205,7 @@ pnpm lint:dead     # Dead code detection (knip)
 │   └── vision-model-selector.ts  # Interactive picker TUI component
 ├── __tests__/unit/
 │   ├── config-dir.test.ts        # Ensures getAgentDir() usage
-│   └── vision-handoff.test.ts    # Config, refs, image-block extraction, round-trip
+│   └── vision-handoff.test.ts    # Config, refs, image-block extraction, insertion, truncation, round-trip
 ├── package.json
 ├── tsconfig.json
 ├── knip.json

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-vision-handoff",
-  "version": "0.2.0",
+  "version": "0.2.1",
   "description": "Give text-only pi models vision — describe images with a vision model you pick via an interactive picker, then hand off the text description to non-vision models",
   "type": "module",
   "author": "Tom X Nguyen",

package/src/index.ts

@@ -57,6 +57,29 @@ export const DESCRIBE_TIMEOUT_MS = 30_000;
  */
 export const DEFAULT_MAX_DESCRIPTION_LINES = 0;
 
+/**
+ * The note pi core's `read` tool appends to image-result text blocks when the
+ * active model lacks image input (see `getNonVisionImageNote` in pi core).
+ *
+ * Once this extension inserts a description, the note becomes self-
+ * contradicting ("image will be omitted" vs. the vivid description that
+ * follows) and confuses the model. {@link stripNonVisionImageNote} removes it.
+ */
+export const NON_VISION_IMAGE_NOTE =
+  "[Current model does not support images. The image will be omitted from this request.]";
+
+/**
+ * Remove {@link NON_VISION_IMAGE_NOTE} from a text block.
+ *
+ * The read tool appends the note as `\n${NOTE}` (always the trailing segment of
+ * the metadata text block), so this also collapses the orphaned newline it
+ * leaves behind. Safe to call on text that does not contain the note (no-op).
+ */
+export function stripNonVisionImageNote(text: string): string {
+  if (!text.includes(NON_VISION_IMAGE_NOTE)) return text;
+  return text.split(NON_VISION_IMAGE_NOTE).join("").replace(/\n+$/, "");
+}
+
 export interface VisionHandoffConfig {
   /** Master switch. When false, no handoff occurs even if a vision model is configured. */
   enabled: boolean;
@@ -292,6 +315,12 @@ export interface ReplacedContent {
  * provider, registry, and API call. The extension wires its real `describeImage`
  * into this helper in its `tool_result` handler.
  *
+ * When at least one description is inserted, also strips pi core's
+ * {@link NON_VISION_IMAGE_NOTE} from text blocks — the note (appended by the
+ * read tool for non-vision models) would otherwise contradict the inserted
+ * description. Text blocks are reassigned (not mutated in place); the input
+ * array is left untouched.
+ *
  * Returns a new array and `changed: false` when there were no images, so
  * callers can short-circuit and avoid mutating pi's stored result unnecessarily.
  */
@@ -315,5 +344,18 @@ export async function insertImageDescriptions(
     next.push(block);
     changed = true;
   }
+  if (!changed) {
+    return { content: next, changed };
+  }
+  // We inserted at least one description. Strip the read tool's
+  // "[Current model does not support images...]" note from text blocks — it
+  // contradicts the description we just inserted ("image will be omitted" vs.
+  // the description that follows) and confuses the model.
+  for (let i = 0; i < next.length; i++) {
+    const block = next[i];
+    if (block.type === "text" && typeof block.text === "string" && block.text.includes(NON_VISION_IMAGE_NOTE)) {
+      next[i] = { type: "text", text: stripNonVisionImageNote(block.text) } satisfies TextContent;
+    }
+  }
   return { content: next, changed };
 }