pi-multimodal-proxy 1.20.0 → 1.22.0

package/CHANGELOG.md

@@ -4,6 +4,29 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
+## [1.22.0] - 2026-10-10
+
+### Added
+
+- **Claude Haiku 5.5 in the implicit-model candidate chain** — `DEFAULT_MODEL_FALLBACKS` now reads `zai/glm-5.3-flash` → `deepseek/deepseek-v4-flash-vision-exp` → `anthropic/claude-haiku-5-5` (Pi ≥ 1.1.0 catalog) → `anthropic/claude-sonnet-5` → `anthropic/claude-sonnet-4-5`. An anthropic-keyed setup whose cheaper candidates are unkeyed lands on Haiku 5.5 ($0.1/M in, $0.5/M out, vision-capable, 1M context; above 100K input tokens $0.5/$2.5 — output still 4× cheaper than Sonnet's flat rate) instead of Sonnet 5 ($2/M in, $10/M out) — 20× cheaper output at typical description-prompt sizes, same API key. Description prompts (a few images + instructions) sit far below the tier threshold. Haiku's catalog `inputLimits.images.resize` (2000 px / q80) feeds the existing model-aware upload downscale. Pre-1.1.0 catalogs simply don't contain Haiku, so the registry check skips it and behavior is unchanged; explicit model choices are never rewritten.
+- **`durationMs` fallback in the `analyze_image` result row (Pi ≥ 1.1.0, feature-detected)** — the pi 1.1.0 tool render context carries `durationMs`, the monotonic-clock `execute()` time recorded with the final result. `formatAnalyzeImageResult` accepts it as a display fallback when `details` carry no `latencyMs`, so rows replayed from foreign sessions, HTML exports, and results recorded before `details` existed still render `✓ … · 1.2s`. `details.latencyMs` (the vision call itself, also reported to codemode scripts via `structuredContent`) always wins — the host duration covers consent checks and crop encoding too, so the two measure different things. Older hosts pass no `durationMs` and rendering is unchanged.
+- **Per-invocation tool toggling documented** — pi 1.1.0's `--tools +name/-name` entries change the default tool selection instead of replacing it. README now shows `pi -t -analyze_image` (one-shot session without the re-query tool) and `pi -t +analyze_image` (add it back to a trimmed selection) alongside the existing persistent toggles (`/multimodal-proxy tool off`, `PI_VISION_PROXY_TOOL=off`). Privacy section updated to spell out the full current default-model chain.
+- Six new tests (546 total, all green): Haiku placement/ordering in the chain, key-aware preference of Haiku over Sonnet, catalog-without-Haiku skip, `durationMs` fallback/preference/finite-guard in the result formatter. Syntax check clean (8 files).
+
+## [1.21.0] - 2026-10-03
+
+### Added
+
+- **Compact transcript rows via `pi.registerToolRenderer()` (Pi ≥ 1.0.1, feature-detected)** — `analyze_image` and `generate_image` calls now render as one compact, themed line each instead of the default JSON-args dump. The `analyze_image` call line shows the question (word-boundary truncated, surrogate-pair safe), image references (recall ids → `image="deadbeef…"`, file paths → file name, `+N more` beyond two shown), a crop summary (`bottom-right@0` / `25%,13% 50%×50%@1` / `128,64 256×256@0` / `crops ×N`), and a model override; `generate_image` shows the prompt and override. Result lines summarize `✓ provider/model · cached · 1.2s` for analyses and `✓ N images via provider/model · $0.0311 · image="<id8>…"` for generations, or `✗ <error>`; expanding a row reveals the full text content. Generated images still render inline from the result's image blocks (independent of `renderResult`), reliable across Kitty/Ghostty/WezTerm/Warp via pi 1.0.1's JPEG/WebP/GIF `Image` fixes.
+- **Renderer coverage for unregistered tools** — registering through `registerToolRenderer` rather than inline `renderCall`/`renderResult` on the tool definitions means the compact rows also apply to calls replayed when the tool is *not registered* at render time: sessions resumed with the tool disabled (`/multimodal-proxy tool off`, mode off), transcript rows rendered before extension load, and HTML exports. Resolver chaining (`next() ?? {}`) preserves other extensions' renderers for the keys we don't set.
+- **Tool-result `details` payloads** — both tools now return a display-sized `details` object (analyze: `ok`/`error`/`cached`/`provider`/`model`/`latencyMs`/`groundingFormat`, with the heavy `text`/`images` stripped; generate: the structured payload as-is — ids/dims/cost) that feeds the renderer and persists in the session log without ever reaching the model.
+- Defensive renderer plumbing: pi-tui's `Text` (ANSI-aware wrapping) is loaded lazily with a width-truncating plain-lines fallback (ANSI-aware, rough wcwidth — pi's differential renderer stops on overwide lines) so an unresolvable package can never block extension load nor crash rendering; the expanded result view renders **every** text block, so a `generate_image` row on a text-only base model (where the `tool_result` hook replaces image blocks with description fences) still shows the per-image descriptions; renderers tolerate partial/garbage streaming args and missing result objects. (PR #37 review: CodeRabbit.)
+- New `tool-render.test.ts` (24 tests: truncation with a surrogate-pair regression trap, ref/crop summarization, call and result lines, garbage tolerance, details stripping) plus a fake-host smoke script (`tools/smoke-renderers.mjs`, real ANSI theme) that loads the real extension module and exercises the resolver contract end-to-end — 540 tests total, all green; also verified loading under the real pi 1.0.1 RPC loader.
+
+### Changed
+
+- Nothing user-visible on pi < 1.0.1 hosts: registration is gated on `typeof pi.registerToolRenderer === "function"`, so older runtimes keep the default tool rendering with zero behavior change.
+
 ## [1.20.0] - 2026-10-02
 
 ### Added

package/README.md

@@ -8,6 +8,28 @@ When **video or audio files** are detected, they are routed to a **multimodal mo
 
 **YouTube links** are detected too: paste a URL (`youtube.com/watch?v=…`, `youtu.be/…`, `/shorts/…`, etc.) and the video is downloaded with [`yt-dlp`](https://github.com/yt-dlp/yt-dlp) and analyzed exactly like a local file.
 
+**Compact tool rows (Pi ≥ 1.0.1)** — `analyze_image` and `generate_image` calls render as one tidy line each (`analyze_image  "What does the legend say?" · 2 images (image="deadbeef…", shot.png) · crop bottom-right@0`) with themed result summaries (`✓ zai/glm-5.3-flash · cached · 1.2s`, `✓ 1 image via google/imagen-4 · $0.0311`), instead of the default JSON-args dump. Generated images still render inline from the result blocks; expand a row to see the full analysis fence.
+
+## What's new in 1.22.0
+
+Borrowing from Pi 1.1.0:
+
+- **Claude Haiku 5.5 joins the implicit-model candidate chain** — `zai/glm-5.3-flash` → `deepseek/deepseek-v4-flash-vision-exp` → `anthropic/claude-haiku-5-5` (new, Pi ≥ 1.1.0) → `anthropic/claude-sonnet-5` → `anthropic/claude-sonnet-4-5`. When your cheaper candidates have no API key, an anthropic-keyed setup now lands on Haiku 5.5 instead of Sonnet 5 for image descriptions — $0.1/M in, $0.5/M out vs $2/M, $10/M (20× cheaper output at typical description-prompt sizes; above 100K input tokens Haiku is $0.5/$2.5, still 4× cheaper output). Same key, still vision-capable, and its catalog `inputLimits.images.resize` feeds the model-aware upload downscale. Catalogs without Haiku (pi < 1.1.0) skip it and behave exactly as before; explicit choices are never rewritten.
+- **Result rows show a duration even without `details`** — pi 1.1.0's render context carries `durationMs` (the recorded `execute()` time of a final tool result). The `analyze_image` result line now falls back to it when its `details` have no `latencyMs`, so rows replayed from foreign/HTML-exported sessions still show `✓ … · 1.2s`. Our own `latencyMs` (the vision call itself, reported to codemode scripts via `structuredContent`) always wins.
+- **Per-invocation tool toggling documented** — pi 1.1.0's `--tools +name/-name` entries modify the default tool selection instead of replacing it, so `pi -t -analyze_image` runs a one-shot session without the re-query tool while keeping everything else (see [Slash commands](#slash-commands)).
+
+## What's new in 1.21.0
+
+Adopting Pi 1.0.1's new `pi.registerToolRenderer()` API:
+
+- **Compact transcript rows for `analyze_image`** — the call line shows the question (truncated on a word boundary), the image refs (recall ids shortened to `image="deadbeef…"`, paths to their file name, `+N more` beyond two), a crop summary (`bottom-right@0`, `25%,13% 50%×50%@1`, `128,64 256×256@0`, or `crops ×N`), and any model override — instead of the raw JSON arguments. The result line summarizes `✓ provider/model · cached · latency` (or `✗ <error>`), and expanding the row reveals the full analysis fence.
+- **Compact transcript rows for `generate_image`** — the call line shows the prompt (and any model override); the result line summarizes `✓ N images via provider/model · $cost · image="<id8>…"`. The generated images themselves keep rendering inline — pi draws image blocks from the result content next to the renderer output — so this is purely a better header/summary, now reliable across Kitty, Ghostty, WezTerm, and Warp thanks to pi 1.0.1's JPEG/WebP/GIF image fixes.
+- **Works even when the tool isn't registered** — registering through `registerToolRenderer` (instead of inline `renderCall`/`renderResult` on the tool definitions) covers rows replayed from a session where the tool is now disabled (`/multimodal-proxy tool off`, mode off), resumed sessions, and HTML exports. `next()` chaining preserves renderers from other extensions for the keys we don't set.
+- **Renderer state via tool `details`** — results now carry a display-sized `details` payload (provider/model, `cached`, `latencyMs`, image ids, `costUSD`; heavy fields stripped) that feeds the renderer but never reaches the model.
+- Fully feature-gated (`typeof pi.registerToolRenderer === "function"`): pi < 1.0.1 hosts keep the default rendering, zero behavior change. pi-tui's `Text` component is loaded lazily with a plain-lines fallback so a missing package can never block extension load.
+
+Also benefits from pi 1.0.1 host fixes (no code needed here): Anthropic prompt caches survive mid-session tool toggles (tools added/redefined mid-conversation are now defined inline), "model at capacity" errors are retried instead of ending the turn, and Bedrock Claude thinking-signature failures after tool/prompt changes are fixed.
+
 ## What's new in 1.20.0
 
 Image generation, riding Pi 1.0's new `models.generateImages()` API:
@@ -235,6 +257,8 @@ Legacy alias: /vision-proxy <args> works identically.
 
 When an env var is set, the matching `/multimodal-proxy` subcommand is locked.
 
+For one-shot sessions you can also toggle tools per invocation (Pi ≥ 1.1.0): `pi -t -analyze_image` removes the re-query tool from the default selection, `pi -t +analyze_image` adds it back to a trimmed one. This complements the persistent `/multimodal-proxy tool off` and `PI_VISION_PROXY_TOOL=off`, which survive across sessions.
+
 ## How it works — Images
 
 ```
@@ -361,7 +385,7 @@ When a model is in the grounding registry, a format-specific instruction is appe
 
 ## Privacy & security
 
-This extension **sends data to a third-party provider**. By default that is `zai/glm-5.3-flash` for images (`anthropic/claude-sonnet-5` on older Pi versions without GLM 5.3 Flash in the catalog, then `anthropic/claude-sonnet-4-5`) and `xai/grok-4.3` for video/audio. Be aware:
+This extension **sends data to a third-party provider**. By default that is `zai/glm-5.3-flash` for images, falling back through `deepseek/deepseek-v4-flash-vision-exp` → `anthropic/claude-haiku-5-5` (Pi ≥ 1.1.0) → `anthropic/claude-sonnet-5` → `anthropic/claude-sonnet-4-5` to the first candidate in your catalog **and keyed** (`xai/grok-4.3` for video/audio). Be aware:
 
 1. **Image and video data is uploaded** to the configured provider on every proxied request. Crop coordinates are applied locally before upload — only the cropped region is sent.
 2. **Recent conversation context** (last 8 messages, truncated) is uploaded with the image unless you set `/multimodal-proxy context off` or `PI_VISION_PROXY_INCLUDE_CONTEXT=false`. Disable it for sensitive sessions.

package/extensions/__tests__/internal.test.ts

@@ -3312,6 +3312,41 @@ describe("1.18.0 key-aware default model resolution", () => {
 		assert.equal(DEFAULT_MODEL_FALLBACKS[0]!.modelId, "deepseek-v4-flash-vision-exp");
 	});
 
+	it("places Claude Haiku 5.5 between DeepSeek and Sonnet 5 (pi ≥ 1.1.0 catalogs)", () => {
+		const haiku = DEFAULT_MODEL_FALLBACKS.find((m) => m.modelId === "claude-haiku-5-5")!;
+		assert.equal(haiku.provider, "anthropic");
+		const deepseekIdx = DEFAULT_MODEL_FALLBACKS.findIndex((m) => m.provider === "deepseek");
+		const sonnetIdx = DEFAULT_MODEL_FALLBACKS.findIndex((m) => m.modelId === "claude-sonnet-5");
+		assert.ok(deepseekIdx < DEFAULT_MODEL_FALLBACKS.indexOf(haiku));
+		assert.ok(DEFAULT_MODEL_FALLBACKS.indexOf(haiku) < sonnetIdx);
+	});
+
+	it("prefers Haiku 5.5 over Sonnet 5 when both are keyed anthropic candidates", () => {
+		const out = applyDefaultModelFallback(
+			builtin,
+			registryWith(
+				`${builtin.provider}/${builtin.modelId}`,
+			"anthropic/claude-haiku-5-5",
+				"anthropic/claude-sonnet-5",
+			),
+			false,
+			keyWith("anthropic/claude-haiku-5-5", "anthropic/claude-sonnet-5"),
+		);
+		assert.equal(out.provider, "anthropic");
+		assert.equal(out.modelId, "claude-haiku-5-5");
+	});
+
+	it("skips Haiku 5.5 on pre-1.1.0 catalogs and still lands on Sonnet 5", () => {
+		const out = applyDefaultModelFallback(
+			builtin,
+			registryWith(`${builtin.provider}/${builtin.modelId}`, "anthropic/claude-sonnet-5"),
+			false,
+			keyWith("anthropic/claude-haiku-5-5", "anthropic/claude-sonnet-5"),
+		);
+		assert.equal(out.provider, "anthropic");
+		assert.equal(out.modelId, "claude-sonnet-5");
+	});
+
 	it("keeps the default when it is in the catalog and keyed", () => {
 		const out = applyDefaultModelFallback(
 			builtin,

package/extensions/__tests__/tool-render.test.ts

@@ -0,0 +1,274 @@
+/**
+ * Unit tests for the tool-transcript rendering helpers (1.21.0).
+ *
+ * These back the pi ≥ 1.0.1 registerToolRenderer() integration: compact
+ * analyze_image / generate_image call and result lines. Pure data in, plain
+ * segments out — theming happens in vision-proxy.ts.
+ *
+ * Run:
+ *   node --experimental-strip-types --test extensions/__tests__/tool-render.test.ts
+ */
+
+import { strict as assert } from "node:assert";
+import { describe, it } from "node:test";
+import {
+	analyzeDetailsForTranscript,
+	formatAnalyzeImageCall,
+	formatAnalyzeImageResult,
+	formatGenerateImageCall,
+	formatGenerateImageResult,
+	summarizeCrop,
+	summarizeForTranscript,
+	summarizeImageRef,
+} from "../internal.ts";
+
+const HASH32 = "deadbeefdeadbeefdeadbeefdeadbeef";
+
+function plain(segments: ReadonlyArray<{ text: string }>): string {
+	return segments.map((s) => s.text).join("");
+}
+
+describe("summarizeForTranscript", () => {
+	it("passes short text through with collapsed whitespace", () => {
+		assert.equal(summarizeForTranscript("  hello   world\n\tnext  ", 50), "hello world next");
+	});
+
+	it("truncates long text on a word boundary", () => {
+		const out = summarizeForTranscript("alpha bravo charlie delta echo foxtrot golf hotel", 20);
+		assert.ok(out.length <= 20, `length ${out.length}: ${out}`);
+		assert.ok(out.endsWith("…"));
+		assert.ok(!out.endsWith(" …"), "no dangling space before ellipsis");
+	});
+
+	it("never splits a surrogate pair", () => {
+		const emoji = "😀".repeat(60); // each emoji is a surrogate pair
+		const out = summarizeForTranscript(emoji, 20); // odd max → cut lands mid-pair
+		assert.ok(out.length <= 20);
+		// No lone high surrogate anywhere: the string still round-trips as text.
+		// (With the guard removed, the hard cut leaves an unpaired high surrogate
+		// right before the ellipsis, which this regex detects.)
+		assert.ok(!/[\uD800-\uDBFF](?![\uDC00-\uDFFF])/.test(out));
+	});
+});
+
+describe("summarizeImageRef", () => {
+	it("shortens a quoted recall reference", () => {
+		assert.equal(summarizeImageRef(`image="${HASH32}"`), `image="deadbeef…"`);
+	});
+
+	it("shortens an unquoted recall reference and sha256:/#crop forms", () => {
+		assert.equal(summarizeImageRef(`image=${HASH32}`), `image="deadbeef…"`);
+		assert.equal(summarizeImageRef(`sha256:${HASH32}#x0y0`), `image="deadbeef…"`);
+		assert.equal(summarizeImageRef(HASH32), `image="deadbeef…"`);
+	});
+
+	it("reduces file paths to the file name (posix + windows)", () => {
+		assert.equal(summarizeImageRef("/home/u/pics/cat.png"), "cat.png");
+		assert.equal(summarizeImageRef("D:\\Users\\u\\Downloads\\shot (1).jpg"), "shot (1).jpg");
+	});
+
+	it("truncates very long file names", () => {
+		const long = "a".repeat(120) + ".png";
+		const out = summarizeImageRef(long);
+		assert.ok(out.length <= 48);
+		assert.ok(out.endsWith("…"));
+	});
+
+	it("leaves non-hash short junk as-is", () => {
+		assert.equal(summarizeImageRef("not-a-hash"), "not-a-hash");
+	});
+});
+
+describe("summarizeCrop", () => {
+	it("renders named regions with the image index", () => {
+		assert.equal(summarizeCrop({ image_index: 0, region: "bottom-right" }), "bottom-right@0");
+	});
+
+	it("renders normalized crops as percentages", () => {
+		assert.equal(
+			summarizeCrop({ image_index: 1, normalized: { x: 0.25, y: 0.133, width: 0.5, height: 0.5 } }),
+			"25%,13.3% 50%×50%@1",
+		);
+	});
+
+	it("renders pixel crops rounded", () => {
+		assert.equal(
+			summarizeCrop({ image_index: 0, pixels: { x: 128.4, y: 64.2, width: 256, height: 256 } }),
+			"128,64 256×256@0",
+		);
+	});
+
+	it("returns empty for unrecognized input", () => {
+		assert.equal(summarizeCrop(undefined), "");
+		assert.equal(summarizeCrop("nonsense"), "");
+		assert.equal(summarizeCrop({ image_index: 0 }), "crop@0");
+	});
+});
+
+describe("formatAnalyzeImageCall", () => {
+	it("builds a full call line", () => {
+		const segs = formatAnalyzeImageCall({
+			images: [`image="${HASH32}"`, "/tmp/chart.png"],
+			question: "What does the legend say?",
+			crop: [{ image_index: 0, region: "center" }],
+			model: "zai/glm-5.3-flash",
+		});
+		assert.equal(
+			plain(segs),
+			`"What does the legend say?" · 2 images (image="deadbeef…", chart.png) · crop center@0 · zai/glm-5.3-flash`,
+		);
+		assert.equal(segs[0]!.style, "accent");
+	});
+
+	it("collapses many images to two shown plus a count", () => {
+		const segs = formatAnalyzeImageCall({
+			images: ["/a.png", "/b.png", "/c.png", "/d.png"],
+			question: "q",
+		});
+		assert.ok(plain(segs).includes("4 images (a.png, b.png +2)"));
+	});
+
+	it("summarizes multiple crops as a count", () => {
+		const segs = formatAnalyzeImageCall({
+			images: ["/a.png", "/b.png"],
+			question: "q",
+			crop: [
+				{ image_index: 0, region: "top" },
+				{ image_index: 1, region: "bottom" },
+			],
+		});
+		assert.ok(plain(segs).includes("crops ×2"));
+	});
+
+	it("truncates long questions and tolerates garbage args", () => {
+		const long = formatAnalyzeImageCall({ images: ["/a.png"], question: "x".repeat(300) });
+		assert.ok(plain(long).length < 200);
+
+		assert.deepEqual(formatAnalyzeImageCall("junk"), [{ text: '"junk"', style: "muted" }]);
+		assert.deepEqual(formatAnalyzeImageCall(null), [{ text: "null", style: "muted" }]);
+	});
+
+	it("handles missing question/images defensively", () => {
+		const segs = formatAnalyzeImageCall({});
+		assert.equal(plain(segs), '""');
+	});
+});
+
+describe("formatGenerateImageCall", () => {
+	it("builds prompt + model line", () => {
+		const segs = formatGenerateImageCall({ prompt: "A red panda astronaut, watercolor", model: "google/imagen-4" });
+		assert.equal(plain(segs), '"A red panda astronaut, watercolor" · google/imagen-4');
+	});
+
+	it("prompt-only line and garbage fallback", () => {
+		assert.equal(plain(formatGenerateImageCall({ prompt: "cat" })), '"cat"');
+		assert.deepEqual(formatGenerateImageCall(42), [{ text: "42", style: "muted" }]);
+	});
+});
+
+describe("formatAnalyzeImageResult", () => {
+	it("summarizes a successful analysis", () => {
+		const segs = formatAnalyzeImageResult(
+			{ ok: true, provider: "zai", model: "glm-5.3-flash", cached: true, latencyMs: 1234 },
+			false,
+		);
+		assert.equal(plain(segs), "✓ zai/glm-5.3-flash · cached · 1.2s");
+	});
+
+	it("summarizes without optional fields and with model only", () => {
+		assert.equal(plain(formatAnalyzeImageResult({ ok: true }, false)), "✓");
+		assert.equal(plain(formatAnalyzeImageResult({ ok: true, model: "glm-5.3-flash" }, false)), "✓ glm-5.3-flash");
+	});
+
+	it("renders errors from details, isError flag, or a generic fallback", () => {
+		assert.equal(
+			plain(formatAnalyzeImageResult({ ok: false, error: "Error: consent required" }, true)),
+			"✗ Error: consent required",
+		);
+		// isError without details
+		assert.equal(plain(formatAnalyzeImageResult(undefined, true)), "✗ analysis failed");
+		// ok:false wins even without the flag (older rows)
+		assert.equal(plain(formatAnalyzeImageResult({ ok: false }, false)), "✗ analysis failed");
+	});
+
+	it("marks the summary segment styles", () => {
+		const segs = formatAnalyzeImageResult({ ok: true, provider: "zai", model: "m" }, false);
+		assert.equal(segs[0]!.style, "success");
+		assert.equal(segs[1]!.style, "muted");
+	});
+
+	describe("durationMs fallback (pi ≥ 1.1.0 render context)", () => {
+		it("uses the host durationMs when details carry no latencyMs", () => {
+			assert.equal(
+				plain(formatAnalyzeImageResult({ ok: true, provider: "zai", model: "m" }, false, 2500)),
+				"✓ zai/m · 2.5s",
+			);
+			// rows with no details at all (replayed foreign sessions, HTML exports)
+			assert.equal(plain(formatAnalyzeImageResult(undefined, false, 987)), "✓ · 1.0s");
+		});
+
+		it("prefers details.latencyMs over the host durationMs", () => {
+			const segs = formatAnalyzeImageResult(
+				{ ok: true, provider: "zai", model: "glm-5.3-flash", latencyMs: 1234 },
+				false,
+				99999,
+			);
+			assert.equal(plain(segs), "✓ zai/glm-5.3-flash · 1.2s");
+		});
+
+		it("ignores non-finite durationMs", () => {
+			assert.equal(plain(formatAnalyzeImageResult({ ok: true }, false, Number.NaN)), "✓");
+		});
+	});
+});
+
+describe("formatGenerateImageResult", () => {
+	it("summarizes generation with images, cost, and first id", () => {
+		const segs = formatGenerateImageResult(
+			{
+				ok: true,
+				provider: "google",
+				model: "imagen-4",
+				images: [
+					{ id: HASH32, mimeType: "image/png", width: 1024, height: 768 },
+					{ id: "f".repeat(32), mimeType: "image/png", width: 1024, height: 768 },
+				],
+				costUSD: 0.0311,
+			},
+			false,
+		);
+		assert.equal(plain(segs), `✓ 2 images via google/imagen-4 · $0.0311 · image="deadbeef…" +1`);
+	});
+
+	it("renders errors from details or generically", () => {
+		assert.equal(
+			plain(formatGenerateImageResult({ ok: false, error: "Error: no keyed image model" }, true)),
+			"✗ Error: no keyed image model",
+		);
+		assert.equal(plain(formatGenerateImageResult(undefined, true)), "✗ generation failed");
+	});
+
+	it("falls back to 'done' when details carry no image list", () => {
+		assert.equal(plain(formatGenerateImageResult({ ok: true, provider: "google", model: "imagen-4" }, false)), "✓ done via google/imagen-4");
+	});
+});
+
+describe("analyzeDetailsForTranscript", () => {
+	it("strips text and images but keeps display fields", () => {
+		const out = analyzeDetailsForTranscript({
+			ok: true,
+			text: "a".repeat(2000),
+			images: [{ id: HASH32 }],
+			provider: "zai",
+			model: "glm-5.3-flash",
+			latencyMs: 120,
+		});
+		assert.deepEqual(out, { ok: true, provider: "zai", model: "glm-5.3-flash", latencyMs: 120 });
+	});
+
+	it("does not mutate the input", () => {
+		const input = { ok: true, text: "keep-me", images: [1, 2] };
+		analyzeDetailsForTranscript(input);
+		assert.deepEqual(input, { ok: true, text: "keep-me", images: [1, 2] });
+	});
+});

package/extensions/internal.ts

@@ -1318,6 +1318,7 @@ export function resolveConfig(
  */
 export const DEFAULT_MODEL_FALLBACKS: ReadonlyArray<{ provider: string; modelId: string }> = [
 	{ provider: "deepseek", modelId: "deepseek-v4-flash-vision-exp" }, // Pi ≥ 0.84.4, cheap, common key
+	{ provider: "anthropic", modelId: "claude-haiku-5-5" }, // Pi ≥ 1.1.0, vision-capable, $0.1/$0.5 per M
 	{ provider: "anthropic", modelId: "claude-sonnet-5" },
 	{ provider: "anthropic", modelId: "claude-sonnet-4-5" },
 ];
@@ -3263,6 +3264,213 @@ export function buildToolCacheKey(
 	return `${sortedHashes.join("+")}${cropSig ? "#crop:" + cropSig : ""}?q=${questionHash}&m=${modelId}`;
 }
 
+// ── Tool transcript rendering (pi ≥ 1.0.1 registerToolRenderer) ────────────
+
+/**
+ * Styling hints for one segment of a tool-transcript line, mapped to
+ * theme.fg tokens by the renderer in vision-proxy.ts. Kept as plain data so
+ * these helpers stay pure and unit-testable without a terminal theme.
+ */
+export type TranscriptStyle = "plain" | "accent" | "muted" | "dim" | "success" | "warning" | "error";
+
+export interface TranscriptSegment {
+	text: string;
+	style: TranscriptStyle;
+}
+
+/** Collapse whitespace and truncate on a word boundary (surrogate-pair safe). */
+export function summarizeForTranscript(text: string, max: number): string {
+	const t = text.replace(/\s+/g, " ").trim();
+	if (t.length <= max) return t;
+	let cut = t.slice(0, Math.max(1, max - 1));
+	// Never split a surrogate pair on a hard cut (lone high surrogate → U+FFFD).
+	if (/[\uD800-\uDBFF]$/.test(cut)) cut = cut.slice(0, -1);
+	const ws = cut.lastIndexOf(" ");
+	if (ws > cut.length * 0.6) cut = cut.slice(0, ws);
+	return `${cut.trimEnd()}…`;
+}
+
+/** Defensive finite-number check for values coming from model-authored args. */
+function isFiniteNumber(v: unknown): v is number {
+	return typeof v === "number" && Number.isFinite(v);
+}
+
+/** Compact percentage for crop summaries: integral values stay integral. */
+function pct(v: number): string {
+	const p = v * 100;
+	return Math.abs(p - Math.round(p)) < 0.05 ? String(Math.round(p)) : String(Math.round(p * 10) / 10);
+}
+
+/** Safe one-line JSON rendering of an arbitrary value (circular-safe). */
+function safeJson(value: unknown, max = 120): string {
+	try {
+		return summarizeForTranscript(JSON.stringify(value) ?? String(value), max);
+	} catch {
+		return String(value);
+	}
+}
+
+/**
+ * Shorten one analyze_image image reference for display: a recall id
+ * (`image="<hash>"`, `sha256:<hash>#…`, or a bare hash) becomes
+ * `image="deadbeef…"`; a file path becomes its file name (Windows or POSIX).
+ */
+export function summarizeImageRef(ref: string): string {
+	const s = ref.trim().replace(/^["']|["']$/g, "");
+	const quoted = /image="?([a-f0-9]{16,64})"?/i.exec(s);
+	if (quoted) return `image="${quoted[1]!.slice(0, 8)}…"`;
+	const bare = parseRecallRef(s);
+	if (bare) return `image="${bare.slice(0, 8)}…"`;
+	const base = s.replace(/\\/g, "/").split("/").filter(Boolean).pop() ?? s;
+	return summarizeForTranscript(base, 48);
+}
+
+/**
+ * Compact crop descriptor for the transcript: `bottom-right@0`,
+ * `25%,13% 50%×50%@1`, or `128,64 256×256@0`. Empty string when nothing
+ * recognizable.
+ */
+export function summarizeCrop(crop: unknown): string {
+	if (typeof crop !== "object" || crop === null) return "";
+	const c = crop as Record<string, unknown>;
+	const idx =
+		isFiniteNumber(c.image_index) ? `@${Math.trunc(c.image_index)}` : "";
+	if (typeof c.region === "string" && c.region.trim()) return `${c.region.trim()}${idx}`;
+	const box = (c.normalized ?? c.pixels) as Record<string, unknown> | undefined;
+	if (
+		typeof box === "object" &&
+		box !== null &&
+		isFiniteNumber(box.x) &&
+		isFiniteNumber(box.y) &&
+		isFiniteNumber(box.width) &&
+		isFiniteNumber(box.height)
+	) {
+		if (c.normalized !== undefined) {
+			return `${pct(box.x)}%,${pct(box.y)}% ${pct(box.width)}%×${pct(box.height)}%${idx}`;
+		}
+		return `${Math.round(box.x)},${Math.round(box.y)} ${Math.round(box.width)}×${Math.round(box.height)}${idx}`;
+	}
+	return idx ? `crop${idx}` : "";
+}
+
+/** One-line transcript summary of an analyze_image call (pi ≥ 1.0.1 renderer). */
+export function formatAnalyzeImageCall(args: unknown): TranscriptSegment[] {
+	if (typeof args !== "object" || args === null || Array.isArray(args)) {
+		return [{ text: safeJson(args), style: "muted" }];
+	}
+	const a = args as Record<string, unknown>;
+	const out: TranscriptSegment[] = [];
+	const question = typeof a.question === "string" ? a.question : "";
+	out.push({ text: `"${summarizeForTranscript(question, 120)}"`, style: "accent" });
+	const images = Array.isArray(a.images)
+		? a.images.filter((r): r is string => typeof r === "string" && r.trim() !== "")
+		: [];
+	if (images.length > 0) {
+		const shown = images.slice(0, 2).map(summarizeImageRef);
+		const more = images.length - shown.length;
+		out.push({
+			text:
+				` · ${images.length} image${images.length === 1 ? "" : "s"}` +
+				(shown.length ? ` (${shown.join(", ")}${more > 0 ? ` +${more}` : ""})` : ""),
+			style: "muted",
+		});
+	}
+	const crops = Array.isArray(a.crop) ? a.crop.map(summarizeCrop).filter((s) => s !== "") : [];
+	if (crops.length === 1) out.push({ text: ` · crop ${crops[0]}`, style: "dim" });
+	else if (crops.length > 1) out.push({ text: ` · crops ×${crops.length}`, style: "dim" });
+	const model = typeof a.model === "string" && a.model.trim() ? a.model.trim() : undefined;
+	if (model) out.push({ text: ` · ${summarizeForTranscript(model, 60)}`, style: "muted" });
+	return out;
+}
+
+/** One-line transcript summary of a generate_image call (pi ≥ 1.0.1 renderer). */
+export function formatGenerateImageCall(args: unknown): TranscriptSegment[] {
+	if (typeof args !== "object" || args === null || Array.isArray(args)) {
+		return [{ text: safeJson(args), style: "muted" }];
+	}
+	const a = args as Record<string, unknown>;
+	const prompt = typeof a.prompt === "string" ? a.prompt : "";
+	const out: TranscriptSegment[] = [{ text: `"${summarizeForTranscript(prompt, 120)}"`, style: "accent" }];
+	const model = typeof a.model === "string" && a.model.trim() ? a.model.trim() : undefined;
+	if (model) out.push({ text: ` · ${summarizeForTranscript(model, 60)}`, style: "muted" });
+	return out;
+}
+
+function recordOf(details: unknown): Record<string, unknown> | undefined {
+	return typeof details === "object" && details !== null && !Array.isArray(details)
+		? (details as Record<string, unknown>)
+		: undefined;
+}
+
+/**
+ * One-line transcript summary of an analyze_image result from its `details`.
+ * `durationMs` (pi ≥ 1.1.0 host render context) is a display fallback for rows
+ * whose `details` carry no `latencyMs` — replayed sessions, foreign/HTML
+ * exports, results recorded before `details` existed. Details always win:
+ * `latencyMs` measures the vision call itself, while the host's `durationMs`
+ * covers the whole `execute()` (consent checks, crop encoding included).
+ */
+export function formatAnalyzeImageResult(
+	details: unknown,
+	isError: boolean,
+	durationMs?: number,
+): TranscriptSegment[] {
+	const d = recordOf(details);
+	if (isError || d?.ok === false) {
+		const msg = typeof d?.error === "string" && d.error ? d.error : "analysis failed";
+		return [{ text: `✗ ${summarizeForTranscript(msg, 160)}`, style: "error" }];
+	}
+	const segments: TranscriptSegment[] = [{ text: "✓", style: "success" }];
+	const provider = typeof d?.provider === "string" && d.provider ? d.provider : undefined;
+	const modelId = typeof d?.model === "string" && d.model ? d.model : undefined;
+	if (provider && modelId) segments.push({ text: ` ${summarizeForTranscript(`${provider}/${modelId}`, 80)}`, style: "muted" });
+	else if (modelId) segments.push({ text: ` ${summarizeForTranscript(modelId, 80)}`, style: "muted" });
+	if (d?.cached === true) segments.push({ text: " · cached", style: "dim" });
+	const latency = isFiniteNumber(d?.latencyMs) ? d.latencyMs : isFiniteNumber(durationMs) ? durationMs : undefined;
+	if (latency !== undefined) {
+		segments.push({ text: ` · ${(Math.max(0, latency) / 1000).toFixed(1)}s`, style: "dim" });
+	}
+	return segments;
+}
+
+/** One-line transcript summary of a generate_image result from its `details`. */
+export function formatGenerateImageResult(details: unknown, isError: boolean): TranscriptSegment[] {
+	const d = recordOf(details);
+	if (isError || d?.ok === false) {
+		const msg = typeof d?.error === "string" && d.error ? d.error : "generation failed";
+		return [{ text: `✗ ${summarizeForTranscript(msg, 160)}`, style: "error" }];
+	}
+	const images = Array.isArray(d?.images)
+		? (d!.images as unknown[]).filter((x): x is Record<string, unknown> => typeof x === "object" && x !== null)
+		: [];
+	const provider = typeof d?.provider === "string" && d.provider ? d.provider : undefined;
+	const modelId = typeof d?.model === "string" && d.model ? d.model : undefined;
+	const count = images.length > 0 ? `${images.length} image${images.length === 1 ? "" : "s"}` : "done";
+	const segments: TranscriptSegment[] =
+		provider && modelId
+			? [{ text: `✓ ${count} via ${summarizeForTranscript(`${provider}/${modelId}`, 80)}`, style: "success" }]
+			: [{ text: `✓ ${count}`, style: "success" }];
+	if (isFiniteNumber(d?.costUSD)) segments.push({ text: ` · $${Math.max(0, d.costUSD).toFixed(4)}`, style: "dim" });
+	const firstId = typeof images[0]?.id === "string" && images[0].id ? images[0].id : undefined;
+	if (firstId) {
+		const more = images.length > 1 ? ` +${images.length - 1}` : "";
+		segments.push({ text: ` · image="${firstId.slice(0, 8)}…"${more}`, style: "dim" });
+	}
+	return segments;
+}
+
+/**
+ * Copy of a structured analyze_image result without the fields the
+ * transcript renderer never reads — `details` must stay small because it is
+ * persisted in the session log alongside the (already stored) text fence.
+ */
+export function analyzeDetailsForTranscript(structured: Record<string, unknown>): Record<string, unknown> {
+	const rest: Record<string, unknown> = { ...structured };
+	delete rest.text;
+	delete rest.images;
+	return rest;
+}
+
 // ── Fence builders ────────────────────────────────────────────────────────
 
 export interface VideoDescriptionEntry {

package/extensions/vision-proxy.ts

@@ -256,6 +256,12 @@ import {
 	buildVideoProxySection,
 	extractXaiResponsesText,
 	formatXaiSttTranscript,
+	analyzeDetailsForTranscript,
+	formatAnalyzeImageCall,
+	formatAnalyzeImageResult,
+	formatGenerateImageCall,
+	formatGenerateImageResult,
+	type TranscriptSegment,
 	isTranscriptionRequest,
 	isXaiProvider,
 	bufferToPiAiImage,
@@ -2381,6 +2387,229 @@ export default function (pi: ExtensionAPI) {
 		);
 	}
 
+	// ── Tool transcript renderers (pi ≥ 1.0.1) ─────────────────────────────
+
+	/** Minimal structural shape of the TUI component the renderers return. */
+	interface ComponentLike {
+		render(width: number): string[];
+		invalidate?(): void;
+	}
+
+	/** Theme subset the renderers use (pi's Theme helpers). */
+	interface RenderThemeLike {
+		fg(token: string, text: string): string;
+		bold(text: string): string;
+	}
+
+	/** ToolRenderContext subset available to renderCall/renderResult. */
+	interface RenderContextLike {
+		isError?: boolean;
+		isPartial?: boolean;
+		expanded?: boolean;
+		showImages?: boolean;
+		/** pi ≥ 1.1.0: recorded execute() duration from the final result. */
+		durationMs?: number;
+	}
+
+	/** AgentToolResult subset the result renderer receives. */
+	interface RenderResultLike {
+		content?: ReadonlyArray<{ type?: string; text?: string }>;
+		details?: unknown;
+	}
+
+	type ToolRenderersLike = {
+		renderCall?: (args: unknown, theme: RenderThemeLike, context: RenderContextLike) => ComponentLike;
+		renderResult?: (
+			result: RenderResultLike,
+			options: { expanded: boolean; isPartial: boolean },
+			theme: RenderThemeLike,
+			context: RenderContextLike,
+		) => ComponentLike;
+	};
+
+	/**
+	 * pi-tui's Text (ANSI-aware word wrap), loaded lazily so an unresolvable
+	 * package can never block extension load — the crash-loop lesson from
+	 * 1.16.0. Warmed at registration time, long before the first tool row
+	 * renders.
+	 */
+	let _textCtor: (new (text: string, paddingX?: number, paddingY?: number) => ComponentLike) | undefined;
+	let _textLoadTried = false;
+
+	function preloadTextComponent(): void {
+		if (_textLoadTried) return;
+		_textLoadTried = true;
+		void import("@earendil-works/pi-tui")
+			.then((m) => {
+				const ctor = (
+					m as { Text?: new (text: string, paddingX?: number, paddingY?: number) => ComponentLike }
+				).Text;
+				if (ctor) _textCtor = ctor;
+			})
+			.catch(() => {
+				// pi-tui unreachable — the plain-lines fallback below keeps rendering
+			});
+	}
+
+	/**
+	 * Fixed-lines component used only when pi-tui's Text cannot load.
+	 * Truncates each line to the render width (ANSI-aware, rough wcwidth) —
+	 * pi's differential renderer stops on "rendered line exceeds terminal
+	 * width", so an overwide fallback line is worse than an ugly one.
+	 */
+	function plainLines(text: string): ComponentLike {
+		return {
+			render: (width: number) => text.split("\n").map((line) => truncateAnsiLine(line, Math.max(1, width))),
+			invalidate: () => {},
+		};
+	}
+
+	/** Rough wcwidth: 2 for the common wide ranges, 0 for combining marks, else 1. */
+	function codePointWidth(cp: number): number {
+		if (cp >= 0x0300 && cp <= 0x036f) return 0; // combining diacritics
+		if (
+			(cp >= 0x1100 && cp <= 0x115f) || // Hangul Jamo
+			(cp >= 0x2e80 && cp <= 0xa4cf) || // CJK radicals … Yi
+			(cp >= 0xac00 && cp <= 0xd7a3) || // Hangul syllables
+			(cp >= 0xf900 && cp <= 0xfaff) || // CJK compat ideographs
+			(cp >= 0xfe30 && cp <= 0xfe4f) ||
+			(cp >= 0xff00 && cp <= 0xff60) ||
+			(cp >= 0xffe0 && cp <= 0xffe6) ||
+			(cp >= 0x1f300 && cp <= 0x1faff) || // emoji blocks
+			(cp >= 0x20000 && cp <= 0x3fffd) // CJK ext
+		) {
+			return 2;
+		}
+		return 1;
+	}
+
+	/** ANSI-preserving truncation of one styled line to a visible width. */
+	function truncateAnsiLine(line: string, width: number): string {
+		let out = "";
+		let w = 0;
+		let i = 0;
+		while (i < line.length) {
+			if (line[i] === "\x1b") {
+				const m = /^\x1b\[[0-9;]*m/.exec(line.slice(i));
+				const seq = m ? m[0] : line[i]!;
+				out += seq;
+				i += seq.length;
+				continue;
+			}
+			const cp = line.codePointAt(i)!;
+			const s = String.fromCodePoint(cp);
+			const cw = codePointWidth(cp);
+			if (w + cw > width) break;
+			out += s;
+			w += cw;
+			i += s.length;
+		}
+		return out;
+	}
+
+	function textComponent(text: string): ComponentLike {
+		return _textCtor ? new _textCtor(text, 0, 0) : plainLines(text);
+	}
+
+	function styleSegments(segments: readonly TranscriptSegment[], theme: RenderThemeLike): string {
+		let out = "";
+		for (const seg of segments) {
+			out += seg.style === "plain" ? seg.text : theme.fg(seg.style, seg.text);
+		}
+		return out;
+	}
+
+	function textBlocks(content: ReadonlyArray<{ type?: string; text?: string }> | undefined): string[] {
+		if (!content) return [];
+		const out: string[] = [];
+		for (const block of content) {
+			if (block?.type === "text" && typeof block.text === "string" && block.text) out.push(block.text);
+		}
+		return out;
+	}
+
+	function toolTitle(name: string, theme: RenderThemeLike): string {
+		return theme.fg("toolTitle", theme.bold(`${name}  `));
+	}
+
+	/** Compact result component: summary line, plus the full text content when expanded. */
+	function resultComponent(
+		summary: readonly TranscriptSegment[],
+		result: RenderResultLike,
+		options: { expanded: boolean },
+		theme: RenderThemeLike,
+	): ComponentLike {
+		let text = styleSegments(summary, theme);
+		if (options.expanded) {
+			// All text blocks: a generate_image result on a text-only base model
+			// carries the header plus per-image description fences that replaced
+			// the image blocks — the first block alone would hide them.
+			const body = textBlocks(result.content).join("\n\n");
+			if (body) {
+				// Re-apply the style per line: pi resets styling after every line.
+				text += `\n${body.split("\n").map((line) => theme.fg("toolOutput", line)).join("\n")}`;
+			}
+		}
+		return textComponent(text);
+	}
+
+	/**
+	 * Register transcript renderers for analyze_image/generate_image (pi ≥
+	 * 1.0.1). Registered through pi.registerToolRenderer — instead of inline
+	 * renderCall/renderResult on the tool definitions — so the compact rows
+	 * also cover tools that are *not registered* at render time: calls
+	 * replayed from a session where the tool is now disabled (`tool off` /
+	 * mode off), resumed sessions rendered before extension load, and HTML
+	 * exports. `next() ?? {}` keeps renderers from later resolvers / the
+	 * registered tool for the keys we don't set. Generated images keep
+	 * rendering inline regardless: pi draws image blocks from the result
+	 * content next to the renderer output, independent of renderResult.
+	 */
+	function registerToolRenderers() {
+		const api = pi as unknown as {
+			registerToolRenderer?: (
+				resolver: (toolName: string, next: () => ToolRenderersLike | undefined) => ToolRenderersLike | undefined,
+			) => void;
+		};
+		if (typeof api.registerToolRenderer !== "function") return; // pi < 1.0.1
+		preloadTextComponent();
+		api.registerToolRenderer((toolName, next) => {
+			if (toolName === "analyze_image") {
+				return {
+					...(next() ?? {}),
+					renderCall: (args, theme) =>
+						textComponent(toolTitle("analyze_image", theme) + styleSegments(formatAnalyzeImageCall(args), theme)),
+					renderResult: (result, options, theme, context) => {
+						if (options.isPartial) return textComponent(theme.fg("warning", "Analyzing…"));
+						return resultComponent(
+							formatAnalyzeImageResult(result?.details, context?.isError === true, context?.durationMs),
+							result ?? {},
+							options,
+							theme,
+						);
+					},
+				};
+			}
+			if (toolName === "generate_image") {
+				return {
+					...(next() ?? {}),
+					renderCall: (args, theme) =>
+						textComponent(toolTitle("generate_image", theme) + styleSegments(formatGenerateImageCall(args), theme)),
+					renderResult: (result, options, theme, context) => {
+						if (options.isPartial) return textComponent(theme.fg("warning", "Generating…"));
+						return resultComponent(
+							formatGenerateImageResult(result?.details, context?.isError === true),
+							result ?? {},
+							options,
+							theme,
+						);
+					},
+				};
+			}
+			return next();
+		});
+	}
+
 	// ── Virtual model (pi ≥ 0.99) ─────────────────────────────────────────────
 
 	const VIRTUAL_PROVIDER = "multimodal-proxy";
@@ -2571,6 +2800,10 @@ export default function (pi: ExtensionAPI) {
 			// unreadable config file — session_start surfaces the error path
 		});
 
+	// Transcript renderers for analyze_image/generate_image rows (pi ≥ 1.0.1).
+	// Registered once at load; the resolver serves any later session state.
+	registerToolRenderers();
+
 	/** Register or unregister the analyze_image tool based on config. */
 	function syncToolRegistration(config: VisionConfig, runtimeCtx?: ExtensionContext) {
 		const shouldHaveTool = config.mode !== "off" && config.tool === "on";
@@ -2630,6 +2863,9 @@ export default function (pi: ExtensionAPI) {
 						// codemode scripts. Older hosts ignore both fields.
 						...(outcome.isError ? { isError: true } : {}),
 						...(outcome.structured ? { structuredContent: outcome.structured } : {}),
+						// `details` feeds the transcript renderer (pi ≥ 1.0.1) without
+						// the model seeing it; heavy fields (text, images) are stripped.
+						...(outcome.structured ? { details: analyzeDetailsForTranscript(outcome.structured) } : {}),
 					};
 				},
 			});
@@ -2705,6 +2941,9 @@ export default function (pi: ExtensionAPI) {
 						content: outcome.content,
 						...(outcome.isError ? { isError: true } : {}),
 						...(outcome.structured ? { structuredContent: outcome.structured } : {}),
+						// `details` feeds the transcript renderer (pi ≥ 1.0.1); the
+						// structured payload is already display-sized (ids, dims, cost).
+						...(outcome.structured ? { details: outcome.structured } : {}),
 						...(outcome.usage !== undefined ? { usage: outcome.usage } : {}),
 					};
 				},

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-multimodal-proxy",
-  "version": "1.20.0",
+  "version": "1.22.0",
   "description": "Automatic image, video and audio description for any model in Pi. Routes media to a multimodal model and injects descriptions into context.",
   "keywords": [
     "pi-package"