pi-multimodal-proxy 1.22.1 → 1.23.0

package/CHANGELOG.md

@@ -4,6 +4,12 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
+## [1.23.0] - 2026-10-10
+
+### Fixed
+
+- **`#` image-recall autocomplete never fired on vision-capable base models** — two compounding defects, found live while dogfooding the 1.22.1 README claims. First, the dropdown drew candidates **only** from persisted `DescriptionEntry`s, so images analyzed via the `analyze_image` tool (which stores bytes in the in-session recall store but appends no description entry) and images retained in the store were invisible to it. Second, in the most common modern configuration (multimodal base model + default `fallback` mode) pasted/attached images pass through as pixels and entered **no** store at all — no description, no bytes, no meta — leaving the autocomplete permanently empty no matter what the user did. Fixes: `collectRecallCandidates` now unions described images with the recall store's keys (described entries win on collision, store-only candidates show the filename or hash-prefix label); and the `before_agent_start` pass-through branch retains attachment bytes+meta in the existing in-memory LRU store (no vision call, no description, no new persistence — the bytes were already in process as message attachments). Net effect: `#` and `analyze_image` recall now work in every mode except `off` (where the extension is disabled by intent and retains nothing), matching the README's "media memory" pillar. Two review fixes folded in (PR #40): retention is skipped entirely in `off` mode, and `storeImageData` runs unconditionally on re-attachment so the LRU refreshes recency (attach A, B, A, C evicts B, not A). Verified against pi 1.1.0's `AutocompleteItem`/`AutocompleteProvider` shapes (no API drift involved).
+
 ## [1.22.1] - 2026-10-10
 
 ### Changed

package/README.md

@@ -7,7 +7,7 @@ Media understanding for [Pi](https://pi.dev): images, video, audio, and YouTube
 - **Video, audio & YouTube** — paste a URL (`youtube.com/watch?v=…`, `youtu.be/…`, `/shorts/…`) or a local file path and a multimodal model (default: Grok 4.3) transcribes speech with speaker diarization, describes visual scenes, reads on-screen text, and reasons about the content in a single call. Almost no chat model or agent ingests video natively: this is the package's deepest capability.
 - **Vision for text-only models** — a large slice of Pi's catalog (DeepSeek, GLM, Nemotron, Devstral, gpt-oss, …) still has no image input. When an image reaches such a model, the extension describes it via your configured vision model, persists the description in the session, and injects it into context. The model "sees" images across turns, cached by content hash.
 - **Cheaper vision for multimodal models** — select `Multimodal Auto` and media turns escalate natively to the vision model (real pixels, no text fence; prompt caches survive the routing) while text turns stay on your base model. Pin it with `/multimodal-proxy virtual-base` to keep the cheap model after media turns too: unpinned, a text turn reuses whatever physical model handled the turn before, which after a media turn is the vision model. Or proxy every image through a flash-priced describer with mode `always`.
-- **Media memory that outlives the turn** — every proxied image gets a stable `image="<id>"`, so the agent can re-query or crop an earlier screenshot without a re-attachment ("zoom into that screenshot from before"), `#` autocompletes image ids in the prompt, and descriptions survive compaction via a recall digest. `analyze_image` answers targeted questions with locally applied crops; `generate_image` creates images that flow straight back through the same pipeline. (Recall applies to described images: text-only base models or mode `always`. Images passed through natively to a vision-capable base model stay in context as pixels and are not duplicated into recall.)
+- **Media memory that outlives the turn** — every image the agent can re-query gets a stable `image="<id>"`: described images (text-only base models, mode `always`, or explicit `analyze_image` calls) carry a description snippet, and pasted attachments on a vision-capable base model are retained in the recall store without a description. The agent can re-query or crop an earlier screenshot without a re-attachment ("zoom into that screenshot from before"), `#` autocompletes image ids in the prompt, and descriptions survive compaction via a recall digest. `analyze_image` answers targeted questions with locally applied crops; `generate_image` creates images that flow straight back through the same pipeline.
 - **Governed by default** — first-use data-egress consent per provider, persisted pre-consent lists, crops applied locally before upload, upload downscaling, decode-bomb protection, and recall bytes kept in memory only. See [Privacy & security](#privacy--security).
 
 Media routing is automatic: content sniffing and path detection feed a pipeline with transient-error retry and model fallback on the image path (the native xAI video path is one-shot), plus compact transcript rows for `analyze_image`/`generate_image` (Pi ≥ 1.0.1).
@@ -206,7 +206,10 @@ Every `<vision_proxy_description>`, `<vision_proxy_analysis>`, and
 can pass that id back to `analyze_image` (or `/multimodal-proxy describe`) to
 re-examine or crop an image the user shared earlier in the session — even once
 it is no longer attached to the current message (e.g. *"zoom into that
-screenshot from before"*). The image bytes are retained **in memory only** for
+screenshot from before"*). Pasted attachments on a vision-capable base model
+are also retained (no description; labeled by filename, or by hash prefix when
+the paste carries no filename), so `#` recall works in every mode except `off`.
+The image bytes are retained **in memory only** for
 the life of the session, never written to the session log or disk, and are
 evicted oldest-first once the recall budget (`PI_VISION_PROXY_IMAGE_RECALL_BYTES`,
 default 64 MB) is exceeded.

package/extensions/__tests__/recall-autocomplete.test.ts

@@ -64,6 +64,40 @@ describe("collectRecallCandidates", () => {
 			{ hash: "h1", description: "first image", filename: undefined },
 		]);
 	});
+
+	it("merges store-only hashes (pass-through attachments) after described ones", () => {
+		const descriptions = new Map([["h1", "described earlier"]]);
+		const meta = new Map([["h3", { width: 8, height: 6, filename: "paste.png" }]]);
+		const out = collectRecallCandidates(descriptions, (h) => meta.get(h), ["h2", "h3"]);
+		assert.deepEqual(out, [
+			{ hash: "h1", description: "described earlier", filename: undefined },
+			{ hash: "h3", description: undefined, filename: "paste.png" }, // newest store entry first
+			{ hash: "h2", description: undefined, filename: undefined },
+		]);
+	});
+
+	it("descriptions win when a hash appears in both sources", () => {
+		const out = collectRecallCandidates(
+			new Map([["shared", "the description"]]),
+			() => undefined,
+			["shared", "other"],
+		);
+		assert.equal(out.length, 2);
+		assert.equal(out[0]!.description, "the description");
+		assert.equal(out[1]!.hash, "other");
+	});
+
+	it("feeds store-only candidates into buildRecallItems labels", () => {
+		const hash = "c".repeat(32);
+		const candidates = collectRecallCandidates(new Map(), () => ({ width: 2, height: 2, filename: "shot.png" }), [hash]);
+		const items = buildRecallItems(candidates, "");
+		assert.equal(items.length, 1);
+		assert.equal(items[0]!.label, "shot.png");
+		assert.ok(items[0]!.value.endsWith(hash));
+		// no filename → hash-prefix label
+		const bare = buildRecallItems(collectRecallCandidates(new Map(), () => undefined, [hash]), "");
+		assert.equal(bare[0]!.label, `${hash.slice(0, 12)}…`);
+	});
 });
 
 describe("buildRecallItems", () => {

package/extensions/internal.ts

@@ -318,16 +318,28 @@ export function extractRecallToken(
 }
 
 /**
- * Collect recall candidates from persisted descriptions, most recent first.
+ * Collect recall candidates for the `#` autocomplete, most recent first.
  * `metaLookup` supplies in-memory filename/dimensions when still available.
+ * `recallableHashes` (keys of the in-session image-data store) adds images
+ * that are re-queryable by id but were never described: pass-through
+ * attachments on vision-capable base models and analyze_image results.
+ * Described images win on collision (they carry description text).
  */
 export function collectRecallCandidates(
 	descriptions: ReadonlyMap<string, string>,
 	metaLookup: (hash: string) => ImageMeta | undefined,
+	recallableHashes?: Iterable<string>,
 ): RecallCandidate[] {
-	return [...descriptions]
-		.reverse()
-		.map(([hash, description]) => ({ hash, description, filename: metaLookup(hash)?.filename }));
+	const out = new Map<string, RecallCandidate>();
+	for (const [hash, description] of [...descriptions].reverse()) {
+		out.set(hash, { hash, description, filename: metaLookup(hash)?.filename });
+	}
+	if (recallableHashes) {
+		for (const hash of [...recallableHashes].reverse()) {
+			if (!out.has(hash)) out.set(hash, { hash, description: undefined, filename: metaLookup(hash)?.filename });
+	}
+	}
+	return [...out.values()];
 }
 
 /** One-line dropdown snippet (no fence markers, single line, hard cap). */

package/extensions/vision-proxy.ts

@@ -2366,9 +2366,14 @@ export default function (pi: ExtensionAPI) {
 						const entries = ctx.sessionManager.getEntries();
 						const config = resolveConfig(entries, process.env, _fileConfig);
 						if (config.mode !== "off") {
+							// Candidates = described images ∪ everything retained in the
+							// in-memory recall store (pass-through attachments, analyze_image
+							// results) — anything analyze_image can resolve by id.
+							const sessionState = getSessionState(ctx);
 							const candidates = collectRecallCandidates(
 								findDescriptions(entries),
-								(hash) => getSessionState(ctx).imageMeta.get(hash),
+								(hash) => sessionState.imageMeta.get(hash),
+								sessionState.imageData.map.keys(),
 							);
 							const items = buildRecallItems(candidates, token.query);
 							if (items.length > 0) return { items, prefix: token.prefix };
@@ -3242,8 +3247,28 @@ export default function (pi: ExtensionAPI) {
 			}
 
 			if (!shouldStripImages(config, ctx.model)) {
-				// off, or fallback + model supports images → pass through unchanged
-				// But still inject video descriptions if we have them
+				// fallback + model supports images → pass through unchanged (mode
+				// "off" retains nothing: the extension is disabled by intent).
+				// Retain the attachments in the in-memory recall store (no vision
+				// call, no description, bytes were already in process as message
+				// attachments; the store is memory-only and LRU-bounded) so `#`
+				// autocomplete and analyze_image can re-query them by id later in
+				// the session. Path-detected images were already stored on read.
+				// storeImageData runs unconditionally so a re-attached image
+				// refreshes its LRU recency (attach A, B, A, C must evict B, not A).
+				if (config.mode !== "off") {
+					for (const raw of images) {
+						try {
+							const img = toPiAiImage(raw);
+							const hash = hashImageData(img.data);
+							if (!imageMeta.has(hash)) storeImageMeta(imageMeta, hash, img.data);
+							storeImageData(imageData, hash, img.data, img.mimeType);
+						} catch {
+							// Undecodable attachment — pass it through untouched.
+						}
+					}
+				}
+				// Still inject video descriptions if we have them
 				if (videoDescriptionFence) {
 					return {
 						systemPrompt:

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-multimodal-proxy",
-  "version": "1.22.1",
+  "version": "1.23.0",
   "description": "Media understanding for any model in Pi: video/audio/YouTube analysis, vision for text-only models, cost-routed vision, and session image recall.",
   "keywords": [
     "pi-package"