pi-multimodal-proxy 1.22.1 → 1.24.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -4,6 +4,18 @@ All notable changes to this project will be documented in this file.
4
4
 
5
5
  The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
6
6
 
7
+ ## [1.24.0] - 2026-10-11
8
+
9
+ ### Added
10
+
11
+ - **Interactive YouTube 403 recovery** — a 403 on the media fetch with no cookies configured no longer dead-ends in a hint telling you to type a slash command. The proxy now opens a picker listing only the browsers actually installed on your machine (profile-dir detection per platform: Windows/macOS/Linux); picking one persists `ytdlpCookiesFromBrowser` (same write path as `/multimodal-proxy ytdlp cookies <browser>`, so future sessions just work) and retries the download immediately. Declining, headless sessions (`ctx.hasUI` false), or no detected browser fall back to a one-line notice. yt-dlp failure notifications also no longer echo the full command line: `summarizeYtDlpError` reduces the dump to the actionable `ERROR: …` line, and a 403 that persists *with* cookies now points at the extractor-args knob instead of the cookies knob. Ten new tests (559 total, all green): per-platform profile candidates, env-var fallback, XDG honoring, browser-set containment, and error summarization.
12
+
13
+ ## [1.23.0] - 2026-10-10
14
+
15
+ ### Fixed
16
+
17
+ - **`#` image-recall autocomplete never fired on vision-capable base models** — two compounding defects, found live while dogfooding the 1.22.1 README claims. First, the dropdown drew candidates **only** from persisted `DescriptionEntry`s, so images analyzed via the `analyze_image` tool (which stores bytes in the in-session recall store but appends no description entry) and images retained in the store were invisible to it. Second, in the most common modern configuration (multimodal base model + default `fallback` mode) pasted/attached images pass through as pixels and entered **no** store at all — no description, no bytes, no meta — leaving the autocomplete permanently empty no matter what the user did. Fixes: `collectRecallCandidates` now unions described images with the recall store's keys (described entries win on collision, store-only candidates show the filename or hash-prefix label); and the `before_agent_start` pass-through branch retains attachment bytes+meta in the existing in-memory LRU store (no vision call, no description, no new persistence — the bytes were already in process as message attachments). Net effect: `#` and `analyze_image` recall now work in every mode except `off` (where the extension is disabled by intent and retains nothing), matching the README's "media memory" pillar. Two review fixes folded in (PR #40): retention is skipped entirely in `off` mode, and `storeImageData` runs unconditionally on re-attachment so the LRU refreshes recency (attach A, B, A, C evicts B, not A). Verified against pi 1.1.0's `AutocompleteItem`/`AutocompleteProvider` shapes (no API drift involved).
18
+
7
19
  ## [1.22.1] - 2026-10-10
8
20
 
9
21
  ### Changed
package/README.md CHANGED
@@ -7,7 +7,7 @@ Media understanding for [Pi](https://pi.dev): images, video, audio, and YouTube
7
7
  - **Video, audio & YouTube** — paste a URL (`youtube.com/watch?v=…`, `youtu.be/…`, `/shorts/…`) or a local file path and a multimodal model (default: Grok 4.3) transcribes speech with speaker diarization, describes visual scenes, reads on-screen text, and reasons about the content in a single call. Almost no chat model or agent ingests video natively: this is the package's deepest capability.
8
8
  - **Vision for text-only models** — a large slice of Pi's catalog (DeepSeek, GLM, Nemotron, Devstral, gpt-oss, …) still has no image input. When an image reaches such a model, the extension describes it via your configured vision model, persists the description in the session, and injects it into context. The model "sees" images across turns, cached by content hash.
9
9
  - **Cheaper vision for multimodal models** — select `Multimodal Auto` and media turns escalate natively to the vision model (real pixels, no text fence; prompt caches survive the routing) while text turns stay on your base model. Pin it with `/multimodal-proxy virtual-base` to keep the cheap model after media turns too: unpinned, a text turn reuses whatever physical model handled the turn before, which after a media turn is the vision model. Or proxy every image through a flash-priced describer with mode `always`.
10
- - **Media memory that outlives the turn** — every proxied image gets a stable `image="<id>"`, so the agent can re-query or crop an earlier screenshot without a re-attachment ("zoom into that screenshot from before"), `#` autocompletes image ids in the prompt, and descriptions survive compaction via a recall digest. `analyze_image` answers targeted questions with locally applied crops; `generate_image` creates images that flow straight back through the same pipeline. (Recall applies to described images: text-only base models or mode `always`. Images passed through natively to a vision-capable base model stay in context as pixels and are not duplicated into recall.)
10
+ - **Media memory that outlives the turn** — every image the agent can re-query gets a stable `image="<id>"`: described images (text-only base models, mode `always`, or explicit `analyze_image` calls) carry a description snippet, and pasted attachments on a vision-capable base model are retained in the recall store without a description. The agent can re-query or crop an earlier screenshot without a re-attachment ("zoom into that screenshot from before"), `#` autocompletes image ids in the prompt, and descriptions survive compaction via a recall digest. `analyze_image` answers targeted questions with locally applied crops; `generate_image` creates images that flow straight back through the same pipeline.
11
11
  - **Governed by default** — first-use data-egress consent per provider, persisted pre-consent lists, crops applied locally before upload, upload downscaling, decode-bomb protection, and recall bytes kept in memory only. See [Privacy & security](#privacy--security).
12
12
 
13
13
  Media routing is automatic: content sniffing and path detection feed a pipeline with transient-error retry and model fallback on the image path (the native xAI video path is one-shot), plus compact transcript rows for `analyze_image`/`generate_image` (Pi ≥ 1.0.1).
@@ -47,9 +47,10 @@ winget install yt-dlp.yt-dlp # Windows
47
47
 
48
48
  #### 403 Forbidden / download failures
49
49
 
50
- YouTube periodically rejects unauthenticated media fetches with `HTTP Error 403: Forbidden`, even on the latest `yt-dlp`. The proxy can't fix YouTube's backend, but you can satisfy the auth check:
50
+ YouTube periodically rejects unauthenticated media fetches with `HTTP Error 403: Forbidden`, even on the latest `yt-dlp`. The proxy can't fix YouTube's backend, but it can satisfy the auth check:
51
51
 
52
- - **Reuse a logged-in browser session** — `/multimodal-proxy ytdlp cookies chrome` (also `firefox`, `edge`, `brave`, `opera`, `safari`, `vivaldi`, `chromium`, `whale`). yt-dlp reads that browser's YouTube cookies. This fixes most 403s.
52
+ - **Interactive fix (default)** — when a download 403s and no cookies are configured, a picker offers the browsers actually installed on your machine. Pick one and the download retries immediately with that browser's YouTube cookies; the choice is persisted, so it only happens once.
53
+ - **Manual setup** — `/multimodal-proxy ytdlp cookies chrome` (also `firefox`, `edge`, `brave`, `opera`, `safari`, `vivaldi`, `chromium`, `whale`). Same effect as the picker, for headless setups or pre-configuration.
53
54
  - **Try alternate player clients** — `/multimodal-proxy ytdlp extractor-args "youtube:player_client=web_safari,web"`.
54
55
  - Show current settings with `/multimodal-proxy ytdlp`; turn either off with `... off`.
55
56
 
@@ -206,7 +207,10 @@ Every `<vision_proxy_description>`, `<vision_proxy_analysis>`, and
206
207
  can pass that id back to `analyze_image` (or `/multimodal-proxy describe`) to
207
208
  re-examine or crop an image the user shared earlier in the session — even once
208
209
  it is no longer attached to the current message (e.g. *"zoom into that
209
- screenshot from before"*). The image bytes are retained **in memory only** for
210
+ screenshot from before"*). Pasted attachments on a vision-capable base model
211
+ are also retained (no description; labeled by filename, or by hash prefix when
212
+ the paste carries no filename), so `#` recall works in every mode except `off`.
213
+ The image bytes are retained **in memory only** for
210
214
  the life of the session, never written to the session log or disk, and are
211
215
  evicted oldest-first once the recall budget (`PI_VISION_PROXY_IMAGE_RECALL_BYTES`,
212
216
  default 64 MB) is exceeded.
@@ -53,6 +53,8 @@ import {
53
53
  sanitizeAllowedFolders,
54
54
  sanitizeYtdlpCookiesFromBrowser,
55
55
  sanitizeYtdlpExtractorArgs,
56
+ browserProfileCandidates,
57
+ summarizeYtDlpError,
56
58
  sanitizeProviderHeaders,
57
59
  YTDLP_COOKIES_BROWSERS,
58
60
  pathAccessFromConfig,
@@ -348,6 +350,63 @@ describe("yt-dlp config sanitizers", () => {
348
350
  });
349
351
  });
350
352
 
353
+ describe("browserProfileCandidates", () => {
354
+ it("lists Windows profile dirs from LOCALAPPDATA/APPDATA, without safari", () => {
355
+ const env = { LOCALAPPDATA: join("C:", "Users", "u", "AppData", "Local"), APPDATA: join("C:", "Users", "u", "AppData", "Roaming") };
356
+ const map = new Map(browserProfileCandidates("win32", env, join("C:", "Users", "u")).map((c) => [c.browser, c.dirs]));
357
+ assert.ok(map.get("chrome")![0].endsWith(join("Google", "Chrome", "User Data")));
358
+ assert.ok(map.get("firefox")![0].endsWith(join("Mozilla", "Firefox", "Profiles")));
359
+ assert.ok(map.has("edge") && map.has("brave") && map.has("vivaldi") && map.has("chromium") && map.has("whale"));
360
+ assert.ok(!map.has("safari"));
361
+ });
362
+ it("falls back to homedir AppData when env vars are missing", () => {
363
+ const home = join("C:", "Users", "u");
364
+ const map = new Map(browserProfileCandidates("win32", {}, home).map((c) => [c.browser, c.dirs]));
365
+ assert.ok(map.get("chrome")![0].startsWith(join(home, "AppData", "Local")));
366
+ assert.ok(map.get("firefox")![0].startsWith(join(home, "AppData", "Roaming")));
367
+ });
368
+ it("lists macOS profile dirs including safari", () => {
369
+ const map = new Map(browserProfileCandidates("darwin", {}, "/Users/u").map((c) => [c.browser, c.dirs]));
370
+ assert.ok(map.get("firefox")![0].endsWith(join("Firefox", "Profiles")));
371
+ assert.ok(map.get("chrome")![0].endsWith(join("Google", "Chrome")));
372
+ assert.ok(map.get("safari")![0].endsWith("Safari.app"));
373
+ assert.ok(!map.has("whale"));
374
+ });
375
+ it("honors XDG_CONFIG_HOME on linux", () => {
376
+ const map = new Map(browserProfileCandidates("linux", { XDG_CONFIG_HOME: "/xdg" }, "/home/u").map((c) => [c.browser, c.dirs]));
377
+ assert.equal(map.get("chrome")![0], join("/xdg", "google-chrome"));
378
+ assert.equal(map.get("firefox")![0], join("/home/u", ".mozilla", "firefox"));
379
+ assert.ok(!map.has("safari"));
380
+ });
381
+ it("returns empty for unsupported platforms", () => {
382
+ assert.deepEqual(browserProfileCandidates("freebsd", {}, "/home/u"), []);
383
+ });
384
+ it("only yields browsers yt-dlp supports", () => {
385
+ for (const platform of ["win32", "darwin", "linux"] as const) {
386
+ for (const { browser } of browserProfileCandidates(platform, {}, "/home/u")) {
387
+ assert.ok(YTDLP_COOKIES_BROWSERS.has(browser), `${browser} on ${platform}`);
388
+ }
389
+ }
390
+ });
391
+ });
392
+
393
+ describe("summarizeYtDlpError", () => {
394
+ it("extracts the last ERROR line without the prefix", () => {
395
+ const msg = "Command failed: yt-dlp --no-playlist --no-warnings https://x\nWARNING: something\nERROR: unable to download video data: HTTP Error 403: Forbidden";
396
+ assert.equal(summarizeYtDlpError(msg), "unable to download video data: HTTP Error 403: Forbidden");
397
+ });
398
+ it("prefers the last ERROR line over earlier ones", () => {
399
+ const msg = "ERROR: first\nnoise\nERROR: second";
400
+ assert.equal(summarizeYtDlpError(msg), "second");
401
+ });
402
+ it("falls back to the last non-empty line", () => {
403
+ assert.equal(summarizeYtDlpError("Command failed: yt-dlp\n\nsome other failure\n"), "some other failure");
404
+ });
405
+ it("returns the message itself when it has no non-empty lines", () => {
406
+ assert.equal(summarizeYtDlpError(""), "");
407
+ });
408
+ });
409
+
351
410
  describe("selectVisionModels", () => {
352
411
  const img = (id: string) => ({ id, input: ["text", "image"] as readonly string[] });
353
412
  const txt = (id: string) => ({ id, input: ["text"] as readonly string[] });
@@ -64,6 +64,40 @@ describe("collectRecallCandidates", () => {
64
64
  { hash: "h1", description: "first image", filename: undefined },
65
65
  ]);
66
66
  });
67
+
68
+ it("merges store-only hashes (pass-through attachments) after described ones", () => {
69
+ const descriptions = new Map([["h1", "described earlier"]]);
70
+ const meta = new Map([["h3", { width: 8, height: 6, filename: "paste.png" }]]);
71
+ const out = collectRecallCandidates(descriptions, (h) => meta.get(h), ["h2", "h3"]);
72
+ assert.deepEqual(out, [
73
+ { hash: "h1", description: "described earlier", filename: undefined },
74
+ { hash: "h3", description: undefined, filename: "paste.png" }, // newest store entry first
75
+ { hash: "h2", description: undefined, filename: undefined },
76
+ ]);
77
+ });
78
+
79
+ it("descriptions win when a hash appears in both sources", () => {
80
+ const out = collectRecallCandidates(
81
+ new Map([["shared", "the description"]]),
82
+ () => undefined,
83
+ ["shared", "other"],
84
+ );
85
+ assert.equal(out.length, 2);
86
+ assert.equal(out[0]!.description, "the description");
87
+ assert.equal(out[1]!.hash, "other");
88
+ });
89
+
90
+ it("feeds store-only candidates into buildRecallItems labels", () => {
91
+ const hash = "c".repeat(32);
92
+ const candidates = collectRecallCandidates(new Map(), () => ({ width: 2, height: 2, filename: "shot.png" }), [hash]);
93
+ const items = buildRecallItems(candidates, "");
94
+ assert.equal(items.length, 1);
95
+ assert.equal(items[0]!.label, "shot.png");
96
+ assert.ok(items[0]!.value.endsWith(hash));
97
+ // no filename → hash-prefix label
98
+ const bare = buildRecallItems(collectRecallCandidates(new Map(), () => undefined, [hash]), "");
99
+ assert.equal(bare[0]!.label, `${hash.slice(0, 12)}…`);
100
+ });
67
101
  });
68
102
 
69
103
  describe("buildRecallItems", () => {
@@ -318,16 +318,28 @@ export function extractRecallToken(
318
318
  }
319
319
 
320
320
  /**
321
- * Collect recall candidates from persisted descriptions, most recent first.
321
+ * Collect recall candidates for the `#` autocomplete, most recent first.
322
322
  * `metaLookup` supplies in-memory filename/dimensions when still available.
323
+ * `recallableHashes` (keys of the in-session image-data store) adds images
324
+ * that are re-queryable by id but were never described: pass-through
325
+ * attachments on vision-capable base models and analyze_image results.
326
+ * Described images win on collision (they carry description text).
323
327
  */
324
328
  export function collectRecallCandidates(
325
329
  descriptions: ReadonlyMap<string, string>,
326
330
  metaLookup: (hash: string) => ImageMeta | undefined,
331
+ recallableHashes?: Iterable<string>,
327
332
  ): RecallCandidate[] {
328
- return [...descriptions]
329
- .reverse()
330
- .map(([hash, description]) => ({ hash, description, filename: metaLookup(hash)?.filename }));
333
+ const out = new Map<string, RecallCandidate>();
334
+ for (const [hash, description] of [...descriptions].reverse()) {
335
+ out.set(hash, { hash, description, filename: metaLookup(hash)?.filename });
336
+ }
337
+ if (recallableHashes) {
338
+ for (const hash of [...recallableHashes].reverse()) {
339
+ if (!out.has(hash)) out.set(hash, { hash, description: undefined, filename: metaLookup(hash)?.filename });
340
+ }
341
+ }
342
+ return [...out.values()];
331
343
  }
332
344
 
333
345
  /** One-line dropdown snippet (no fence markers, single line, hard cap). */
@@ -1151,6 +1163,68 @@ export function sanitizeYtdlpExtractorArgs(value: unknown): string {
1151
1163
  return value.replace(/[\x00-\x1f\x7f]/g, "").trim().slice(0, YTDLP_EXTRACTOR_ARGS_MAX);
1152
1164
  }
1153
1165
 
1166
+ /**
1167
+ * Well-known profile-dir candidates per browser, per platform (1.24.0). Pure:
1168
+ * the caller does the fs checks and keeps only browsers whose dir exists.
1169
+ * Drives the interactive 403 picker so it offers browsers that can actually
1170
+ * work instead of listing all nine yt-dlp values blindly.
1171
+ */
1172
+ export function browserProfileCandidates(
1173
+ platform: NodeJS.Platform,
1174
+ env: NodeJS.ProcessEnv,
1175
+ homedir: string,
1176
+ ): Array<{ browser: string; dirs: string[] }> {
1177
+ const out: Array<{ browser: string; dirs: string[] }> = [];
1178
+ const push = (browser: string, dirs: string[]) => out.push({ browser, dirs });
1179
+ if (platform === "win32") {
1180
+ const local = env.LOCALAPPDATA ?? join(homedir, "AppData", "Local");
1181
+ const roaming = env.APPDATA ?? join(homedir, "AppData", "Roaming");
1182
+ push("chrome", [join(local, "Google", "Chrome", "User Data")]);
1183
+ push("firefox", [join(roaming, "Mozilla", "Firefox", "Profiles")]);
1184
+ push("edge", [join(local, "Microsoft", "Edge", "User Data")]);
1185
+ push("brave", [join(local, "BraveSoftware", "Brave-Browser", "User Data")]);
1186
+ push("opera", [join(roaming, "Opera Software", "Opera Stable")]);
1187
+ push("vivaldi", [join(local, "Vivaldi", "User Data")]);
1188
+ push("chromium", [join(local, "Chromium", "User Data")]);
1189
+ push("whale", [join(local, "Naver", "Naver Whale", "User Data")]);
1190
+ } else if (platform === "darwin") {
1191
+ const appSupport = join(homedir, "Library", "Application Support");
1192
+ push("chrome", [join(appSupport, "Google", "Chrome")]);
1193
+ push("firefox", [join(appSupport, "Firefox", "Profiles")]);
1194
+ push("edge", [join(appSupport, "Microsoft Edge")]);
1195
+ push("brave", [join(appSupport, "BraveSoftware", "Brave-Browser")]);
1196
+ push("opera", [join(appSupport, "com.operasoftware.Opera")]);
1197
+ push("vivaldi", [join(appSupport, "Vivaldi")]);
1198
+ push("chromium", [join(appSupport, "Chromium")]);
1199
+ push("safari", [join("/", "Applications", "Safari.app")]);
1200
+ } else if (platform === "linux") {
1201
+ const config = env.XDG_CONFIG_HOME ?? join(homedir, ".config");
1202
+ push("chrome", [join(config, "google-chrome")]);
1203
+ push("firefox", [join(homedir, ".mozilla", "firefox")]);
1204
+ push("edge", [join(config, "microsoft-edge")]);
1205
+ push("brave", [join(config, "BraveSoftware", "Brave-Browser")]);
1206
+ push("opera", [join(config, "opera")]);
1207
+ push("vivaldi", [join(config, "vivaldi")]);
1208
+ push("chromium", [join(config, "chromium")]);
1209
+ push("whale", [join(config, "naver-whale")]);
1210
+ }
1211
+ return out;
1212
+ }
1213
+
1214
+ /**
1215
+ * Reduce an execFile failure dump to the single actionable yt-dlp line: the
1216
+ * last `ERROR: …` from stderr, else the last non-empty line. Keeps user-facing
1217
+ * notifications to one line instead of echoing the full command (1.24.0).
1218
+ */
1219
+ export function summarizeYtDlpError(message: string): string {
1220
+ const lines = message.split(/\r?\n/).map((l) => l.trim()).filter((l) => l.length > 0);
1221
+ for (let i = lines.length - 1; i >= 0; i--) {
1222
+ const m = /^ERROR:\s*(.+)$/i.exec(lines[i]);
1223
+ if (m) return m[1].trim();
1224
+ }
1225
+ return lines[lines.length - 1] ?? message;
1226
+ }
1227
+
1154
1228
  /**
1155
1229
  * Drop `null` header values from a `ProviderHeaders` map. Since pi 0.84.0,
1156
1230
  * `ModelRegistry.getApiKeyAndHeaders()` returns `Record<string, string | null>`
@@ -254,6 +254,7 @@ import {
254
254
  buildVideoDescriptionFence,
255
255
  buildVideoEmptyResponseError,
256
256
  buildVideoProxySection,
257
+ browserProfileCandidates,
257
258
  extractXaiResponsesText,
258
259
  formatXaiSttTranscript,
259
260
  analyzeDetailsForTranscript,
@@ -321,6 +322,7 @@ import {
321
322
  isUncPath,
322
323
  MAX_ALLOWED_FOLDERS,
323
324
  sanitizeYtdlpExtractorArgs,
325
+ summarizeYtDlpError,
324
326
  YTDLP_COOKIES_BROWSERS,
325
327
 
326
328
  readMediaFileWithReason,
@@ -1448,6 +1450,65 @@ interface DownloadedMedia {
1448
1450
  title: string;
1449
1451
  }
1450
1452
 
1453
+ /**
1454
+ * Browsers from yt-dlp's --cookies-from-browser set whose profile dir exists
1455
+ * on this machine (1.24.0). Drives the interactive 403 picker so it only
1456
+ * offers browsers that can actually work.
1457
+ */
1458
+ async function detectCookieBrowsers(): Promise<string[]> {
1459
+ const found: string[] = [];
1460
+ for (const { browser, dirs } of browserProfileCandidates(os.platform(), process.env, os.homedir())) {
1461
+ for (const d of dirs) {
1462
+ if (await pathExists(d)) {
1463
+ found.push(browser);
1464
+ break;
1465
+ }
1466
+ }
1467
+ }
1468
+ return found;
1469
+ }
1470
+
1471
+ /**
1472
+ * Persist a single config patch outside the slash-command handler (1.24.0).
1473
+ * Mirrors commandHandler's writePersisted: sanitize → session entry → file,
1474
+ * keeping allowedProviders file-scoped. Used by the interactive 403 flow.
1475
+ */
1476
+ function persistConfigPatch(pi: ExtensionAPI, ctx: ExtensionContext, patch: Partial<VisionConfig>): void {
1477
+ const validated = sanitize({ ...persistedBase(ctx.sessionManager.getEntries()), ...patch });
1478
+ delete validated.allowedProviders;
1479
+ pi.appendEntry(CUSTOM_TYPE_CONFIG, validated);
1480
+ const fileNext: Partial<VisionConfig> = { ...validated };
1481
+ if (_fileConfig.allowedProviders !== undefined) {
1482
+ fileNext.allowedProviders = normalizeAllowedProviders(_fileConfig.allowedProviders) ?? [];
1483
+ }
1484
+ writePersistentFile(fileNext);
1485
+ _fileConfig = fileNext;
1486
+ }
1487
+
1488
+ /**
1489
+ * Interactive half of the 403 flow (1.24.0): offer the installed browsers,
1490
+ * persist the pick (same as `/multimodal-proxy ytdlp cookies <browser>`), and
1491
+ * return it so the caller retries. Returns null when declined, when no UI is
1492
+ * available, or when no supported browser is detected.
1493
+ */
1494
+ async function promptForCookieBrowser(pi: ExtensionAPI, ctx: ExtensionContext): Promise<string | null> {
1495
+ if (!ctx.hasUI) return null;
1496
+ const browsers = await detectCookieBrowsers();
1497
+ if (browsers.length === 0) return null;
1498
+ const CANCEL = "Don't retry";
1499
+ const choice = await ctx.ui.select(
1500
+ "YouTube blocked the download (403 — it wants a logged-in session). Retry with cookies from:",
1501
+ [...browsers, CANCEL],
1502
+ );
1503
+ if (!choice || choice === CANCEL || !YTDLP_COOKIES_BROWSERS.has(choice)) return null;
1504
+ persistConfigPatch(pi, ctx, { ytdlpCookiesFromBrowser: choice });
1505
+ ctx.ui.notify(
1506
+ `[multimodal-proxy] Saved: yt-dlp will reuse ${choice} cookies from now on (change via /multimodal-proxy ytdlp cookies). Retrying…`,
1507
+ "info",
1508
+ );
1509
+ return choice;
1510
+ }
1511
+
1451
1512
  /**
1452
1513
  * Download a media URL via yt-dlp into a fresh temp dir.
1453
1514
  *
@@ -1464,6 +1525,7 @@ async function downloadMediaUrl(
1464
1525
  signal: AbortSignal | undefined,
1465
1526
  ctx: ExtensionContext,
1466
1527
  ytdlp?: { cookiesFromBrowser?: string; extractorArgs?: string },
1528
+ hooks?: { onForbidden?: () => Promise<string | null> },
1467
1529
  ): Promise<DownloadedMedia | null> {
1468
1530
  if (!(await checkYtDlp())) {
1469
1531
  ctx.ui.notify(
@@ -1524,12 +1586,32 @@ async function downloadMediaUrl(
1524
1586
  } catch (err) {
1525
1587
  if (signal?.aborted) return null;
1526
1588
  const msg = err instanceof Error ? err.message : String(err);
1527
- // A 403 on the media fetch usually means YouTube wants a logged-in
1528
- // session; point the user at the cookies knob if it isn't already set.
1529
- const hint = /403|forbidden/i.test(msg) && !ytdlp?.cookiesFromBrowser
1530
- ? " (a 403 often means YouTube wants a logged-in session — try /multimodal-proxy ytdlp cookies <browser>)"
1589
+ const forbidden = /403|forbidden/i.test(msg);
1590
+ // A 403 without cookies means YouTube wants a logged-in session. Offer to
1591
+ // reuse a browser's cookies interactively (1.24.0) and retry once with
1592
+ // them; the hook also persists the choice so later downloads just work.
1593
+ if (forbidden && !ytdlp?.cookiesFromBrowser && hooks?.onForbidden) {
1594
+ let browser: string | null = null;
1595
+ try {
1596
+ browser = await hooks.onForbidden();
1597
+ } catch {
1598
+ browser = null;
1599
+ }
1600
+ if (signal?.aborted) return null;
1601
+ if (browser) {
1602
+ return downloadMediaUrl(url, signal, ctx, { ...ytdlp, cookiesFromBrowser: browser });
1603
+ }
1604
+ // Declined (or nothing to offer): keep the notice to one line.
1605
+ ctx.ui.notify(`[multimodal-proxy] YouTube download failed: ${summarizeYtDlpError(msg)}`, "warning");
1606
+ return null;
1607
+ }
1608
+ // Point the user at the right knob depending on what is already set.
1609
+ const hint = forbidden
1610
+ ? ytdlp?.cookiesFromBrowser
1611
+ ? " (still blocked with cookies — try /multimodal-proxy ytdlp extractor-args \"youtube:player_client=android\")"
1612
+ : " (a 403 often means YouTube wants a logged-in session — try /multimodal-proxy ytdlp cookies <browser>)"
1531
1613
  : "";
1532
- ctx.ui.notify(`[multimodal-proxy] YouTube download failed for ${url}: ${msg}${hint}`, "warning");
1614
+ ctx.ui.notify(`[multimodal-proxy] YouTube download failed for ${url}: ${summarizeYtDlpError(msg)}${hint}`, "warning");
1533
1615
  return null;
1534
1616
  }
1535
1617
  }
@@ -2366,9 +2448,14 @@ export default function (pi: ExtensionAPI) {
2366
2448
  const entries = ctx.sessionManager.getEntries();
2367
2449
  const config = resolveConfig(entries, process.env, _fileConfig);
2368
2450
  if (config.mode !== "off") {
2451
+ // Candidates = described images ∪ everything retained in the
2452
+ // in-memory recall store (pass-through attachments, analyze_image
2453
+ // results) — anything analyze_image can resolve by id.
2454
+ const sessionState = getSessionState(ctx);
2369
2455
  const candidates = collectRecallCandidates(
2370
2456
  findDescriptions(entries),
2371
- (hash) => getSessionState(ctx).imageMeta.get(hash),
2457
+ (hash) => sessionState.imageMeta.get(hash),
2458
+ sessionState.imageData.map.keys(),
2372
2459
  );
2373
2460
  const items = buildRecallItems(candidates, token.query);
2374
2461
  if (items.length > 0) return { items, prefix: token.prefix };
@@ -3086,6 +3173,10 @@ export default function (pi: ExtensionAPI) {
3086
3173
  () => downloadMediaUrl(watch, ctx.signal, ctx, {
3087
3174
  cookiesFromBrowser: config.ytdlpCookiesFromBrowser,
3088
3175
  extractorArgs: config.ytdlpExtractorArgs,
3176
+ }, {
3177
+ // 403 without cookies → offer to reuse an installed browser's
3178
+ // session, persist the pick, and retry (1.24.0).
3179
+ onForbidden: () => promptForCookieBrowser(pi, ctx),
3089
3180
  }),
3090
3181
  );
3091
3182
  if (!dl) continue;
@@ -3242,8 +3333,28 @@ export default function (pi: ExtensionAPI) {
3242
3333
  }
3243
3334
 
3244
3335
  if (!shouldStripImages(config, ctx.model)) {
3245
- // off, or fallback + model supports images → pass through unchanged
3246
- // But still inject video descriptions if we have them
3336
+ // fallback + model supports images → pass through unchanged (mode
3337
+ // "off" retains nothing: the extension is disabled by intent).
3338
+ // Retain the attachments in the in-memory recall store (no vision
3339
+ // call, no description, bytes were already in process as message
3340
+ // attachments; the store is memory-only and LRU-bounded) so `#`
3341
+ // autocomplete and analyze_image can re-query them by id later in
3342
+ // the session. Path-detected images were already stored on read.
3343
+ // storeImageData runs unconditionally so a re-attached image
3344
+ // refreshes its LRU recency (attach A, B, A, C must evict B, not A).
3345
+ if (config.mode !== "off") {
3346
+ for (const raw of images) {
3347
+ try {
3348
+ const img = toPiAiImage(raw);
3349
+ const hash = hashImageData(img.data);
3350
+ if (!imageMeta.has(hash)) storeImageMeta(imageMeta, hash, img.data);
3351
+ storeImageData(imageData, hash, img.data, img.mimeType);
3352
+ } catch {
3353
+ // Undecodable attachment — pass it through untouched.
3354
+ }
3355
+ }
3356
+ }
3357
+ // Still inject video descriptions if we have them
3247
3358
  if (videoDescriptionFence) {
3248
3359
  return {
3249
3360
  systemPrompt:
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "pi-multimodal-proxy",
3
- "version": "1.22.1",
3
+ "version": "1.24.0",
4
4
  "description": "Media understanding for any model in Pi: video/audio/YouTube analysis, vision for text-only models, cost-routed vision, and session image recall.",
5
5
  "keywords": [
6
6
  "pi-package"