pi-multimodal-proxy 1.22.1 → 1.24.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,18 @@ All notable changes to this project will be documented in this file.
|
|
|
4
4
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
|
6
6
|
|
|
7
|
+
## [1.24.0] - 2026-10-11
|
|
8
|
+
|
|
9
|
+
### Added
|
|
10
|
+
|
|
11
|
+
- **Interactive YouTube 403 recovery** — a 403 on the media fetch with no cookies configured no longer dead-ends in a hint telling you to type a slash command. The proxy now opens a picker listing only the browsers actually installed on your machine (profile-dir detection per platform: Windows/macOS/Linux); picking one persists `ytdlpCookiesFromBrowser` (same write path as `/multimodal-proxy ytdlp cookies <browser>`, so future sessions just work) and retries the download immediately. Declining, headless sessions (`ctx.hasUI` false), or no detected browser fall back to a one-line notice. yt-dlp failure notifications also no longer echo the full command line: `summarizeYtDlpError` reduces the dump to the actionable `ERROR: …` line, and a 403 that persists *with* cookies now points at the extractor-args knob instead of the cookies knob. Ten new tests (559 total, all green): per-platform profile candidates, env-var fallback, XDG honoring, browser-set containment, and error summarization.
|
|
12
|
+
|
|
13
|
+
## [1.23.0] - 2026-10-10
|
|
14
|
+
|
|
15
|
+
### Fixed
|
|
16
|
+
|
|
17
|
+
- **`#` image-recall autocomplete never fired on vision-capable base models** — two compounding defects, found live while dogfooding the 1.22.1 README claims. First, the dropdown drew candidates **only** from persisted `DescriptionEntry`s, so images analyzed via the `analyze_image` tool (which stores bytes in the in-session recall store but appends no description entry) and images retained in the store were invisible to it. Second, in the most common modern configuration (multimodal base model + default `fallback` mode) pasted/attached images pass through as pixels and entered **no** store at all — no description, no bytes, no meta — leaving the autocomplete permanently empty no matter what the user did. Fixes: `collectRecallCandidates` now unions described images with the recall store's keys (described entries win on collision, store-only candidates show the filename or hash-prefix label); and the `before_agent_start` pass-through branch retains attachment bytes+meta in the existing in-memory LRU store (no vision call, no description, no new persistence — the bytes were already in process as message attachments). Net effect: `#` and `analyze_image` recall now work in every mode except `off` (where the extension is disabled by intent and retains nothing), matching the README's "media memory" pillar. Two review fixes folded in (PR #40): retention is skipped entirely in `off` mode, and `storeImageData` runs unconditionally on re-attachment so the LRU refreshes recency (attach A, B, A, C evicts B, not A). Verified against pi 1.1.0's `AutocompleteItem`/`AutocompleteProvider` shapes (no API drift involved).
|
|
18
|
+
|
|
7
19
|
## [1.22.1] - 2026-10-10
|
|
8
20
|
|
|
9
21
|
### Changed
|
package/README.md
CHANGED
|
@@ -7,7 +7,7 @@ Media understanding for [Pi](https://pi.dev): images, video, audio, and YouTube
|
|
|
7
7
|
- **Video, audio & YouTube** — paste a URL (`youtube.com/watch?v=…`, `youtu.be/…`, `/shorts/…`) or a local file path and a multimodal model (default: Grok 4.3) transcribes speech with speaker diarization, describes visual scenes, reads on-screen text, and reasons about the content in a single call. Almost no chat model or agent ingests video natively: this is the package's deepest capability.
|
|
8
8
|
- **Vision for text-only models** — a large slice of Pi's catalog (DeepSeek, GLM, Nemotron, Devstral, gpt-oss, …) still has no image input. When an image reaches such a model, the extension describes it via your configured vision model, persists the description in the session, and injects it into context. The model "sees" images across turns, cached by content hash.
|
|
9
9
|
- **Cheaper vision for multimodal models** — select `Multimodal Auto` and media turns escalate natively to the vision model (real pixels, no text fence; prompt caches survive the routing) while text turns stay on your base model. Pin it with `/multimodal-proxy virtual-base` to keep the cheap model after media turns too: unpinned, a text turn reuses whatever physical model handled the turn before, which after a media turn is the vision model. Or proxy every image through a flash-priced describer with mode `always`.
|
|
10
|
-
- **Media memory that outlives the turn** — every
|
|
10
|
+
- **Media memory that outlives the turn** — every image the agent can re-query gets a stable `image="<id>"`: described images (text-only base models, mode `always`, or explicit `analyze_image` calls) carry a description snippet, and pasted attachments on a vision-capable base model are retained in the recall store without a description. The agent can re-query or crop an earlier screenshot without a re-attachment ("zoom into that screenshot from before"), `#` autocompletes image ids in the prompt, and descriptions survive compaction via a recall digest. `analyze_image` answers targeted questions with locally applied crops; `generate_image` creates images that flow straight back through the same pipeline.
|
|
11
11
|
- **Governed by default** — first-use data-egress consent per provider, persisted pre-consent lists, crops applied locally before upload, upload downscaling, decode-bomb protection, and recall bytes kept in memory only. See [Privacy & security](#privacy--security).
|
|
12
12
|
|
|
13
13
|
Media routing is automatic: content sniffing and path detection feed a pipeline with transient-error retry and model fallback on the image path (the native xAI video path is one-shot), plus compact transcript rows for `analyze_image`/`generate_image` (Pi ≥ 1.0.1).
|
|
@@ -47,9 +47,10 @@ winget install yt-dlp.yt-dlp # Windows
|
|
|
47
47
|
|
|
48
48
|
#### 403 Forbidden / download failures
|
|
49
49
|
|
|
50
|
-
YouTube periodically rejects unauthenticated media fetches with `HTTP Error 403: Forbidden`, even on the latest `yt-dlp`. The proxy can't fix YouTube's backend, but
|
|
50
|
+
YouTube periodically rejects unauthenticated media fetches with `HTTP Error 403: Forbidden`, even on the latest `yt-dlp`. The proxy can't fix YouTube's backend, but it can satisfy the auth check:
|
|
51
51
|
|
|
52
|
-
- **
|
|
52
|
+
- **Interactive fix (default)** — when a download 403s and no cookies are configured, a picker offers the browsers actually installed on your machine. Pick one and the download retries immediately with that browser's YouTube cookies; the choice is persisted, so it only happens once.
|
|
53
|
+
- **Manual setup** — `/multimodal-proxy ytdlp cookies chrome` (also `firefox`, `edge`, `brave`, `opera`, `safari`, `vivaldi`, `chromium`, `whale`). Same effect as the picker, for headless setups or pre-configuration.
|
|
53
54
|
- **Try alternate player clients** — `/multimodal-proxy ytdlp extractor-args "youtube:player_client=web_safari,web"`.
|
|
54
55
|
- Show current settings with `/multimodal-proxy ytdlp`; turn either off with `... off`.
|
|
55
56
|
|
|
@@ -206,7 +207,10 @@ Every `<vision_proxy_description>`, `<vision_proxy_analysis>`, and
|
|
|
206
207
|
can pass that id back to `analyze_image` (or `/multimodal-proxy describe`) to
|
|
207
208
|
re-examine or crop an image the user shared earlier in the session — even once
|
|
208
209
|
it is no longer attached to the current message (e.g. *"zoom into that
|
|
209
|
-
screenshot from before"*).
|
|
210
|
+
screenshot from before"*). Pasted attachments on a vision-capable base model
|
|
211
|
+
are also retained (no description; labeled by filename, or by hash prefix when
|
|
212
|
+
the paste carries no filename), so `#` recall works in every mode except `off`.
|
|
213
|
+
The image bytes are retained **in memory only** for
|
|
210
214
|
the life of the session, never written to the session log or disk, and are
|
|
211
215
|
evicted oldest-first once the recall budget (`PI_VISION_PROXY_IMAGE_RECALL_BYTES`,
|
|
212
216
|
default 64 MB) is exceeded.
|
|
@@ -53,6 +53,8 @@ import {
|
|
|
53
53
|
sanitizeAllowedFolders,
|
|
54
54
|
sanitizeYtdlpCookiesFromBrowser,
|
|
55
55
|
sanitizeYtdlpExtractorArgs,
|
|
56
|
+
browserProfileCandidates,
|
|
57
|
+
summarizeYtDlpError,
|
|
56
58
|
sanitizeProviderHeaders,
|
|
57
59
|
YTDLP_COOKIES_BROWSERS,
|
|
58
60
|
pathAccessFromConfig,
|
|
@@ -348,6 +350,63 @@ describe("yt-dlp config sanitizers", () => {
|
|
|
348
350
|
});
|
|
349
351
|
});
|
|
350
352
|
|
|
353
|
+
describe("browserProfileCandidates", () => {
|
|
354
|
+
it("lists Windows profile dirs from LOCALAPPDATA/APPDATA, without safari", () => {
|
|
355
|
+
const env = { LOCALAPPDATA: join("C:", "Users", "u", "AppData", "Local"), APPDATA: join("C:", "Users", "u", "AppData", "Roaming") };
|
|
356
|
+
const map = new Map(browserProfileCandidates("win32", env, join("C:", "Users", "u")).map((c) => [c.browser, c.dirs]));
|
|
357
|
+
assert.ok(map.get("chrome")![0].endsWith(join("Google", "Chrome", "User Data")));
|
|
358
|
+
assert.ok(map.get("firefox")![0].endsWith(join("Mozilla", "Firefox", "Profiles")));
|
|
359
|
+
assert.ok(map.has("edge") && map.has("brave") && map.has("vivaldi") && map.has("chromium") && map.has("whale"));
|
|
360
|
+
assert.ok(!map.has("safari"));
|
|
361
|
+
});
|
|
362
|
+
it("falls back to homedir AppData when env vars are missing", () => {
|
|
363
|
+
const home = join("C:", "Users", "u");
|
|
364
|
+
const map = new Map(browserProfileCandidates("win32", {}, home).map((c) => [c.browser, c.dirs]));
|
|
365
|
+
assert.ok(map.get("chrome")![0].startsWith(join(home, "AppData", "Local")));
|
|
366
|
+
assert.ok(map.get("firefox")![0].startsWith(join(home, "AppData", "Roaming")));
|
|
367
|
+
});
|
|
368
|
+
it("lists macOS profile dirs including safari", () => {
|
|
369
|
+
const map = new Map(browserProfileCandidates("darwin", {}, "/Users/u").map((c) => [c.browser, c.dirs]));
|
|
370
|
+
assert.ok(map.get("firefox")![0].endsWith(join("Firefox", "Profiles")));
|
|
371
|
+
assert.ok(map.get("chrome")![0].endsWith(join("Google", "Chrome")));
|
|
372
|
+
assert.ok(map.get("safari")![0].endsWith("Safari.app"));
|
|
373
|
+
assert.ok(!map.has("whale"));
|
|
374
|
+
});
|
|
375
|
+
it("honors XDG_CONFIG_HOME on linux", () => {
|
|
376
|
+
const map = new Map(browserProfileCandidates("linux", { XDG_CONFIG_HOME: "/xdg" }, "/home/u").map((c) => [c.browser, c.dirs]));
|
|
377
|
+
assert.equal(map.get("chrome")![0], join("/xdg", "google-chrome"));
|
|
378
|
+
assert.equal(map.get("firefox")![0], join("/home/u", ".mozilla", "firefox"));
|
|
379
|
+
assert.ok(!map.has("safari"));
|
|
380
|
+
});
|
|
381
|
+
it("returns empty for unsupported platforms", () => {
|
|
382
|
+
assert.deepEqual(browserProfileCandidates("freebsd", {}, "/home/u"), []);
|
|
383
|
+
});
|
|
384
|
+
it("only yields browsers yt-dlp supports", () => {
|
|
385
|
+
for (const platform of ["win32", "darwin", "linux"] as const) {
|
|
386
|
+
for (const { browser } of browserProfileCandidates(platform, {}, "/home/u")) {
|
|
387
|
+
assert.ok(YTDLP_COOKIES_BROWSERS.has(browser), `${browser} on ${platform}`);
|
|
388
|
+
}
|
|
389
|
+
}
|
|
390
|
+
});
|
|
391
|
+
});
|
|
392
|
+
|
|
393
|
+
describe("summarizeYtDlpError", () => {
|
|
394
|
+
it("extracts the last ERROR line without the prefix", () => {
|
|
395
|
+
const msg = "Command failed: yt-dlp --no-playlist --no-warnings https://x\nWARNING: something\nERROR: unable to download video data: HTTP Error 403: Forbidden";
|
|
396
|
+
assert.equal(summarizeYtDlpError(msg), "unable to download video data: HTTP Error 403: Forbidden");
|
|
397
|
+
});
|
|
398
|
+
it("prefers the last ERROR line over earlier ones", () => {
|
|
399
|
+
const msg = "ERROR: first\nnoise\nERROR: second";
|
|
400
|
+
assert.equal(summarizeYtDlpError(msg), "second");
|
|
401
|
+
});
|
|
402
|
+
it("falls back to the last non-empty line", () => {
|
|
403
|
+
assert.equal(summarizeYtDlpError("Command failed: yt-dlp\n\nsome other failure\n"), "some other failure");
|
|
404
|
+
});
|
|
405
|
+
it("returns the message itself when it has no non-empty lines", () => {
|
|
406
|
+
assert.equal(summarizeYtDlpError(""), "");
|
|
407
|
+
});
|
|
408
|
+
});
|
|
409
|
+
|
|
351
410
|
describe("selectVisionModels", () => {
|
|
352
411
|
const img = (id: string) => ({ id, input: ["text", "image"] as readonly string[] });
|
|
353
412
|
const txt = (id: string) => ({ id, input: ["text"] as readonly string[] });
|
|
@@ -64,6 +64,40 @@ describe("collectRecallCandidates", () => {
|
|
|
64
64
|
{ hash: "h1", description: "first image", filename: undefined },
|
|
65
65
|
]);
|
|
66
66
|
});
|
|
67
|
+
|
|
68
|
+
it("merges store-only hashes (pass-through attachments) after described ones", () => {
|
|
69
|
+
const descriptions = new Map([["h1", "described earlier"]]);
|
|
70
|
+
const meta = new Map([["h3", { width: 8, height: 6, filename: "paste.png" }]]);
|
|
71
|
+
const out = collectRecallCandidates(descriptions, (h) => meta.get(h), ["h2", "h3"]);
|
|
72
|
+
assert.deepEqual(out, [
|
|
73
|
+
{ hash: "h1", description: "described earlier", filename: undefined },
|
|
74
|
+
{ hash: "h3", description: undefined, filename: "paste.png" }, // newest store entry first
|
|
75
|
+
{ hash: "h2", description: undefined, filename: undefined },
|
|
76
|
+
]);
|
|
77
|
+
});
|
|
78
|
+
|
|
79
|
+
it("descriptions win when a hash appears in both sources", () => {
|
|
80
|
+
const out = collectRecallCandidates(
|
|
81
|
+
new Map([["shared", "the description"]]),
|
|
82
|
+
() => undefined,
|
|
83
|
+
["shared", "other"],
|
|
84
|
+
);
|
|
85
|
+
assert.equal(out.length, 2);
|
|
86
|
+
assert.equal(out[0]!.description, "the description");
|
|
87
|
+
assert.equal(out[1]!.hash, "other");
|
|
88
|
+
});
|
|
89
|
+
|
|
90
|
+
it("feeds store-only candidates into buildRecallItems labels", () => {
|
|
91
|
+
const hash = "c".repeat(32);
|
|
92
|
+
const candidates = collectRecallCandidates(new Map(), () => ({ width: 2, height: 2, filename: "shot.png" }), [hash]);
|
|
93
|
+
const items = buildRecallItems(candidates, "");
|
|
94
|
+
assert.equal(items.length, 1);
|
|
95
|
+
assert.equal(items[0]!.label, "shot.png");
|
|
96
|
+
assert.ok(items[0]!.value.endsWith(hash));
|
|
97
|
+
// no filename → hash-prefix label
|
|
98
|
+
const bare = buildRecallItems(collectRecallCandidates(new Map(), () => undefined, [hash]), "");
|
|
99
|
+
assert.equal(bare[0]!.label, `${hash.slice(0, 12)}…`);
|
|
100
|
+
});
|
|
67
101
|
});
|
|
68
102
|
|
|
69
103
|
describe("buildRecallItems", () => {
|
package/extensions/internal.ts
CHANGED
|
@@ -318,16 +318,28 @@ export function extractRecallToken(
|
|
|
318
318
|
}
|
|
319
319
|
|
|
320
320
|
/**
|
|
321
|
-
* Collect recall candidates
|
|
321
|
+
* Collect recall candidates for the `#` autocomplete, most recent first.
|
|
322
322
|
* `metaLookup` supplies in-memory filename/dimensions when still available.
|
|
323
|
+
* `recallableHashes` (keys of the in-session image-data store) adds images
|
|
324
|
+
* that are re-queryable by id but were never described: pass-through
|
|
325
|
+
* attachments on vision-capable base models and analyze_image results.
|
|
326
|
+
* Described images win on collision (they carry description text).
|
|
323
327
|
*/
|
|
324
328
|
export function collectRecallCandidates(
|
|
325
329
|
descriptions: ReadonlyMap<string, string>,
|
|
326
330
|
metaLookup: (hash: string) => ImageMeta | undefined,
|
|
331
|
+
recallableHashes?: Iterable<string>,
|
|
327
332
|
): RecallCandidate[] {
|
|
328
|
-
|
|
329
|
-
|
|
330
|
-
.
|
|
333
|
+
const out = new Map<string, RecallCandidate>();
|
|
334
|
+
for (const [hash, description] of [...descriptions].reverse()) {
|
|
335
|
+
out.set(hash, { hash, description, filename: metaLookup(hash)?.filename });
|
|
336
|
+
}
|
|
337
|
+
if (recallableHashes) {
|
|
338
|
+
for (const hash of [...recallableHashes].reverse()) {
|
|
339
|
+
if (!out.has(hash)) out.set(hash, { hash, description: undefined, filename: metaLookup(hash)?.filename });
|
|
340
|
+
}
|
|
341
|
+
}
|
|
342
|
+
return [...out.values()];
|
|
331
343
|
}
|
|
332
344
|
|
|
333
345
|
/** One-line dropdown snippet (no fence markers, single line, hard cap). */
|
|
@@ -1151,6 +1163,68 @@ export function sanitizeYtdlpExtractorArgs(value: unknown): string {
|
|
|
1151
1163
|
return value.replace(/[\x00-\x1f\x7f]/g, "").trim().slice(0, YTDLP_EXTRACTOR_ARGS_MAX);
|
|
1152
1164
|
}
|
|
1153
1165
|
|
|
1166
|
+
/**
|
|
1167
|
+
* Well-known profile-dir candidates per browser, per platform (1.24.0). Pure:
|
|
1168
|
+
* the caller does the fs checks and keeps only browsers whose dir exists.
|
|
1169
|
+
* Drives the interactive 403 picker so it offers browsers that can actually
|
|
1170
|
+
* work instead of listing all nine yt-dlp values blindly.
|
|
1171
|
+
*/
|
|
1172
|
+
export function browserProfileCandidates(
|
|
1173
|
+
platform: NodeJS.Platform,
|
|
1174
|
+
env: NodeJS.ProcessEnv,
|
|
1175
|
+
homedir: string,
|
|
1176
|
+
): Array<{ browser: string; dirs: string[] }> {
|
|
1177
|
+
const out: Array<{ browser: string; dirs: string[] }> = [];
|
|
1178
|
+
const push = (browser: string, dirs: string[]) => out.push({ browser, dirs });
|
|
1179
|
+
if (platform === "win32") {
|
|
1180
|
+
const local = env.LOCALAPPDATA ?? join(homedir, "AppData", "Local");
|
|
1181
|
+
const roaming = env.APPDATA ?? join(homedir, "AppData", "Roaming");
|
|
1182
|
+
push("chrome", [join(local, "Google", "Chrome", "User Data")]);
|
|
1183
|
+
push("firefox", [join(roaming, "Mozilla", "Firefox", "Profiles")]);
|
|
1184
|
+
push("edge", [join(local, "Microsoft", "Edge", "User Data")]);
|
|
1185
|
+
push("brave", [join(local, "BraveSoftware", "Brave-Browser", "User Data")]);
|
|
1186
|
+
push("opera", [join(roaming, "Opera Software", "Opera Stable")]);
|
|
1187
|
+
push("vivaldi", [join(local, "Vivaldi", "User Data")]);
|
|
1188
|
+
push("chromium", [join(local, "Chromium", "User Data")]);
|
|
1189
|
+
push("whale", [join(local, "Naver", "Naver Whale", "User Data")]);
|
|
1190
|
+
} else if (platform === "darwin") {
|
|
1191
|
+
const appSupport = join(homedir, "Library", "Application Support");
|
|
1192
|
+
push("chrome", [join(appSupport, "Google", "Chrome")]);
|
|
1193
|
+
push("firefox", [join(appSupport, "Firefox", "Profiles")]);
|
|
1194
|
+
push("edge", [join(appSupport, "Microsoft Edge")]);
|
|
1195
|
+
push("brave", [join(appSupport, "BraveSoftware", "Brave-Browser")]);
|
|
1196
|
+
push("opera", [join(appSupport, "com.operasoftware.Opera")]);
|
|
1197
|
+
push("vivaldi", [join(appSupport, "Vivaldi")]);
|
|
1198
|
+
push("chromium", [join(appSupport, "Chromium")]);
|
|
1199
|
+
push("safari", [join("/", "Applications", "Safari.app")]);
|
|
1200
|
+
} else if (platform === "linux") {
|
|
1201
|
+
const config = env.XDG_CONFIG_HOME ?? join(homedir, ".config");
|
|
1202
|
+
push("chrome", [join(config, "google-chrome")]);
|
|
1203
|
+
push("firefox", [join(homedir, ".mozilla", "firefox")]);
|
|
1204
|
+
push("edge", [join(config, "microsoft-edge")]);
|
|
1205
|
+
push("brave", [join(config, "BraveSoftware", "Brave-Browser")]);
|
|
1206
|
+
push("opera", [join(config, "opera")]);
|
|
1207
|
+
push("vivaldi", [join(config, "vivaldi")]);
|
|
1208
|
+
push("chromium", [join(config, "chromium")]);
|
|
1209
|
+
push("whale", [join(config, "naver-whale")]);
|
|
1210
|
+
}
|
|
1211
|
+
return out;
|
|
1212
|
+
}
|
|
1213
|
+
|
|
1214
|
+
/**
|
|
1215
|
+
* Reduce an execFile failure dump to the single actionable yt-dlp line: the
|
|
1216
|
+
* last `ERROR: …` from stderr, else the last non-empty line. Keeps user-facing
|
|
1217
|
+
* notifications to one line instead of echoing the full command (1.24.0).
|
|
1218
|
+
*/
|
|
1219
|
+
export function summarizeYtDlpError(message: string): string {
|
|
1220
|
+
const lines = message.split(/\r?\n/).map((l) => l.trim()).filter((l) => l.length > 0);
|
|
1221
|
+
for (let i = lines.length - 1; i >= 0; i--) {
|
|
1222
|
+
const m = /^ERROR:\s*(.+)$/i.exec(lines[i]);
|
|
1223
|
+
if (m) return m[1].trim();
|
|
1224
|
+
}
|
|
1225
|
+
return lines[lines.length - 1] ?? message;
|
|
1226
|
+
}
|
|
1227
|
+
|
|
1154
1228
|
/**
|
|
1155
1229
|
* Drop `null` header values from a `ProviderHeaders` map. Since pi 0.84.0,
|
|
1156
1230
|
* `ModelRegistry.getApiKeyAndHeaders()` returns `Record<string, string | null>`
|
|
@@ -254,6 +254,7 @@ import {
|
|
|
254
254
|
buildVideoDescriptionFence,
|
|
255
255
|
buildVideoEmptyResponseError,
|
|
256
256
|
buildVideoProxySection,
|
|
257
|
+
browserProfileCandidates,
|
|
257
258
|
extractXaiResponsesText,
|
|
258
259
|
formatXaiSttTranscript,
|
|
259
260
|
analyzeDetailsForTranscript,
|
|
@@ -321,6 +322,7 @@ import {
|
|
|
321
322
|
isUncPath,
|
|
322
323
|
MAX_ALLOWED_FOLDERS,
|
|
323
324
|
sanitizeYtdlpExtractorArgs,
|
|
325
|
+
summarizeYtDlpError,
|
|
324
326
|
YTDLP_COOKIES_BROWSERS,
|
|
325
327
|
|
|
326
328
|
readMediaFileWithReason,
|
|
@@ -1448,6 +1450,65 @@ interface DownloadedMedia {
|
|
|
1448
1450
|
title: string;
|
|
1449
1451
|
}
|
|
1450
1452
|
|
|
1453
|
+
/**
|
|
1454
|
+
* Browsers from yt-dlp's --cookies-from-browser set whose profile dir exists
|
|
1455
|
+
* on this machine (1.24.0). Drives the interactive 403 picker so it only
|
|
1456
|
+
* offers browsers that can actually work.
|
|
1457
|
+
*/
|
|
1458
|
+
async function detectCookieBrowsers(): Promise<string[]> {
|
|
1459
|
+
const found: string[] = [];
|
|
1460
|
+
for (const { browser, dirs } of browserProfileCandidates(os.platform(), process.env, os.homedir())) {
|
|
1461
|
+
for (const d of dirs) {
|
|
1462
|
+
if (await pathExists(d)) {
|
|
1463
|
+
found.push(browser);
|
|
1464
|
+
break;
|
|
1465
|
+
}
|
|
1466
|
+
}
|
|
1467
|
+
}
|
|
1468
|
+
return found;
|
|
1469
|
+
}
|
|
1470
|
+
|
|
1471
|
+
/**
|
|
1472
|
+
* Persist a single config patch outside the slash-command handler (1.24.0).
|
|
1473
|
+
* Mirrors commandHandler's writePersisted: sanitize → session entry → file,
|
|
1474
|
+
* keeping allowedProviders file-scoped. Used by the interactive 403 flow.
|
|
1475
|
+
*/
|
|
1476
|
+
function persistConfigPatch(pi: ExtensionAPI, ctx: ExtensionContext, patch: Partial<VisionConfig>): void {
|
|
1477
|
+
const validated = sanitize({ ...persistedBase(ctx.sessionManager.getEntries()), ...patch });
|
|
1478
|
+
delete validated.allowedProviders;
|
|
1479
|
+
pi.appendEntry(CUSTOM_TYPE_CONFIG, validated);
|
|
1480
|
+
const fileNext: Partial<VisionConfig> = { ...validated };
|
|
1481
|
+
if (_fileConfig.allowedProviders !== undefined) {
|
|
1482
|
+
fileNext.allowedProviders = normalizeAllowedProviders(_fileConfig.allowedProviders) ?? [];
|
|
1483
|
+
}
|
|
1484
|
+
writePersistentFile(fileNext);
|
|
1485
|
+
_fileConfig = fileNext;
|
|
1486
|
+
}
|
|
1487
|
+
|
|
1488
|
+
/**
|
|
1489
|
+
* Interactive half of the 403 flow (1.24.0): offer the installed browsers,
|
|
1490
|
+
* persist the pick (same as `/multimodal-proxy ytdlp cookies <browser>`), and
|
|
1491
|
+
* return it so the caller retries. Returns null when declined, when no UI is
|
|
1492
|
+
* available, or when no supported browser is detected.
|
|
1493
|
+
*/
|
|
1494
|
+
async function promptForCookieBrowser(pi: ExtensionAPI, ctx: ExtensionContext): Promise<string | null> {
|
|
1495
|
+
if (!ctx.hasUI) return null;
|
|
1496
|
+
const browsers = await detectCookieBrowsers();
|
|
1497
|
+
if (browsers.length === 0) return null;
|
|
1498
|
+
const CANCEL = "Don't retry";
|
|
1499
|
+
const choice = await ctx.ui.select(
|
|
1500
|
+
"YouTube blocked the download (403 — it wants a logged-in session). Retry with cookies from:",
|
|
1501
|
+
[...browsers, CANCEL],
|
|
1502
|
+
);
|
|
1503
|
+
if (!choice || choice === CANCEL || !YTDLP_COOKIES_BROWSERS.has(choice)) return null;
|
|
1504
|
+
persistConfigPatch(pi, ctx, { ytdlpCookiesFromBrowser: choice });
|
|
1505
|
+
ctx.ui.notify(
|
|
1506
|
+
`[multimodal-proxy] Saved: yt-dlp will reuse ${choice} cookies from now on (change via /multimodal-proxy ytdlp cookies). Retrying…`,
|
|
1507
|
+
"info",
|
|
1508
|
+
);
|
|
1509
|
+
return choice;
|
|
1510
|
+
}
|
|
1511
|
+
|
|
1451
1512
|
/**
|
|
1452
1513
|
* Download a media URL via yt-dlp into a fresh temp dir.
|
|
1453
1514
|
*
|
|
@@ -1464,6 +1525,7 @@ async function downloadMediaUrl(
|
|
|
1464
1525
|
signal: AbortSignal | undefined,
|
|
1465
1526
|
ctx: ExtensionContext,
|
|
1466
1527
|
ytdlp?: { cookiesFromBrowser?: string; extractorArgs?: string },
|
|
1528
|
+
hooks?: { onForbidden?: () => Promise<string | null> },
|
|
1467
1529
|
): Promise<DownloadedMedia | null> {
|
|
1468
1530
|
if (!(await checkYtDlp())) {
|
|
1469
1531
|
ctx.ui.notify(
|
|
@@ -1524,12 +1586,32 @@ async function downloadMediaUrl(
|
|
|
1524
1586
|
} catch (err) {
|
|
1525
1587
|
if (signal?.aborted) return null;
|
|
1526
1588
|
const msg = err instanceof Error ? err.message : String(err);
|
|
1527
|
-
|
|
1528
|
-
//
|
|
1529
|
-
|
|
1530
|
-
|
|
1589
|
+
const forbidden = /403|forbidden/i.test(msg);
|
|
1590
|
+
// A 403 without cookies means YouTube wants a logged-in session. Offer to
|
|
1591
|
+
// reuse a browser's cookies interactively (1.24.0) and retry once with
|
|
1592
|
+
// them; the hook also persists the choice so later downloads just work.
|
|
1593
|
+
if (forbidden && !ytdlp?.cookiesFromBrowser && hooks?.onForbidden) {
|
|
1594
|
+
let browser: string | null = null;
|
|
1595
|
+
try {
|
|
1596
|
+
browser = await hooks.onForbidden();
|
|
1597
|
+
} catch {
|
|
1598
|
+
browser = null;
|
|
1599
|
+
}
|
|
1600
|
+
if (signal?.aborted) return null;
|
|
1601
|
+
if (browser) {
|
|
1602
|
+
return downloadMediaUrl(url, signal, ctx, { ...ytdlp, cookiesFromBrowser: browser });
|
|
1603
|
+
}
|
|
1604
|
+
// Declined (or nothing to offer): keep the notice to one line.
|
|
1605
|
+
ctx.ui.notify(`[multimodal-proxy] YouTube download failed: ${summarizeYtDlpError(msg)}`, "warning");
|
|
1606
|
+
return null;
|
|
1607
|
+
}
|
|
1608
|
+
// Point the user at the right knob depending on what is already set.
|
|
1609
|
+
const hint = forbidden
|
|
1610
|
+
? ytdlp?.cookiesFromBrowser
|
|
1611
|
+
? " (still blocked with cookies — try /multimodal-proxy ytdlp extractor-args \"youtube:player_client=android\")"
|
|
1612
|
+
: " (a 403 often means YouTube wants a logged-in session — try /multimodal-proxy ytdlp cookies <browser>)"
|
|
1531
1613
|
: "";
|
|
1532
|
-
ctx.ui.notify(`[multimodal-proxy] YouTube download failed for ${url}: ${msg}${hint}`, "warning");
|
|
1614
|
+
ctx.ui.notify(`[multimodal-proxy] YouTube download failed for ${url}: ${summarizeYtDlpError(msg)}${hint}`, "warning");
|
|
1533
1615
|
return null;
|
|
1534
1616
|
}
|
|
1535
1617
|
}
|
|
@@ -2366,9 +2448,14 @@ export default function (pi: ExtensionAPI) {
|
|
|
2366
2448
|
const entries = ctx.sessionManager.getEntries();
|
|
2367
2449
|
const config = resolveConfig(entries, process.env, _fileConfig);
|
|
2368
2450
|
if (config.mode !== "off") {
|
|
2451
|
+
// Candidates = described images ∪ everything retained in the
|
|
2452
|
+
// in-memory recall store (pass-through attachments, analyze_image
|
|
2453
|
+
// results) — anything analyze_image can resolve by id.
|
|
2454
|
+
const sessionState = getSessionState(ctx);
|
|
2369
2455
|
const candidates = collectRecallCandidates(
|
|
2370
2456
|
findDescriptions(entries),
|
|
2371
|
-
(hash) =>
|
|
2457
|
+
(hash) => sessionState.imageMeta.get(hash),
|
|
2458
|
+
sessionState.imageData.map.keys(),
|
|
2372
2459
|
);
|
|
2373
2460
|
const items = buildRecallItems(candidates, token.query);
|
|
2374
2461
|
if (items.length > 0) return { items, prefix: token.prefix };
|
|
@@ -3086,6 +3173,10 @@ export default function (pi: ExtensionAPI) {
|
|
|
3086
3173
|
() => downloadMediaUrl(watch, ctx.signal, ctx, {
|
|
3087
3174
|
cookiesFromBrowser: config.ytdlpCookiesFromBrowser,
|
|
3088
3175
|
extractorArgs: config.ytdlpExtractorArgs,
|
|
3176
|
+
}, {
|
|
3177
|
+
// 403 without cookies → offer to reuse an installed browser's
|
|
3178
|
+
// session, persist the pick, and retry (1.24.0).
|
|
3179
|
+
onForbidden: () => promptForCookieBrowser(pi, ctx),
|
|
3089
3180
|
}),
|
|
3090
3181
|
);
|
|
3091
3182
|
if (!dl) continue;
|
|
@@ -3242,8 +3333,28 @@ export default function (pi: ExtensionAPI) {
|
|
|
3242
3333
|
}
|
|
3243
3334
|
|
|
3244
3335
|
if (!shouldStripImages(config, ctx.model)) {
|
|
3245
|
-
//
|
|
3246
|
-
//
|
|
3336
|
+
// fallback + model supports images → pass through unchanged (mode
|
|
3337
|
+
// "off" retains nothing: the extension is disabled by intent).
|
|
3338
|
+
// Retain the attachments in the in-memory recall store (no vision
|
|
3339
|
+
// call, no description, bytes were already in process as message
|
|
3340
|
+
// attachments; the store is memory-only and LRU-bounded) so `#`
|
|
3341
|
+
// autocomplete and analyze_image can re-query them by id later in
|
|
3342
|
+
// the session. Path-detected images were already stored on read.
|
|
3343
|
+
// storeImageData runs unconditionally so a re-attached image
|
|
3344
|
+
// refreshes its LRU recency (attach A, B, A, C must evict B, not A).
|
|
3345
|
+
if (config.mode !== "off") {
|
|
3346
|
+
for (const raw of images) {
|
|
3347
|
+
try {
|
|
3348
|
+
const img = toPiAiImage(raw);
|
|
3349
|
+
const hash = hashImageData(img.data);
|
|
3350
|
+
if (!imageMeta.has(hash)) storeImageMeta(imageMeta, hash, img.data);
|
|
3351
|
+
storeImageData(imageData, hash, img.data, img.mimeType);
|
|
3352
|
+
} catch {
|
|
3353
|
+
// Undecodable attachment — pass it through untouched.
|
|
3354
|
+
}
|
|
3355
|
+
}
|
|
3356
|
+
}
|
|
3357
|
+
// Still inject video descriptions if we have them
|
|
3247
3358
|
if (videoDescriptionFence) {
|
|
3248
3359
|
return {
|
|
3249
3360
|
systemPrompt:
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-multimodal-proxy",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.24.0",
|
|
4
4
|
"description": "Media understanding for any model in Pi: video/audio/YouTube analysis, vision for text-only models, cost-routed vision, and session image recall.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"pi-package"
|