pi-multimodal-proxy 1.12.0 → 1.12.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +10 -0
- package/README.md +15 -0
- package/extensions/__tests__/internal.test.ts +32 -3
- package/extensions/internal.ts +48 -1
- package/extensions/vision-proxy.ts +177 -20
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,16 @@ All notable changes to this project will be documented in this file.
|
|
|
4
4
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
|
6
6
|
|
|
7
|
+
## [1.12.1] - 2026-08-07
|
|
8
|
+
|
|
9
|
+
### Fixed
|
|
10
|
+
|
|
11
|
+
- **Downloaded videos now save to the real Downloads folder.** The post-analysis "Save downloaded video?" prompt hardcoded `join(os.homedir(), "Downloads")` and wrapped `copyFile` in a silent `catch {}`. On machines where the Windows Downloads folder is relocated (e.g. `D:\Downloads`) or OneDrive-redirected, the default path doesn't exist, so `copyFile` threw `ENOENT`, the error was swallowed, and the user saw no file and no message after clicking **Yes**. `vision-proxy.ts` now resolves the real Downloads folder via the Windows Shell known-folder API (`FOLDERID_Downloads` / `shell:Downloads`, which respects relocation and OneDrive redirection), creates it on demand (`mkdir -p`), and surfaces any save failure with the target path. The success notification now includes the full saved path. New helpers: `resolveDownloadsDir`, `pathExists`.
|
|
12
|
+
|
|
13
|
+
### Added
|
|
14
|
+
|
|
15
|
+
- **yt-dlp auth knobs to defeat YouTube 403s.** YouTube increasingly returns `HTTP Error 403: Forbidden` on the media fetch even with the latest `yt-dlp`; the proxy previously passed no credentials. Two opt-in, persisted settings now forward to yt-dlp: `ytdlpCookiesFromBrowser` (`/multimodal-proxy ytdlp cookies <browser|off>`, validated against `chrome|firefox|edge|brave|opera|safari|vivaldi|chromium|whale`) reuses a logged-in browser session, and `ytdlpExtractorArgs` (`/multimodal-proxy ytdlp extractor-args "<text>|off>`) forwards arbitrary `--extractor-args` (e.g. `youtube:player_client=web_safari,web`). A 403 download failure now appends a hint pointing at the cookies knob. Env overrides: `PI_VISION_PROXY_YTDLP_COOKIES_FROM_BROWSER`, `PI_VISION_PROXY_YTDLP_EXTRACTOR_ARGS`. New sanitizers + tests: `sanitizeYtdlpCookiesFromBrowser`, `sanitizeYtdlpExtractorArgs`, `YTDLP_COOKIES_BROWSERS`.
|
|
16
|
+
|
|
7
17
|
## [1.11.0] - 2026-07-29
|
|
8
18
|
|
|
9
19
|
### Added
|
package/README.md
CHANGED
|
@@ -8,6 +8,11 @@ When **video or audio files** are detected, they are routed to a **multimodal mo
|
|
|
8
8
|
|
|
9
9
|
**YouTube links** are detected too: paste a URL (`youtube.com/watch?v=…`, `youtu.be/…`, `/shorts/…`, etc.) and the video is downloaded with [`yt-dlp`](https://github.com/yt-dlp/yt-dlp) and analyzed exactly like a local file.
|
|
10
10
|
|
|
11
|
+
## What's new in 1.12.1
|
|
12
|
+
|
|
13
|
+
- **Fix: downloaded videos now save to your real Downloads folder.** The "Save downloaded video?" prompt hardcoded `~/Downloads` and wrapped the copy in a silent `catch`. If you relocated Downloads (e.g. to `D:\Downloads`) or use OneDrive redirection, that path doesn't exist — so clicking **Yes** silently failed with no file and no error. The proxy now resolves the real Downloads folder via the Windows Shell known-folder API (`FOLDERID_Downloads`, respects relocation + OneDrive), creates it if missing, and surfaces any save error with the target path. The success toast now shows the full saved path.
|
|
14
|
+
- **Configurable yt-dlp auth (defeats YouTube 403s).** YouTube increasingly rejects unauthenticated media fetches with `HTTP 403`, even on the latest `yt-dlp`. Two opt-in knobs satisfy the check: `/multimodal-proxy ytdlp cookies <browser>` reuses a logged-in browser session (`chrome`, `firefox`, `edge`, `brave`, …), and `/multimodal-proxy ytdlp extractor-args "<args>"` forwards arbitrary `--extractor-args` (e.g. `youtube:player_client=web_safari,web`). A 403 failure now also hints at the cookies knob. Env overrides: `PI_VISION_PROXY_YTDLP_COOKIES_FROM_BROWSER`, `PI_VISION_PROXY_YTDLP_EXTRACTOR_ARGS`.
|
|
15
|
+
|
|
11
16
|
## What's new in 1.12.0
|
|
12
17
|
|
|
13
18
|
- **YouTube video download** — paste a YouTube URL in your prompt and the extension downloads it via `yt-dlp` and analyzes it like any local video file. Gated by path detection (on by default); requires `yt-dlp` on your PATH. See [YouTube videos](#youtube-videos). After successful analysis you are prompted to optionally keep the downloaded file in your Downloads folder.
|
|
@@ -76,6 +81,16 @@ winget install yt-dlp.yt-dlp # Windows
|
|
|
76
81
|
|
|
77
82
|
`ffmpeg` is also used (for duration probing and stream merging) — it usually ships alongside yt-dlp. Downloads are capped to ≤720p and rejected past the configured size limit (`PI_VISION_PROXY_MAX_VIDEO_BYTES`, default 200 MB). To disable URL auto-download, turn off path detection: `/multimodal-proxy path-detection off`.
|
|
78
83
|
|
|
84
|
+
#### 403 Forbidden / download failures
|
|
85
|
+
|
|
86
|
+
YouTube periodically rejects unauthenticated media fetches with `HTTP Error 403: Forbidden`, even on the latest `yt-dlp`. The proxy can't fix YouTube's backend, but you can satisfy the auth check:
|
|
87
|
+
|
|
88
|
+
- **Reuse a logged-in browser session** — `/multimodal-proxy ytdlp cookies chrome` (also `firefox`, `edge`, `brave`, `opera`, `safari`, `vivaldi`, `chromium`, `whale`). yt-dlp reads that browser's YouTube cookies. This fixes most 403s.
|
|
89
|
+
- **Try alternate player clients** — `/multimodal-proxy ytdlp extractor-args "youtube:player_client=web_safari,web"`.
|
|
90
|
+
- Show current settings with `/multimodal-proxy ytdlp`; turn either off with `... off`.
|
|
91
|
+
|
|
92
|
+
Both are persisted in `~/.pi/agent/multimodal-proxy.json` and can be set via env (`PI_VISION_PROXY_YTDLP_COOKIES_FROM_BROWSER`, `PI_VISION_PROXY_YTDLP_EXTRACTOR_ARGS`), which override and lock the command.
|
|
93
|
+
|
|
79
94
|
## Modes
|
|
80
95
|
|
|
81
96
|
| Mode | Behavior |
|
|
@@ -51,6 +51,9 @@ import {
|
|
|
51
51
|
isValidNamedRegion,
|
|
52
52
|
expandLeadingTilde,
|
|
53
53
|
sanitizeAllowedFolders,
|
|
54
|
+
sanitizeYtdlpCookiesFromBrowser,
|
|
55
|
+
sanitizeYtdlpExtractorArgs,
|
|
56
|
+
YTDLP_COOKIES_BROWSERS,
|
|
54
57
|
pathAccessFromConfig,
|
|
55
58
|
MAX_ALLOWED_FOLDERS,
|
|
56
59
|
LRUCache,
|
|
@@ -265,18 +268,18 @@ describe("readEnvOverrides", () => {
|
|
|
265
268
|
|
|
266
269
|
describe("envFlags", () => {
|
|
267
270
|
it("reports presence per variable", () => {
|
|
268
|
-
assert.deepEqual(envFlags({}), { mode: false, model: false, context: false, tool: false, maxImagesPerCall: false, maxBatch: false, cacheSize: false, videoModel: false, allowedProviders: false, allowHome: false, allowedFolders: false, statusLine: false, pathDetection: false });
|
|
271
|
+
assert.deepEqual(envFlags({}), { mode: false, model: false, context: false, tool: false, maxImagesPerCall: false, maxBatch: false, cacheSize: false, videoModel: false, allowedProviders: false, allowHome: false, allowedFolders: false, statusLine: false, pathDetection: false, ytdlpCookies: false, ytdlpExtractorArgs: false });
|
|
269
272
|
assert.deepEqual(
|
|
270
273
|
envFlags({
|
|
271
274
|
PI_VISION_PROXY_MODE: "x",
|
|
272
275
|
PI_VISION_PROXY_MODEL: "y",
|
|
273
276
|
PI_VISION_PROXY_INCLUDE_CONTEXT: "",
|
|
274
277
|
}),
|
|
275
|
-
{ mode: true, model: true, context: true, tool: false, maxImagesPerCall: false, maxBatch: false, cacheSize: false, videoModel: false, allowedProviders: false, allowHome: false, allowedFolders: false, statusLine: false, pathDetection: false },
|
|
278
|
+
{ mode: true, model: true, context: true, tool: false, maxImagesPerCall: false, maxBatch: false, cacheSize: false, videoModel: false, allowedProviders: false, allowHome: false, allowedFolders: false, statusLine: false, pathDetection: false, ytdlpCookies: false, ytdlpExtractorArgs: false },
|
|
276
279
|
);
|
|
277
280
|
assert.deepEqual(
|
|
278
281
|
envFlags({ PI_VISION_PROXY_ALLOW_HOME: "1", PI_VISION_PROXY_ALLOWED_FOLDERS: "/a" }),
|
|
279
|
-
{ mode: false, model: false, context: false, tool: false, maxImagesPerCall: false, maxBatch: false, cacheSize: false, videoModel: false, allowedProviders: false, allowHome: true, allowedFolders: true, statusLine: false, pathDetection: false },
|
|
282
|
+
{ mode: false, model: false, context: false, tool: false, maxImagesPerCall: false, maxBatch: false, cacheSize: false, videoModel: false, allowedProviders: false, allowHome: true, allowedFolders: true, statusLine: false, pathDetection: false, ytdlpCookies: false, ytdlpExtractorArgs: false },
|
|
280
283
|
);
|
|
281
284
|
assert.equal(envFlags({ PI_VISION_PROXY_ALLOWED_PROVIDERS: "" }).allowedProviders, true);
|
|
282
285
|
// An unrecognized ALLOW_HOME value is not an override and must not lock the command
|
|
@@ -287,6 +290,32 @@ describe("envFlags", () => {
|
|
|
287
290
|
// An invalid value is not applied by readEnvOverrides, so it must not
|
|
288
291
|
// report (and thereby lock) the setting as env-overridden either.
|
|
289
292
|
assert.equal(envFlags({ PI_VISION_PROXY_STATUS_LINE: "bogus" }).statusLine, false);
|
|
293
|
+
assert.equal(envFlags({ PI_VISION_PROXY_YTDLP_COOKIES_FROM_BROWSER: "chrome" }).ytdlpCookies, true);
|
|
294
|
+
assert.equal(envFlags({ PI_VISION_PROXY_YTDLP_EXTRACTOR_ARGS: "youtube:player_client=web" }).ytdlpExtractorArgs, true);
|
|
295
|
+
});
|
|
296
|
+
});
|
|
297
|
+
|
|
298
|
+
describe("yt-dlp config sanitizers", () => {
|
|
299
|
+
it("accepts known browsers case-insensitively, rejects others", () => {
|
|
300
|
+
assert.equal(sanitizeYtdlpCookiesFromBrowser("Chrome"), "chrome");
|
|
301
|
+
assert.equal(sanitizeYtdlpCookiesFromBrowser("FIREFOX"), "firefox");
|
|
302
|
+
assert.equal(sanitizeYtdlpCookiesFromBrowser("edge"), "edge");
|
|
303
|
+
assert.equal(sanitizeYtdlpCookiesFromBrowser("internet explorer"), "");
|
|
304
|
+
assert.equal(sanitizeYtdlpCookiesFromBrowser(""), "");
|
|
305
|
+
assert.equal(sanitizeYtdlpCookiesFromBrowser(undefined), "");
|
|
306
|
+
assert.equal(sanitizeYtdlpCookiesFromBrowser(123 as any), "");
|
|
307
|
+
});
|
|
308
|
+
it("exposes the supported browser set", () => {
|
|
309
|
+
assert.ok(YTDLP_COOKIES_BROWSERS.has("chrome"));
|
|
310
|
+
assert.ok(YTDLP_COOKIES_BROWSERS.has("brave"));
|
|
311
|
+
});
|
|
312
|
+
it("strips control chars and caps extractor-args length", () => {
|
|
313
|
+
assert.equal(sanitizeYtdlpExtractorArgs("youtube:player_client=web"), "youtube:player_client=web");
|
|
314
|
+
assert.equal(sanitizeYtdlpExtractorArgs(" trim me "), "trim me");
|
|
315
|
+
assert.equal(sanitizeYtdlpExtractorArgs("a\x00b\x07c"), "abc");
|
|
316
|
+
assert.equal(sanitizeYtdlpExtractorArgs(""), "");
|
|
317
|
+
const long = "x".repeat(600);
|
|
318
|
+
assert.equal(sanitizeYtdlpExtractorArgs(long).length, 500);
|
|
290
319
|
});
|
|
291
320
|
});
|
|
292
321
|
|
package/extensions/internal.ts
CHANGED
|
@@ -79,6 +79,12 @@ export interface VisionConfig {
|
|
|
79
79
|
// (event.images) are always processed; only the convenience path-scan is
|
|
80
80
|
// gated. Explicit references via /multimodal-proxy describe still work.
|
|
81
81
|
pathDetection: ToolSetting;
|
|
82
|
+
// 1.12.1 — yt-dlp tuning to defeat YouTube 403s on the media fetch. Both
|
|
83
|
+
// empty by default. cookiesFromBrowser reuses a logged-in browser session
|
|
84
|
+
// (e.g. "chrome"); extractorArgs forwards arbitrary --extractor-args (e.g.
|
|
85
|
+
// "youtube:player_client=web_safari,web").
|
|
86
|
+
ytdlpCookiesFromBrowser: string;
|
|
87
|
+
ytdlpExtractorArgs: string;
|
|
82
88
|
}
|
|
83
89
|
|
|
84
90
|
export interface ImageMeta {
|
|
@@ -704,6 +710,8 @@ export const DEFAULT_CONFIG: VisionConfig = {
|
|
|
704
710
|
"google/gemini-2.5-pro": { format: "gemini_normalized_1000" },
|
|
705
711
|
"google/gemini-3-pro": { format: "gemini_normalized_1000" },
|
|
706
712
|
},
|
|
713
|
+
ytdlpCookiesFromBrowser: "",
|
|
714
|
+
ytdlpExtractorArgs: "",
|
|
707
715
|
};
|
|
708
716
|
|
|
709
717
|
// ── Persistent file storage ────────────────────────────────────────────────
|
|
@@ -729,6 +737,8 @@ const PERSISTED_CONFIG_KEYS = new Set([
|
|
|
729
737
|
"allowedFolders", "allowHome",
|
|
730
738
|
"statusLine",
|
|
731
739
|
"pathDetection",
|
|
740
|
+
"ytdlpCookiesFromBrowser",
|
|
741
|
+
"ytdlpExtractorArgs",
|
|
732
742
|
]);
|
|
733
743
|
|
|
734
744
|
/** Read config from the persistent file. Returns empty object on any failure. */
|
|
@@ -868,10 +878,15 @@ export function readEnvOverrides(env: NodeJS.ProcessEnv = process.env): Partial<
|
|
|
868
878
|
if (foldersEnv !== undefined) {
|
|
869
879
|
overrides.allowedFolders = sanitizeAllowedFolders(foldersEnv.split(delimiter));
|
|
870
880
|
}
|
|
881
|
+
// 1.12.1 yt-dlp env overrides
|
|
882
|
+
const cookiesEnv = env.PI_VISION_PROXY_YTDLP_COOKIES_FROM_BROWSER;
|
|
883
|
+
if (cookiesEnv !== undefined) overrides.ytdlpCookiesFromBrowser = sanitizeYtdlpCookiesFromBrowser(cookiesEnv);
|
|
884
|
+
const extractorArgsEnv = env.PI_VISION_PROXY_YTDLP_EXTRACTOR_ARGS;
|
|
885
|
+
if (extractorArgsEnv !== undefined) overrides.ytdlpExtractorArgs = sanitizeYtdlpExtractorArgs(extractorArgsEnv);
|
|
871
886
|
return overrides;
|
|
872
887
|
}
|
|
873
888
|
|
|
874
|
-
export function envFlags(env: NodeJS.ProcessEnv = process.env): { mode: boolean; model: boolean; context: boolean; tool: boolean; maxImagesPerCall: boolean; maxBatch: boolean; cacheSize: boolean; videoModel: boolean; allowedProviders: boolean; allowHome: boolean; allowedFolders: boolean; statusLine: boolean; pathDetection: boolean } {
|
|
889
|
+
export function envFlags(env: NodeJS.ProcessEnv = process.env): { mode: boolean; model: boolean; context: boolean; tool: boolean; maxImagesPerCall: boolean; maxBatch: boolean; cacheSize: boolean; videoModel: boolean; allowedProviders: boolean; allowHome: boolean; allowedFolders: boolean; statusLine: boolean; pathDetection: boolean; ytdlpCookies: boolean; ytdlpExtractorArgs: boolean } {
|
|
875
890
|
return {
|
|
876
891
|
mode: Boolean(env.PI_VISION_PROXY_MODE),
|
|
877
892
|
model: Boolean(env.PI_VISION_PROXY_MODEL),
|
|
@@ -891,6 +906,8 @@ export function envFlags(env: NodeJS.ProcessEnv = process.env): { mode: boolean;
|
|
|
891
906
|
statusLine: env.PI_VISION_PROXY_STATUS_LINE === "on" || env.PI_VISION_PROXY_STATUS_LINE === "off",
|
|
892
907
|
// Same rule: only a recognized value overrides path detection.
|
|
893
908
|
pathDetection: env.PI_VISION_PROXY_PATH_DETECTION === "on" || env.PI_VISION_PROXY_PATH_DETECTION === "off",
|
|
909
|
+
ytdlpCookies: env.PI_VISION_PROXY_YTDLP_COOKIES_FROM_BROWSER !== undefined,
|
|
910
|
+
ytdlpExtractorArgs: env.PI_VISION_PROXY_YTDLP_EXTRACTOR_ARGS !== undefined,
|
|
894
911
|
};
|
|
895
912
|
}
|
|
896
913
|
|
|
@@ -984,6 +1001,33 @@ export function sanitizeAllowedFolders(value: unknown): string[] {
|
|
|
984
1001
|
return out;
|
|
985
1002
|
}
|
|
986
1003
|
|
|
1004
|
+
/** Browsers accepted by yt-dlp's --cookies-from-browser (1.12.1). */
|
|
1005
|
+
export const YTDLP_COOKIES_BROWSERS = new Set([
|
|
1006
|
+
"chrome", "firefox", "edge", "brave", "opera", "safari", "vivaldi", "chromium", "whale",
|
|
1007
|
+
]);
|
|
1008
|
+
|
|
1009
|
+
/** Max length of the freeform --extractor-args value (1.12.1). */
|
|
1010
|
+
export const YTDLP_EXTRACTOR_ARGS_MAX = 500;
|
|
1011
|
+
|
|
1012
|
+
/**
|
|
1013
|
+
* Validate a yt-dlp --cookies-from-browser value (1.12.1). Lowercases and
|
|
1014
|
+
* accepts only known browser names; anything else collapses to "" (off).
|
|
1015
|
+
*/
|
|
1016
|
+
export function sanitizeYtdlpCookiesFromBrowser(value: unknown): string {
|
|
1017
|
+
if (typeof value !== "string") return "";
|
|
1018
|
+
const v = value.trim().toLowerCase();
|
|
1019
|
+
return YTDLP_COOKIES_BROWSERS.has(v) ? v : "";
|
|
1020
|
+
}
|
|
1021
|
+
|
|
1022
|
+
/**
|
|
1023
|
+
* Validate a yt-dlp --extractor-args value (1.12.1). Strips control/null bytes
|
|
1024
|
+
* and caps length; returned verbatim (yt-dlp parses the single arg value).
|
|
1025
|
+
*/
|
|
1026
|
+
export function sanitizeYtdlpExtractorArgs(value: unknown): string {
|
|
1027
|
+
if (typeof value !== "string") return "";
|
|
1028
|
+
return value.replace(/[\x00-\x1f\x7f]/g, "").trim().slice(0, YTDLP_EXTRACTOR_ARGS_MAX);
|
|
1029
|
+
}
|
|
1030
|
+
|
|
987
1031
|
export function sanitize(config: VisionConfig): VisionConfig {
|
|
988
1032
|
const safe: VisionConfig = { ...config };
|
|
989
1033
|
if (typeof safe.provider === "string") safe.provider = canonicalProvider(safe.provider);
|
|
@@ -1052,6 +1096,9 @@ export function sanitize(config: VisionConfig): VisionConfig {
|
|
|
1052
1096
|
if (safe.statusLine !== "on" && safe.statusLine !== "off") safe.statusLine = DEFAULT_CONFIG.statusLine;
|
|
1053
1097
|
// 1.11.0 path-detection field
|
|
1054
1098
|
if (safe.pathDetection !== "on" && safe.pathDetection !== "off") safe.pathDetection = DEFAULT_CONFIG.pathDetection;
|
|
1099
|
+
// 1.12.1 yt-dlp tuning fields
|
|
1100
|
+
safe.ytdlpCookiesFromBrowser = sanitizeYtdlpCookiesFromBrowser(safe.ytdlpCookiesFromBrowser);
|
|
1101
|
+
safe.ytdlpExtractorArgs = sanitizeYtdlpExtractorArgs(safe.ytdlpExtractorArgs);
|
|
1055
1102
|
return safe;
|
|
1056
1103
|
}
|
|
1057
1104
|
|
|
@@ -37,13 +37,15 @@
|
|
|
37
37
|
* PI_VISION_PROXY_MAX_VIDEO_BYTES - positive integer
|
|
38
38
|
* PI_VISION_PROXY_ALLOWED_PROVIDERS - comma-separated pre-consented providers
|
|
39
39
|
* PI_VISION_PROXY_STATUS_LINE - "on" | "off"
|
|
40
|
+
* PI_VISION_PROXY_YTDLP_COOKIES_FROM_BROWSER - chrome|firefox|edge|brave|opera|safari|vivaldi|chromium|whale (defeats YouTube 403s)
|
|
41
|
+
* PI_VISION_PROXY_YTDLP_EXTRACTOR_ARGS - e.g. "youtube:player_client=web_safari,web"
|
|
40
42
|
*
|
|
41
43
|
* Install:
|
|
42
44
|
* pi install ./packages/pi-multimodal-proxy
|
|
43
45
|
*/
|
|
44
46
|
|
|
45
47
|
import { execFile } from "node:child_process";
|
|
46
|
-
import { copyFile, mkdtemp, readFile, readdir, rm } from "node:fs/promises";
|
|
48
|
+
import { access, copyFile, mkdir, mkdtemp, readFile, readdir, rm } from "node:fs/promises";
|
|
47
49
|
import os from "node:os";
|
|
48
50
|
import { isAbsolute, join } from "node:path";
|
|
49
51
|
import { promisify } from "node:util";
|
|
@@ -128,6 +130,8 @@ import {
|
|
|
128
130
|
expandLeadingTilde,
|
|
129
131
|
isUncPath,
|
|
130
132
|
MAX_ALLOWED_FOLDERS,
|
|
133
|
+
sanitizeYtdlpExtractorArgs,
|
|
134
|
+
YTDLP_COOKIES_BROWSERS,
|
|
131
135
|
|
|
132
136
|
readMediaFileWithReason,
|
|
133
137
|
type ReadMediaReason,
|
|
@@ -975,6 +979,7 @@ async function downloadMediaUrl(
|
|
|
975
979
|
url: string,
|
|
976
980
|
signal: AbortSignal | undefined,
|
|
977
981
|
ctx: ExtensionContext,
|
|
982
|
+
ytdlp?: { cookiesFromBrowser?: string; extractorArgs?: string },
|
|
978
983
|
): Promise<DownloadedMedia | null> {
|
|
979
984
|
if (!(await checkYtDlp())) {
|
|
980
985
|
ctx.ui.notify(
|
|
@@ -991,19 +996,27 @@ async function downloadMediaUrl(
|
|
|
991
996
|
// actual download — it can satisfy the print from metadata alone. Adding
|
|
992
997
|
// an after_move field forces the download (it is only known once the file
|
|
993
998
|
// is in its final location) and also hands us the exact output path.
|
|
999
|
+
const args = [
|
|
1000
|
+
"--no-playlist",
|
|
1001
|
+
"--no-warnings",
|
|
1002
|
+
"--no-progress",
|
|
1003
|
+
"-f", "best[ext=mp4][height<=720]/best[height<=720]/best",
|
|
1004
|
+
"--merge-output-format", "mp4",
|
|
1005
|
+
"-o", join(tempDir, "%(id)s.%(ext)s"),
|
|
1006
|
+
];
|
|
1007
|
+
// Optional auth / player-client knobs (1.12.1) — defeat YouTube 403s on
|
|
1008
|
+
// the media fetch by reusing a logged-in browser session and/or alternate
|
|
1009
|
+
// player clients. Both opt-in; absent by default.
|
|
1010
|
+
if (ytdlp?.cookiesFromBrowser) {
|
|
1011
|
+
args.push("--cookies-from-browser", ytdlp.cookiesFromBrowser);
|
|
1012
|
+
}
|
|
1013
|
+
if (ytdlp?.extractorArgs) {
|
|
1014
|
+
args.push("--extractor-args", ytdlp.extractorArgs);
|
|
1015
|
+
}
|
|
1016
|
+
args.push("--print", "%(title)s", "--print", "after_move:filepath", url);
|
|
994
1017
|
const { stdout } = await execFileAsync(
|
|
995
1018
|
"yt-dlp",
|
|
996
|
-
|
|
997
|
-
"--no-playlist",
|
|
998
|
-
"--no-warnings",
|
|
999
|
-
"--no-progress",
|
|
1000
|
-
"-f", "best[ext=mp4][height<=720]/best[height<=720]/best",
|
|
1001
|
-
"--merge-output-format", "mp4",
|
|
1002
|
-
"-o", join(tempDir, "%(id)s.%(ext)s"),
|
|
1003
|
-
"--print", "%(title)s",
|
|
1004
|
-
"--print", "after_move:filepath",
|
|
1005
|
-
url,
|
|
1006
|
-
],
|
|
1019
|
+
args,
|
|
1007
1020
|
{ windowsHide: true, timeout: 240_000, maxBuffer: 4 * 1024 * 1024, signal },
|
|
1008
1021
|
);
|
|
1009
1022
|
|
|
@@ -1027,11 +1040,77 @@ async function downloadMediaUrl(
|
|
|
1027
1040
|
} catch (err) {
|
|
1028
1041
|
if (signal?.aborted) return null;
|
|
1029
1042
|
const msg = err instanceof Error ? err.message : String(err);
|
|
1030
|
-
|
|
1043
|
+
// A 403 on the media fetch usually means YouTube wants a logged-in
|
|
1044
|
+
// session; point the user at the cookies knob if it isn't already set.
|
|
1045
|
+
const hint = /403|forbidden/i.test(msg) && !ytdlp?.cookiesFromBrowser
|
|
1046
|
+
? " (a 403 often means YouTube wants a logged-in session — try /multimodal-proxy ytdlp cookies <browser>)"
|
|
1047
|
+
: "";
|
|
1048
|
+
ctx.ui.notify(`[multimodal-proxy] YouTube download failed for ${url}: ${msg}${hint}`, "warning");
|
|
1031
1049
|
return null;
|
|
1032
1050
|
}
|
|
1033
1051
|
}
|
|
1034
1052
|
|
|
1053
|
+
// ── Downloads folder resolution ─────────────────────────────────────────────
|
|
1054
|
+
//
|
|
1055
|
+
// The Windows default `C:\Users\<user>\Downloads` is frequently NOT the folder
|
|
1056
|
+
// File Explorer shows: OneDrive redirection or a custom relocation (e.g.
|
|
1057
|
+
// `D:\Downloads`) moves it elsewhere. Hardcoding the default then makes
|
|
1058
|
+
// `copyFile` throw ENOENT — and pre-fix that error was swallowed silently, so
|
|
1059
|
+
// the user clicked "Save", nothing appeared, and no error was surfaced.
|
|
1060
|
+
//
|
|
1061
|
+
// We resolve the real folder via the Shell known-folder (FOLDERID_Downloads)
|
|
1062
|
+
// on Windows, cache the result, and fall back to `~/Downloads` elsewhere.
|
|
1063
|
+
|
|
1064
|
+
let downloadsDirCache: string | null = null;
|
|
1065
|
+
|
|
1066
|
+
async function pathExists(p: string): Promise<boolean> {
|
|
1067
|
+
try {
|
|
1068
|
+
await access(p);
|
|
1069
|
+
return true;
|
|
1070
|
+
} catch {
|
|
1071
|
+
return false;
|
|
1072
|
+
}
|
|
1073
|
+
}
|
|
1074
|
+
|
|
1075
|
+
/**
|
|
1076
|
+
* Resolve the user's real Downloads folder, respecting Windows relocation /
|
|
1077
|
+
* OneDrive redirection. Falls back to the default `~/Downloads` (which the
|
|
1078
|
+
* caller creates on demand). Result is cached for the process lifetime.
|
|
1079
|
+
*/
|
|
1080
|
+
async function resolveDownloadsDir(): Promise<string> {
|
|
1081
|
+
if (downloadsDirCache) return downloadsDirCache;
|
|
1082
|
+
const defaultDir = join(os.homedir(), "Downloads");
|
|
1083
|
+
if (await pathExists(defaultDir)) {
|
|
1084
|
+
downloadsDirCache = defaultDir;
|
|
1085
|
+
return defaultDir;
|
|
1086
|
+
}
|
|
1087
|
+
// Default is missing (relocated / OneDrive-redirected). On Windows, ask the
|
|
1088
|
+
// Shell for the real Downloads known-folder path; this respects redirection.
|
|
1089
|
+
if (process.platform === "win32") {
|
|
1090
|
+
try {
|
|
1091
|
+
const { stdout } = await execFileAsync(
|
|
1092
|
+
"powershell.exe",
|
|
1093
|
+
[
|
|
1094
|
+
"-NoProfile",
|
|
1095
|
+
"-NonInteractive",
|
|
1096
|
+
"-Command",
|
|
1097
|
+
"(New-Object -ComObject Shell.Application).NameSpace('shell:Downloads').Self.Path",
|
|
1098
|
+
],
|
|
1099
|
+
{ windowsHide: true, timeout: 10_000 },
|
|
1100
|
+
);
|
|
1101
|
+
const resolved = stdout.split(/\r?\n/).map((l) => l.trim()).find((l) => l.length > 0) ?? "";
|
|
1102
|
+
if (resolved && (await pathExists(resolved))) {
|
|
1103
|
+
downloadsDirCache = resolved;
|
|
1104
|
+
return resolved;
|
|
1105
|
+
}
|
|
1106
|
+
} catch {
|
|
1107
|
+
// PowerShell unavailable or failed — fall back to default (created below).
|
|
1108
|
+
}
|
|
1109
|
+
}
|
|
1110
|
+
downloadsDirCache = defaultDir;
|
|
1111
|
+
return defaultDir;
|
|
1112
|
+
}
|
|
1113
|
+
|
|
1035
1114
|
async function analyzeVideoViaXaiStt(
|
|
1036
1115
|
mediaFile: { type: "image"; data: string; mimeType: string },
|
|
1037
1116
|
filename: string,
|
|
@@ -1693,7 +1772,10 @@ export default function (pi: ExtensionAPI) {
|
|
|
1693
1772
|
ctx,
|
|
1694
1773
|
() => `Downloading YouTube ${id}…`,
|
|
1695
1774
|
`Downloading YouTube ${id}…`,
|
|
1696
|
-
() => downloadMediaUrl(watch, ctx.signal, ctx
|
|
1775
|
+
() => downloadMediaUrl(watch, ctx.signal, ctx, {
|
|
1776
|
+
cookiesFromBrowser: config.ytdlpCookiesFromBrowser,
|
|
1777
|
+
extractorArgs: config.ytdlpExtractorArgs,
|
|
1778
|
+
}),
|
|
1697
1779
|
);
|
|
1698
1780
|
if (!dl) continue;
|
|
1699
1781
|
downloadedTempDirs.push(dl.tempDir);
|
|
@@ -1813,15 +1895,23 @@ export default function (pi: ExtensionAPI) {
|
|
|
1813
1895
|
"Save downloaded video?",
|
|
1814
1896
|
`"${info.title}"\n\n${sizeMB} MB — save to your Downloads folder?`,
|
|
1815
1897
|
);
|
|
1816
|
-
if (save)
|
|
1817
|
-
|
|
1818
|
-
|
|
1898
|
+
if (!save) continue;
|
|
1899
|
+
const safeTitle = info.title.replace(/[<>:"\/\\|?*]/g, "_").replace(/\s+/g, " ").trim().slice(0, 200);
|
|
1900
|
+
const destDir = await resolveDownloadsDir();
|
|
1901
|
+
const dest = join(destDir, `${safeTitle}.mp4`);
|
|
1902
|
+
try {
|
|
1903
|
+
await mkdir(destDir, { recursive: true });
|
|
1819
1904
|
await copyFile(path, dest);
|
|
1820
|
-
ctx.ui.notify(`[multimodal-proxy] ✓ Saved "${safeTitle}.mp4" to
|
|
1905
|
+
ctx.ui.notify(`[multimodal-proxy] ✓ Saved "${safeTitle}.mp4" to ${dest}`, "info");
|
|
1906
|
+
} catch (saveErr) {
|
|
1907
|
+
// Surface save failures (missing dir, permissions, disk) — the
|
|
1908
|
+
// previous silent catch hid them and the video just vanished.
|
|
1909
|
+
const reason = saveErr instanceof Error ? saveErr.message : String(saveErr);
|
|
1910
|
+
ctx.ui.notify(`[multimodal-proxy] ✗ Could not save video to ${dest}: ${reason}`, "warning");
|
|
1821
1911
|
}
|
|
1822
1912
|
} catch {
|
|
1823
|
-
//
|
|
1824
|
-
// cleaned below regardless.
|
|
1913
|
+
// confirm() dialog itself failed — don't block the agent. Temp
|
|
1914
|
+
// files are cleaned below regardless.
|
|
1825
1915
|
}
|
|
1826
1916
|
}
|
|
1827
1917
|
await cleanupDownloads();
|
|
@@ -2522,6 +2612,73 @@ Use "*" or "all" to grant consent for all providers globally.`,
|
|
|
2522
2612
|
return;
|
|
2523
2613
|
}
|
|
2524
2614
|
|
|
2615
|
+
// ── yt-dlp cookies / extractor-args (defeat YouTube 403s) ───────
|
|
2616
|
+
if (sub === "ytdlp") {
|
|
2617
|
+
const { sub: ySub, value: yValue } = splitSubcommand(value);
|
|
2618
|
+
|
|
2619
|
+
if (ySub === "cookies") {
|
|
2620
|
+
if (env.ytdlpCookies) {
|
|
2621
|
+
ctx.ui.notify(
|
|
2622
|
+
"[multimodal-proxy] PI_VISION_PROXY_YTDLP_COOKIES_FROM_BROWSER is set - env overrides commands. Unset to change.",
|
|
2623
|
+
"warning",
|
|
2624
|
+
);
|
|
2625
|
+
return;
|
|
2626
|
+
}
|
|
2627
|
+
const trimmed = yValue.trim().toLowerCase();
|
|
2628
|
+
if (!trimmed || trimmed === "off") {
|
|
2629
|
+
writePersisted({ ...persisted, ytdlpCookiesFromBrowser: "" });
|
|
2630
|
+
ctx.ui.notify("[multimodal-proxy] yt-dlp cookies-from-browser: off", "info");
|
|
2631
|
+
return;
|
|
2632
|
+
}
|
|
2633
|
+
if (!YTDLP_COOKIES_BROWSERS.has(trimmed)) {
|
|
2634
|
+
ctx.ui.notify(
|
|
2635
|
+
`[multimodal-proxy] Unknown browser "${trimmed}". Supported: ${[...YTDLP_COOKIES_BROWSERS].join(", ")}`,
|
|
2636
|
+
"warning",
|
|
2637
|
+
);
|
|
2638
|
+
return;
|
|
2639
|
+
}
|
|
2640
|
+
writePersisted({ ...persisted, ytdlpCookiesFromBrowser: trimmed });
|
|
2641
|
+
ctx.ui.notify(
|
|
2642
|
+
`[multimodal-proxy] yt-dlp cookies-from-browser: ${trimmed} (applied on next YouTube download)`,
|
|
2643
|
+
"info",
|
|
2644
|
+
);
|
|
2645
|
+
return;
|
|
2646
|
+
}
|
|
2647
|
+
|
|
2648
|
+
if (ySub === "extractor-args") {
|
|
2649
|
+
if (env.ytdlpExtractorArgs) {
|
|
2650
|
+
ctx.ui.notify(
|
|
2651
|
+
"[multimodal-proxy] PI_VISION_PROXY_YTDLP_EXTRACTOR_ARGS is set - env overrides commands. Unset to change.",
|
|
2652
|
+
"warning",
|
|
2653
|
+
);
|
|
2654
|
+
return;
|
|
2655
|
+
}
|
|
2656
|
+
const trimmed = yValue.trim();
|
|
2657
|
+
if (!trimmed || trimmed.toLowerCase() === "off") {
|
|
2658
|
+
writePersisted({ ...persisted, ytdlpExtractorArgs: "" });
|
|
2659
|
+
ctx.ui.notify("[multimodal-proxy] yt-dlp extractor-args: off", "info");
|
|
2660
|
+
return;
|
|
2661
|
+
}
|
|
2662
|
+
const cleaned = sanitizeYtdlpExtractorArgs(trimmed);
|
|
2663
|
+
writePersisted({ ...persisted, ytdlpExtractorArgs: cleaned });
|
|
2664
|
+
ctx.ui.notify(`[multimodal-proxy] yt-dlp extractor-args: ${cleaned}`, "info");
|
|
2665
|
+
return;
|
|
2666
|
+
}
|
|
2667
|
+
|
|
2668
|
+
ctx.ui.notify(
|
|
2669
|
+
"[multimodal-proxy] yt-dlp options:\n" +
|
|
2670
|
+
` cookies-from-browser: ${effective.ytdlpCookiesFromBrowser || "(off)"}\n` +
|
|
2671
|
+
` extractor-args: ${effective.ytdlpExtractorArgs || "(off)"}\n` +
|
|
2672
|
+
(env.ytdlpCookies || env.ytdlpExtractorArgs ? " (env override active)\n" : "") +
|
|
2673
|
+
"Usage:\n" +
|
|
2674
|
+
" /multimodal-proxy ytdlp cookies <browser|off> reuse a logged-in YouTube session (fixes most 403s)\n" +
|
|
2675
|
+
' /multimodal-proxy ytdlp extractor-args <text|off> e.g. youtube:player_client=web_safari,web\n' +
|
|
2676
|
+
`Browsers: ${[...YTDLP_COOKIES_BROWSERS].join(", ")}`,
|
|
2677
|
+
"info",
|
|
2678
|
+
);
|
|
2679
|
+
return;
|
|
2680
|
+
}
|
|
2681
|
+
|
|
2525
2682
|
// ── max-images-per-call ────────────────────────────
|
|
2526
2683
|
if (sub === "max-images-per-call") {
|
|
2527
2684
|
if (env.maxImagesPerCall) {
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-multimodal-proxy",
|
|
3
|
-
"version": "1.12.
|
|
3
|
+
"version": "1.12.1",
|
|
4
4
|
"description": "Automatic image, video and audio description for any model in Pi. Routes media to a multimodal model and injects descriptions into context.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"pi-package"
|