pi-multimodal-proxy 1.18.1 → 1.19.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +30 -0
- package/README.md +31 -0
- package/extensions/__tests__/internal.test.ts +199 -3
- package/extensions/__tests__/recall-autocomplete.test.ts +10 -0
- package/extensions/internal.ts +215 -19
- package/extensions/vision-proxy.ts +635 -52
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,36 @@ All notable changes to this project will be documented in this file.
|
|
|
4
4
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
|
6
6
|
|
|
7
|
+
## [1.19.1] - 2026-10-02
|
|
8
|
+
|
|
9
|
+
### Changed
|
|
10
|
+
|
|
11
|
+
- **Leaner `analyze_image` description** — the tool description shrinks ~40% (1.7 KB → ~1.0 KB) while keeping every actionable detail (when to use, the three crop forms with shapes, recall ids, fence attributes, `crop_origin` math). Pi 1.0's codemode lists tool declarations under a 3000-token inline budget and drops what doesn't fit, so a long description both costs tokens on every request and risks pushing other tools out of the listing.
|
|
12
|
+
- **Recovery-style errors (Pi 1.0 codemode philosophy: errors say how to recover)** — `analyze_image`'s model-not-found error now names close registry matches (`anthropic/claude-sont-5` → suggests `anthropic/claude-sonnet-5`, `…-sonnet-4-5`), the recall-id-not-found error lists close ids still retained in the session cache, and zero-area crop errors explain the expected shape (fraction/absolute bounds) instead of just "zero area".
|
|
13
|
+
- **Dynamic implicit-vision-model fallback (Pi ≥ 1.0, feature-detected)** — when the curated fallback chain (`zai/glm-5.3-flash` → `deepseek/…-flash-vision-exp` → `anthropic/claude-sonnet-5/4-5`) is entirely unkeyed but another keyed vision-capable chat model exists (openrouter, google, …), `ModelRegistry.getAvailableOfType("chat")` resolves the implicit default to the first one (cheap-tier ids — flash/mini/fast/haiku/express/lite — preferred so an implicit default never lands on a frontier-priced model). Probe cached per session; older runtimes and the "nothing keyed anywhere" notice path are unchanged.
|
|
14
|
+
- **`#` recall autocomplete regression tests** mirroring pi #10218 (autocomplete must trigger when input starts with whitespace) — the `(?:^|\s)#` token regex already handled it; the tests pin it.
|
|
15
|
+
- Six new tests: `closeMatches` (near-typo ranking, substring bonus, relevance floor, deterministic ties), `pickDynamicVisionCandidate` (cheap-tier preference, first-capable fallback, input filtering), zero-area crop recovery hints (507 total, all green).
|
|
16
|
+
|
|
17
|
+
## [1.19.0] - 2026-10-01
|
|
18
|
+
|
|
19
|
+
### Added
|
|
20
|
+
|
|
21
|
+
- **Virtual model — `Multimodal Auto`** (`multimodal-proxy/auto`, Pi ≥ 0.99, feature-detected). The extension registers a virtual model via `pi.registerVirtualModel()`; selecting it in `/model` routes each request: turns whose pending user input carries image blocks go natively to the configured vision model (real pixels, no description fence), everything else goes to the base model — the pinned `virtual-base` model, else the physical model that answered last (`request.previous`), else the vision model. Sticky routing (`retry` → `failed`, `continuation`/`direct` → `previous`) keeps prompt caches and thinking signatures valid. `route()` resolves config per request (no registration churn on model changes) and throws an actionable error pointing at `/multimodal-proxy pick` when no vision model resolves. New commands `/multimodal-proxy virtual on|off` (default on; `unregisterVirtualModel` when off), `/multimodal-proxy virtual-base <provider/model-id>|off`, env overrides `PI_VISION_PROXY_VIRTUAL` / `PI_VISION_PROXY_VIRTUAL_BASE`, status-line `[+virtual]` marker, interactive-menu entry, and summary/help lines. With the virtual model selected, image stripping never applies (it advertises image input); video/audio detection, `analyze_image`, and consent gates are unchanged.
|
|
22
|
+
- **`analyze_image` adopts the Pi 0.99 extension tool API** (older hosts ignore the new fields): results carry `isError: true` for failures instead of "Error: …" text masquerading as content (all ~18 failure paths now route through an `AnalyzeImageOutcome`), successful calls return an `outputSchema`-shaped `structuredContent` — analysis text, `cached`, fallback-aware `provider`/`model`, `latencyMs`, `groundingFormat`, and per-image `id`/`filename`/dimensions/absolute-pixel `crop` — so codemode scripts and other `ctx.executeTool()` callers receive machine-readable results, and MCP-style `annotations` (`readOnlyHint: true`, `openWorldHint: true`) plus a `multimodal-proxy` namespace group the tool in codemode listings.
|
|
23
|
+
- **Model-aware upload downscale** (Pi ≥ 0.87). `modelImageResizeLimits()` reads the vision model catalog entry's `inputLimits.images.resize` (`maxWidth`/`maxHeight`/`maxBytes`/`jpegQuality`, all optional) and `downscaleForUpload()` tightens the user's `maxUploadDim`/`maxUploadBytes`/quality with it — the derived long-edge target is `min(maxWidth, maxHeight)`, the base64 `maxBytes` budget is converted to raw bytes (×0.75) before mixing with the user's raw budget, and a user-disabled downscale (`maxUploadDim: 0`) always wins. `downscaleImage` gained a `quality` parameter threaded through both the worker task and the in-thread path. All four upload paths (auto-proxy, `analyze_image`, joint descriptions, `/describe`) pass the resolved vision model's limits.
|
|
24
|
+
- **Compaction-proof, cache-stable recall hint** (Pi ≥ 0.87). A `context_with_system` handler — probed on first invocation so the affordance never appears twice — takes over the `analyze_image` recall affordance once media has been described: a stable system-layer block replaces the per-user-message restatement. It survives compaction (which summarized the old hint away) and reaches vision-capable session models, where the strip path never fired. Older hosts keep the exact previous behavior.
|
|
25
|
+
|
|
26
|
+
### Changed
|
|
27
|
+
|
|
28
|
+
- `envFlags()` reports two new presence flags (`virtual`, `virtualBase`); the config summary, no-UI command list, and interactive menu show the virtual-model state (including a "(needs pi ≥ 0.99)" qualifier when registered on an older host).
|
|
29
|
+
- Six new tests: env override + flag presence for the virtual vars, `modelImageResizeLimits` derivation (null cases, box target, quality default, raw-byte conversion), model-limits tightening vs. user-disabled downscale vs. in-budget images, and sanitize validation of the new fields (494 total, all green).
|
|
30
|
+
|
|
31
|
+
### Fixed (PR #34 review)
|
|
32
|
+
|
|
33
|
+
- **`modelImageResizeLimits` no longer drops maxBytes-only declarations** — a catalog entry with `inputLimits.images.resize.maxBytes` but no `maxWidth`/`maxHeight` is now honored: `targetDim` stays unset (the user's dim target applies) while the byte budget (converted to raw bytes, floored at 1) and JPEG quality still tighten the upload path. Quality-only declarations remain a no-op.
|
|
34
|
+
- **Virtual-model routing filters for vision-capable entries** — `resolveVisionTarget` only accepts models whose `input` includes `image`, so a text-only pin (via `/multimodal-proxy model` or `PI_VISION_PROXY_MODEL`) can no longer route a media turn to a model that would receive image placeholders; `route()` throws the actionable `/multimodal-proxy pick` error instead.
|
|
35
|
+
- **Cache-hit `structuredContent.text` now matches fresh calls** — the analyze_image tool cache stores `{ fence, raw }`, so cached requests return the raw analysis text in `structuredContent` (as `AnalyzeImageOutput` advertises) while the model still sees the identical fence.
|
|
36
|
+
|
|
7
37
|
## [1.18.1] - 2026-08-30
|
|
8
38
|
|
|
9
39
|
### Fixed (post-release review of 1.18.0)
|
package/README.md
CHANGED
|
@@ -8,6 +8,15 @@ When **video or audio files** are detected, they are routed to a **multimodal mo
|
|
|
8
8
|
|
|
9
9
|
**YouTube links** are detected too: paste a URL (`youtube.com/watch?v=…`, `youtu.be/…`, `/shorts/…`, etc.) and the video is downloaded with [`yt-dlp`](https://github.com/yt-dlp/yt-dlp) and analyzed exactly like a local file.
|
|
10
10
|
|
|
11
|
+
## What's new in 1.19.0
|
|
12
|
+
|
|
13
|
+
Deep integration with recent Pi releases (0.87 / 0.99):
|
|
14
|
+
|
|
15
|
+
- **Virtual model — `Multimodal Auto`** (Pi ≥ 0.99, on by default): a new entry in `/model` that routes each request for you — turns whose pending input carries images go **natively** to your configured vision model (real pixels, no text fence), everything else stays on your base model (auto = whatever physical model handled the session before, or pin one with `/multimodal-proxy virtual-base`). Sticky routing keeps prompt caches valid within a turn; the footer shows the routed model and `/session` lists cost per physical model. Disable with `/multimodal-proxy virtual off`.
|
|
16
|
+
- **`analyze_image` speaks the 0.99 tool API** — errors now surface as real tool errors (`isError`) instead of fake text, results carry a machine-readable `structuredContent` (analysis text, provider/model, latency, grounding format, per-image ids/dimensions/crop) so **codemode scripts** can consume them programmatically, and the tool is annotated (`readOnlyHint`) and namespaced (`multimodal-proxy`) for codemode listings.
|
|
17
|
+
- **Model-aware upload downscale** (Pi ≥ 0.87) — when the configured vision model declares `inputLimits.images.resize` in its catalog entry, the proxy derives the downscale target, JPEG quality, and byte budget from it (tightening, never loosening, your `max-upload` settings). No more double-resizing against provider limits.
|
|
18
|
+
- **Compaction-proof recall hint** (Pi ≥ 0.87) — the `analyze_image` recall affordance moves to a stable system-layer block via `context_with_system` once media has been described: it survives compaction (the old per-user-message hint was summarized away) and also reaches vision-capable session models, where the strip path never fired. Older hosts keep the previous behavior.
|
|
19
|
+
|
|
11
20
|
## What's new in 1.18.0
|
|
12
21
|
|
|
13
22
|
- **Key-aware default model resolution** — the implicit vision default now lands on a model you can actually call. If the default (`zai/glm-5.3-flash`) has no API key in your setup, the proxy walks a preference-ordered candidate chain — `zai/glm-5.3-flash` → `deepseek/deepseek-v4-flash-vision-exp` (new in Pi 0.84.4, cheap, built-in provider) → `anthropic/claude-sonnet-5` — and picks the first candidate that is in the catalog **and keyed**. A keyed legacy default is never traded for an unkeyed model; explicit choices (`/multimodal-proxy model`, `pick`, `PI_VISION_PROXY_MODEL`) are still never rewritten; and when nothing is keyed you get the actionable `pi --login` hint as before. A substitution is announced once per session.
|
|
@@ -118,6 +127,24 @@ Both are persisted in `~/.pi/agent/multimodal-proxy.json` and can be set via env
|
|
|
118
127
|
| **`always`** | Always uses the proxy, even if the active model supports images |
|
|
119
128
|
| **`off`** | Disabled entirely |
|
|
120
129
|
|
|
130
|
+
## Virtual model (Pi ≥ 0.99)
|
|
131
|
+
|
|
132
|
+
`Multimodal Auto` (`multimodal-proxy/auto`) is a [virtual model](https://pi.dev/docs/latest/virtual-models) the extension registers on Pi 0.99+ (on by default; `/multimodal-proxy virtual off` removes it). Select it once in `/model` and every request is routed for you:
|
|
133
|
+
|
|
134
|
+
| Turn | Routed to |
|
|
135
|
+
|------|-----------|
|
|
136
|
+
| Pending input carries an image attachment | Your configured **vision model** — natively, so the model sees the actual pixels |
|
|
137
|
+
| Everything else | Your **base model**: the pinned `/multimodal-proxy virtual-base` model, else the physical model that handled the session before, else the vision model |
|
|
138
|
+
|
|
139
|
+
Why you'd want it:
|
|
140
|
+
|
|
141
|
+
- **One selection, automatic escalation** — keep a cheap/fast model for text turns and pay for the vision model only when media actually appears. No manual `/model` switching.
|
|
142
|
+
- **Native vision instead of descriptions** — unlike proxy mode (which converts images to text fences for non-vision models), media turns get real pixel fidelity while `analyze_image` recall stays available for re-querying.
|
|
143
|
+
- **Cost transparency for free** — the footer shows `Multimodal Auto → <routed model>`, and `/session` lists cost per physical model.
|
|
144
|
+
- **Cache-safe** — continuations and retries stay on the model that handled the turn, so prompt caches and thinking signatures survive.
|
|
145
|
+
|
|
146
|
+
Virtual-model routing is orthogonal to proxy mode: with `Multimodal Auto` selected, image stripping never applies (the virtual model advertises image input), while video/audio path detection, `analyze_image`, and consent gates keep working exactly as before.
|
|
147
|
+
|
|
121
148
|
## Configuration
|
|
122
149
|
|
|
123
150
|
Settings persist across sessions in `~/.pi/agent/multimodal-proxy.json`. Environment variables override file settings; in-session commands override both.
|
|
@@ -147,6 +174,8 @@ Settings persist across sessions in `~/.pi/agent/multimodal-proxy.json`. Environ
|
|
|
147
174
|
(clear with: /multimodal-proxy fallback-model clear)
|
|
148
175
|
/multimodal-proxy retry <0-5> → retries on transient errors (429/5xx/network), default 2
|
|
149
176
|
/multimodal-proxy max-upload <dim | n mb | off> → downscale uploads larger than this (default 2048 px / 5 MB; off = 8192 px / 20 MB)
|
|
177
|
+
/multimodal-proxy virtual on | off → register/remove the 'Multimodal Auto' virtual model (Pi ≥ 0.99, default on)
|
|
178
|
+
/multimodal-proxy virtual-base <provider/model-id> → pin the text-turn model for the virtual model ('off' clears → auto)
|
|
150
179
|
/multimodal-proxy status on | off → show/hide the steady status line
|
|
151
180
|
/multimodal-proxy grounding-models list → show grounding-capable models
|
|
152
181
|
/multimodal-proxy grounding-models add <provider/id> [--format <fmt>]
|
|
@@ -188,6 +217,8 @@ Legacy alias: /vision-proxy <args> works identically.
|
|
|
188
217
|
| `PI_VISION_PROXY_MAX_UPLOAD_DIM` | `0` (disable downscaling entirely), or 512–8192 px long-edge threshold | `2048` |
|
|
189
218
|
| `PI_VISION_PROXY_MAX_UPLOAD_MB` | 0.5–20 upload byte budget before downscale | `5` |
|
|
190
219
|
| `PI_VISION_PROXY_FALLBACK_MODEL` | `provider/model-id`, or `none`/`off` to clear | not set |
|
|
220
|
+
| `PI_VISION_PROXY_VIRTUAL` | `on`, `off` — register the `Multimodal Auto` virtual model (Pi ≥ 0.99) | `on` |
|
|
221
|
+
| `PI_VISION_PROXY_VIRTUAL_BASE` | `provider/model-id` pinning the virtual model's text-turn model, or `none` to clear | not set |
|
|
191
222
|
| `PI_VISION_PROXY_STATUS_LINE` | `on`, `off` | `on` |
|
|
192
223
|
| `PI_VISION_PROXY_PATH_DETECTION` | `on`, `off` — `off` disables scanning prompt text for media file paths; structured attachments are always processed | `on` |
|
|
193
224
|
|
|
@@ -79,6 +79,8 @@ import {
|
|
|
79
79
|
bufferToPiAiImage,
|
|
80
80
|
shouldStripImages,
|
|
81
81
|
selectVisionModels,
|
|
82
|
+
closeMatches,
|
|
83
|
+
pickDynamicVisionCandidate,
|
|
82
84
|
splitSubcommand,
|
|
83
85
|
stripImagePaths,
|
|
84
86
|
stripMediaPaths,
|
|
@@ -98,6 +100,7 @@ import {
|
|
|
98
100
|
RECALL_HINT,
|
|
99
101
|
createAbortError,
|
|
100
102
|
downscaleForUpload,
|
|
103
|
+
modelImageResizeLimits,
|
|
101
104
|
overUploadDim,
|
|
102
105
|
downscaleTargetDim,
|
|
103
106
|
isAbortError,
|
|
@@ -278,18 +281,18 @@ describe("readEnvOverrides", () => {
|
|
|
278
281
|
|
|
279
282
|
describe("envFlags", () => {
|
|
280
283
|
it("reports presence per variable", () => {
|
|
281
|
-
assert.deepEqual(envFlags({}), { mode: false, model: false, context: false, tool: false, maxImagesPerCall: false, maxBatch: false, cacheSize: false, videoModel: false, allowedProviders: false, allowHome: false, allowedFolders: false, statusLine: false, pathDetection: false, ytdlpCookies: false, ytdlpExtractorArgs: false, retryMax: false, maxUpload: false, fallbackModel: false });
|
|
284
|
+
assert.deepEqual(envFlags({}), { mode: false, model: false, context: false, tool: false, maxImagesPerCall: false, maxBatch: false, cacheSize: false, videoModel: false, allowedProviders: false, allowHome: false, allowedFolders: false, statusLine: false, pathDetection: false, ytdlpCookies: false, ytdlpExtractorArgs: false, retryMax: false, maxUpload: false, fallbackModel: false, virtual: false, virtualBase: false });
|
|
282
285
|
assert.deepEqual(
|
|
283
286
|
envFlags({
|
|
284
287
|
PI_VISION_PROXY_MODE: "x",
|
|
285
288
|
PI_VISION_PROXY_MODEL: "y",
|
|
286
289
|
PI_VISION_PROXY_INCLUDE_CONTEXT: "",
|
|
287
290
|
}),
|
|
288
|
-
{ mode: true, model: true, context: true, tool: false, maxImagesPerCall: false, maxBatch: false, cacheSize: false, videoModel: false, allowedProviders: false, allowHome: false, allowedFolders: false, statusLine: false, pathDetection: false, ytdlpCookies: false, ytdlpExtractorArgs: false, retryMax: false, maxUpload: false, fallbackModel: false },
|
|
291
|
+
{ mode: true, model: true, context: true, tool: false, maxImagesPerCall: false, maxBatch: false, cacheSize: false, videoModel: false, allowedProviders: false, allowHome: false, allowedFolders: false, statusLine: false, pathDetection: false, ytdlpCookies: false, ytdlpExtractorArgs: false, retryMax: false, maxUpload: false, fallbackModel: false, virtual: false, virtualBase: false },
|
|
289
292
|
);
|
|
290
293
|
assert.deepEqual(
|
|
291
294
|
envFlags({ PI_VISION_PROXY_ALLOW_HOME: "1", PI_VISION_PROXY_ALLOWED_FOLDERS: "/a" }),
|
|
292
|
-
{ mode: false, model: false, context: false, tool: false, maxImagesPerCall: false, maxBatch: false, cacheSize: false, videoModel: false, allowedProviders: false, allowHome: true, allowedFolders: true, statusLine: false, pathDetection: false, ytdlpCookies: false, ytdlpExtractorArgs: false, retryMax: false, maxUpload: false, fallbackModel: false },
|
|
295
|
+
{ mode: false, model: false, context: false, tool: false, maxImagesPerCall: false, maxBatch: false, cacheSize: false, videoModel: false, allowedProviders: false, allowHome: true, allowedFolders: true, statusLine: false, pathDetection: false, ytdlpCookies: false, ytdlpExtractorArgs: false, retryMax: false, maxUpload: false, fallbackModel: false, virtual: false, virtualBase: false },
|
|
293
296
|
);
|
|
294
297
|
assert.equal(envFlags({ PI_VISION_PROXY_ALLOWED_PROVIDERS: "" }).allowedProviders, true);
|
|
295
298
|
// An unrecognized ALLOW_HOME value is not an override and must not lock the command
|
|
@@ -302,6 +305,21 @@ describe("envFlags", () => {
|
|
|
302
305
|
assert.equal(envFlags({ PI_VISION_PROXY_STATUS_LINE: "bogus" }).statusLine, false);
|
|
303
306
|
assert.equal(envFlags({ PI_VISION_PROXY_YTDLP_COOKIES_FROM_BROWSER: "chrome" }).ytdlpCookies, true);
|
|
304
307
|
assert.equal(envFlags({ PI_VISION_PROXY_YTDLP_EXTRACTOR_ARGS: "youtube:player_client=web" }).ytdlpExtractorArgs, true);
|
|
308
|
+
// 1.19.0 virtual-model overrides
|
|
309
|
+
assert.equal(envFlags({ PI_VISION_PROXY_VIRTUAL: "on" }).virtual, true);
|
|
310
|
+
assert.equal(envFlags({ PI_VISION_PROXY_VIRTUAL: "bogus" }).virtual, false);
|
|
311
|
+
assert.equal(envFlags({ PI_VISION_PROXY_VIRTUAL_BASE: "zai/glm-5.3-flash" }).virtualBase, true);
|
|
312
|
+
assert.equal(envFlags({ PI_VISION_PROXY_VIRTUAL_BASE: "none" }).virtualBase, true);
|
|
313
|
+
assert.deepEqual(readEnvOverrides({ PI_VISION_PROXY_VIRTUAL: "off" }).virtualModel, "off");
|
|
314
|
+
assert.deepEqual(readEnvOverrides({ PI_VISION_PROXY_VIRTUAL: "bogus" }).virtualModel, undefined);
|
|
315
|
+
assert.deepEqual(readEnvOverrides({ PI_VISION_PROXY_VIRTUAL_BASE: "zai/glm-5.3-flash" }), {
|
|
316
|
+
virtualBaseProvider: "zai",
|
|
317
|
+
virtualBaseModelId: "glm-5.3-flash",
|
|
318
|
+
});
|
|
319
|
+
assert.deepEqual(readEnvOverrides({ PI_VISION_PROXY_VIRTUAL_BASE: "none" }), {
|
|
320
|
+
virtualBaseProvider: undefined,
|
|
321
|
+
virtualBaseModelId: undefined,
|
|
322
|
+
});
|
|
305
323
|
});
|
|
306
324
|
});
|
|
307
325
|
|
|
@@ -358,6 +376,66 @@ describe("selectVisionModels", () => {
|
|
|
358
376
|
});
|
|
359
377
|
});
|
|
360
378
|
|
|
379
|
+
describe("closeMatches", () => {
|
|
380
|
+
const catalog = [
|
|
381
|
+
"anthropic/claude-sonnet-5",
|
|
382
|
+
"anthropic/claude-sonnet-4-5",
|
|
383
|
+
"deepseek/deepseek-v4-flash-vision-exp",
|
|
384
|
+
"zai/glm-5.3-flash",
|
|
385
|
+
"openai/gpt-5.6",
|
|
386
|
+
];
|
|
387
|
+
|
|
388
|
+
it("ranks a near-typo above unrelated models", () => {
|
|
389
|
+
assert.deepEqual(closeMatches("anthropic/claude-sont-5", catalog, 2), [
|
|
390
|
+
"anthropic/claude-sonnet-5",
|
|
391
|
+
"anthropic/claude-sonnet-4-5",
|
|
392
|
+
]);
|
|
393
|
+
});
|
|
394
|
+
|
|
395
|
+
it("gives substring matches a strong bonus", () => {
|
|
396
|
+
assert.deepEqual(closeMatches("glm", catalog, 1), ["zai/glm-5.3-flash"]);
|
|
397
|
+
});
|
|
398
|
+
|
|
399
|
+
it("returns nothing for queries with no reasonable match", () => {
|
|
400
|
+
assert.deepEqual(closeMatches("banana-pudding", catalog), []);
|
|
401
|
+
});
|
|
402
|
+
|
|
403
|
+
it("is deterministic on ties (alphabetical)", () => {
|
|
404
|
+
assert.deepEqual(closeMatches("mod", ["mod-b", "mod-a", "mod-c"], 3), ["mod-a", "mod-b", "mod-c"]);
|
|
405
|
+
});
|
|
406
|
+
|
|
407
|
+
it("handles empty input", () => {
|
|
408
|
+
assert.deepEqual(closeMatches("", catalog), []);
|
|
409
|
+
assert.deepEqual(closeMatches("x", []), []);
|
|
410
|
+
});
|
|
411
|
+
});
|
|
412
|
+
|
|
413
|
+
describe("pickDynamicVisionCandidate", () => {
|
|
414
|
+
const m = (provider: string, modelId: string, input: readonly string[] = ["text", "image"]) => ({
|
|
415
|
+
provider,
|
|
416
|
+
modelId,
|
|
417
|
+
input,
|
|
418
|
+
});
|
|
419
|
+
|
|
420
|
+
it("prefers a cheap-tier model over registry order", () => {
|
|
421
|
+
const out = pickDynamicVisionCandidate([
|
|
422
|
+
m("anthropic", "claude-opus-6"),
|
|
423
|
+
m("zai", "glm-5.3-flash"),
|
|
424
|
+
]);
|
|
425
|
+
assert.equal(out?.modelId, "glm-5.3-flash");
|
|
426
|
+
});
|
|
427
|
+
|
|
428
|
+
it("falls back to the first image-capable model when none is cheap-tier", () => {
|
|
429
|
+
const out = pickDynamicVisionCandidate([m("anthropic", "claude-opus-6"), m("x", "y")]);
|
|
430
|
+
assert.equal(out?.modelId, "claude-opus-6");
|
|
431
|
+
});
|
|
432
|
+
|
|
433
|
+
it("skips models without image input and empty lists", () => {
|
|
434
|
+
assert.equal(pickDynamicVisionCandidate([m("a", "b", ["text"])]), undefined);
|
|
435
|
+
assert.equal(pickDynamicVisionCandidate([]), undefined);
|
|
436
|
+
});
|
|
437
|
+
});
|
|
438
|
+
|
|
361
439
|
describe("sanitizeProviderHeaders", () => {
|
|
362
440
|
it("drops null deletion markers", () => {
|
|
363
441
|
assert.deepEqual(
|
|
@@ -1828,6 +1906,21 @@ describe("resolveCropEntry", () => {
|
|
|
1828
1906
|
/zero area/,
|
|
1829
1907
|
);
|
|
1830
1908
|
});
|
|
1909
|
+
|
|
1910
|
+
it("zero-area normalized errors carry a recovery hint (pi 1.0 style)", () => {
|
|
1911
|
+
assert.throws(
|
|
1912
|
+
() =>
|
|
1913
|
+
resolveCropEntry({ image_index: 0, normalized: { x: 1.0, y: 1.0, width: 0, height: 0 } }, 100, 100),
|
|
1914
|
+
/fractions of the image/,
|
|
1915
|
+
);
|
|
1916
|
+
});
|
|
1917
|
+
|
|
1918
|
+
it("zero-area pixel errors carry a recovery hint (pi 1.0 style)", () => {
|
|
1919
|
+
assert.throws(
|
|
1920
|
+
() => resolveCropEntry({ image_index: 0, pixels: { x: 200, y: 200, width: 10, height: 10 } }, 100, 100),
|
|
1921
|
+
/inside the image/,
|
|
1922
|
+
);
|
|
1923
|
+
});
|
|
1831
1924
|
});
|
|
1832
1925
|
|
|
1833
1926
|
describe("cropSignature", () => {
|
|
@@ -3480,6 +3573,109 @@ describe("1.16.0 downscaleForUpload (integration)", () => {
|
|
|
3480
3573
|
});
|
|
3481
3574
|
});
|
|
3482
3575
|
|
|
3576
|
+
describe("1.19.0 model image limits (pi ≥ 0.87 inputLimits.images.resize)", () => {
|
|
3577
|
+
it("modelImageResizeLimits returns null without a declaration", () => {
|
|
3578
|
+
assert.equal(modelImageResizeLimits(undefined), null);
|
|
3579
|
+
assert.equal(modelImageResizeLimits({}), null);
|
|
3580
|
+
assert.equal(modelImageResizeLimits({ inputLimits: {} }), null);
|
|
3581
|
+
assert.equal(modelImageResizeLimits({ inputLimits: { images: {} } }), null);
|
|
3582
|
+
// No numeric dimension → no usable target
|
|
3583
|
+
assert.equal(modelImageResizeLimits({ inputLimits: { images: { resize: { jpegQuality: 75 } } } }), null);
|
|
3584
|
+
});
|
|
3585
|
+
|
|
3586
|
+
it("modelImageResizeLimits derives a long-edge target, quality, and raw-byte budget", () => {
|
|
3587
|
+
const limits = modelImageResizeLimits({
|
|
3588
|
+
inputLimits: { images: { resize: { maxWidth: 1568, maxHeight: 1040, jpegQuality: 75, maxBytes: 524288 } } },
|
|
3589
|
+
});
|
|
3590
|
+
assert.equal(limits?.targetDim, 1040);
|
|
3591
|
+
assert.equal(limits?.quality, 75);
|
|
3592
|
+
assert.equal(limits?.maxBytes, 393216); // 524288 × 0.75
|
|
3593
|
+
const partial = modelImageResizeLimits({ inputLimits: { images: { resize: { maxWidth: 1568 } } } });
|
|
3594
|
+
assert.equal(partial?.targetDim, 1568);
|
|
3595
|
+
assert.equal(partial?.quality, 88);
|
|
3596
|
+
assert.equal(partial?.maxBytes, undefined);
|
|
3597
|
+
});
|
|
3598
|
+
|
|
3599
|
+
it("maxBytes-only declarations are honored — targetDim stays unset, budget and quality tighten (PR #34 review)", () => {
|
|
3600
|
+
const limits = modelImageResizeLimits({
|
|
3601
|
+
inputLimits: { images: { resize: { maxBytes: 524288, jpegQuality: 70 } } },
|
|
3602
|
+
});
|
|
3603
|
+
assert.notEqual(limits, null);
|
|
3604
|
+
assert.equal(limits?.targetDim, undefined);
|
|
3605
|
+
assert.equal(limits?.quality, 70);
|
|
3606
|
+
assert.equal(limits?.maxBytes, 393216);
|
|
3607
|
+
// A tiny maxBytes never collapses to a zero budget
|
|
3608
|
+
assert.equal(modelImageResizeLimits({ inputLimits: { images: { resize: { maxBytes: 1 } } } })?.maxBytes, 1);
|
|
3609
|
+
});
|
|
3610
|
+
|
|
3611
|
+
it("maxBytes-only limits trigger the byte budget on downscaleForUpload (PR #34 review)", async () => {
|
|
3612
|
+
const { Image } = await import("imagescript");
|
|
3613
|
+
const noise = new Image(200, 150);
|
|
3614
|
+
for (let x = 1; x <= 200; x++) {
|
|
3615
|
+
for (let y = 1; y <= 150; y++) {
|
|
3616
|
+
noise.setPixelAt(x, y, (Math.random() * 0xffffffff) >>> 0);
|
|
3617
|
+
}
|
|
3618
|
+
}
|
|
3619
|
+
const img = bufferToPiAiImage(Buffer.from(await noise.encode(1)), "image/png");
|
|
3620
|
+
// Dims are inside every dim threshold; only the model's byte budget
|
|
3621
|
+
// (2048 raw) is exceeded by the ~100 KB noise PNG → must re-encode.
|
|
3622
|
+
const out = await downscaleForUpload(
|
|
3623
|
+
img,
|
|
3624
|
+
{ maxUploadDim: 2048, maxUploadBytes: 5 * 1024 * 1024 },
|
|
3625
|
+
{ quality: 75, maxBytes: 2048 },
|
|
3626
|
+
);
|
|
3627
|
+
assert.equal(out.mimeType, "image/jpeg");
|
|
3628
|
+
});
|
|
3629
|
+
|
|
3630
|
+
it("model limits tighten downscaleForUpload below the user config", async () => {
|
|
3631
|
+
const { Image } = await import("imagescript");
|
|
3632
|
+
const big = bufferToPiAiImage(Buffer.from(await new Image(3000, 200).encode(1)), "image/png");
|
|
3633
|
+
const out = await downscaleForUpload(
|
|
3634
|
+
big,
|
|
3635
|
+
{ maxUploadDim: 2048, maxUploadBytes: 5 * 1024 * 1024 },
|
|
3636
|
+
{ targetDim: 1024, quality: 75 },
|
|
3637
|
+
);
|
|
3638
|
+
assert.equal(out.mimeType, "image/jpeg");
|
|
3639
|
+
const dims = extractDimensions(piAiImageToBuffer(out));
|
|
3640
|
+
assert.equal(dims?.width, 1024);
|
|
3641
|
+
});
|
|
3642
|
+
|
|
3643
|
+
it("model limits never re-enable a user-disabled downscale", async () => {
|
|
3644
|
+
const { Image } = await import("imagescript");
|
|
3645
|
+
const big = bufferToPiAiImage(Buffer.from(await new Image(3000, 200).encode(1)), "image/png");
|
|
3646
|
+
const out = await downscaleForUpload(
|
|
3647
|
+
big,
|
|
3648
|
+
{ maxUploadDim: 0, maxUploadBytes: 5 * 1024 * 1024 },
|
|
3649
|
+
{ targetDim: 1024, quality: 75 },
|
|
3650
|
+
);
|
|
3651
|
+
assert.equal(out, big);
|
|
3652
|
+
});
|
|
3653
|
+
|
|
3654
|
+
it("model limits under the user config do not shrink an in-budget image", async () => {
|
|
3655
|
+
const { Image } = await import("imagescript");
|
|
3656
|
+
const small = bufferToPiAiImage(Buffer.from(await new Image(800, 600).encode(1)), "image/png");
|
|
3657
|
+
const out = await downscaleForUpload(
|
|
3658
|
+
small,
|
|
3659
|
+
{ maxUploadDim: 2048, maxUploadBytes: 5 * 1024 * 1024 },
|
|
3660
|
+
{ targetDim: 1024, quality: 75 },
|
|
3661
|
+
);
|
|
3662
|
+
assert.equal(out, small);
|
|
3663
|
+
});
|
|
3664
|
+
|
|
3665
|
+
it("sanitize validates the 1.19.0 virtual-model fields", () => {
|
|
3666
|
+
assert.equal(sanitize({ ...DEFAULT_CONFIG }).virtualModel, "on");
|
|
3667
|
+
assert.equal(sanitize({ ...DEFAULT_CONFIG, virtualModel: "off" }).virtualModel, "off");
|
|
3668
|
+
assert.equal(sanitize({ ...DEFAULT_CONFIG, virtualModel: "bogus" as any }).virtualModel, "on");
|
|
3669
|
+
const pinned = sanitize({ ...DEFAULT_CONFIG, virtualBaseProvider: "x-ai", virtualBaseModelId: "some-model" });
|
|
3670
|
+
assert.equal(pinned.virtualBaseProvider, "xai");
|
|
3671
|
+
assert.equal(pinned.virtualBaseModelId, "some-model");
|
|
3672
|
+
// Both halves required — a lone provider is dropped
|
|
3673
|
+
const lone = sanitize({ ...DEFAULT_CONFIG, virtualBaseProvider: "zai" });
|
|
3674
|
+
assert.equal("virtualBaseProvider" in lone, false);
|
|
3675
|
+
assert.equal("virtualBaseModelId" in lone, false);
|
|
3676
|
+
});
|
|
3677
|
+
});
|
|
3678
|
+
|
|
3483
3679
|
describe("1.16.0 review fixes (PR #27)", () => {
|
|
3484
3680
|
it("createAbortError is abort-shaped with the exact 'aborted' message", () => {
|
|
3485
3681
|
const err = createAbortError();
|
|
@@ -16,6 +16,16 @@ describe("extractRecallToken", () => {
|
|
|
16
16
|
assert.deepEqual(extractRecallToken(["#"], 0, 1), { query: "", prefix: "#" });
|
|
17
17
|
});
|
|
18
18
|
|
|
19
|
+
it("matches a # token after leading whitespace (pi #10218 mirror)", () => {
|
|
20
|
+
// Pi 1.0 fixed slash-command autocomplete not triggering when the input
|
|
21
|
+
// starts with whitespace; the # recall provider must behave the same.
|
|
22
|
+
assert.deepEqual(extractRecallToken([" #shot"], 0, 7), {
|
|
23
|
+
query: "shot",
|
|
24
|
+
prefix: "#shot",
|
|
25
|
+
});
|
|
26
|
+
assert.deepEqual(extractRecallToken([" #"], 0, 4), { query: "", prefix: "#" });
|
|
27
|
+
});
|
|
28
|
+
|
|
19
29
|
it("matches a # token after whitespace with a partial query", () => {
|
|
20
30
|
assert.deepEqual(extractRecallToken(["zoom into #shot"], 0, 15), {
|
|
21
31
|
query: "shot",
|