pi-multimodal-proxy 1.19.1 → 1.21.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +21 -0
- package/README.md +25 -0
- package/extensions/__tests__/internal.test.ts +51 -0
- package/extensions/__tests__/tool-render.test.ts +250 -0
- package/extensions/internal.ts +239 -0
- package/extensions/vision-proxy.ts +553 -6
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,27 @@ All notable changes to this project will be documented in this file.
|
|
|
4
4
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
|
6
6
|
|
|
7
|
+
## [1.21.0] - 2026-10-03
|
|
8
|
+
|
|
9
|
+
### Added
|
|
10
|
+
|
|
11
|
+
- **Compact transcript rows via `pi.registerToolRenderer()` (Pi ≥ 1.0.1, feature-detected)** — `analyze_image` and `generate_image` calls now render as one compact, themed line each instead of the default JSON-args dump. The `analyze_image` call line shows the question (word-boundary truncated, surrogate-pair safe), image references (recall ids → `image="deadbeef…"`, file paths → file name, `+N more` beyond two shown), a crop summary (`bottom-right@0` / `25%,13% 50%×50%@1` / `128,64 256×256@0` / `crops ×N`), and a model override; `generate_image` shows the prompt and override. Result lines summarize `✓ provider/model · cached · 1.2s` for analyses and `✓ N images via provider/model · $0.0311 · image="<id8>…"` for generations, or `✗ <error>`; expanding a row reveals the full text content. Generated images still render inline from the result's image blocks (independent of `renderResult`), reliable across Kitty/Ghostty/WezTerm/Warp via pi 1.0.1's JPEG/WebP/GIF `Image` fixes.
|
|
12
|
+
- **Renderer coverage for unregistered tools** — registering through `registerToolRenderer` rather than inline `renderCall`/`renderResult` on the tool definitions means the compact rows also apply to calls replayed when the tool is *not registered* at render time: sessions resumed with the tool disabled (`/multimodal-proxy tool off`, mode off), transcript rows rendered before extension load, and HTML exports. Resolver chaining (`next() ?? {}`) preserves other extensions' renderers for the keys we don't set.
|
|
13
|
+
- **Tool-result `details` payloads** — both tools now return a display-sized `details` object (analyze: `ok`/`error`/`cached`/`provider`/`model`/`latencyMs`/`groundingFormat`, with the heavy `text`/`images` stripped; generate: the structured payload as-is — ids/dims/cost) that feeds the renderer and persists in the session log without ever reaching the model.
|
|
14
|
+
- Defensive renderer plumbing: pi-tui's `Text` (ANSI-aware wrapping) is loaded lazily with a width-truncating plain-lines fallback (ANSI-aware, rough wcwidth — pi's differential renderer stops on overwide lines) so an unresolvable package can never block extension load nor crash rendering; the expanded result view renders **every** text block, so a `generate_image` row on a text-only base model (where the `tool_result` hook replaces image blocks with description fences) still shows the per-image descriptions; renderers tolerate partial/garbage streaming args and missing result objects. (PR #37 review: CodeRabbit.)
|
|
15
|
+
- New `tool-render.test.ts` (24 tests: truncation with a surrogate-pair regression trap, ref/crop summarization, call and result lines, garbage tolerance, details stripping) plus a fake-host smoke script (`tools/smoke-renderers.mjs`, real ANSI theme) that loads the real extension module and exercises the resolver contract end-to-end — 540 tests total, all green; also verified loading under the real pi 1.0.1 RPC loader.
|
|
16
|
+
|
|
17
|
+
### Changed
|
|
18
|
+
|
|
19
|
+
- Nothing user-visible on pi < 1.0.1 hosts: registration is gated on `typeof pi.registerToolRenderer === "function"`, so older runtimes keep the default tool rendering with zero behavior change.
|
|
20
|
+
|
|
21
|
+
## [1.20.0] - 2026-10-02
|
|
22
|
+
|
|
23
|
+
### Added
|
|
24
|
+
|
|
25
|
+
- **`generate_image` tool (Pi ≥ 1.0, feature-detected)** — agent-facing image creation through `ModelRegistry.generateImages()` with the session's credentials and request-time auth; usage counts toward session cost. Parameters: `prompt` (1..4000 chars), optional `model` (`provider/model-id` of an image-generation model). Model resolution walks `getAvailableOfType("image")` (cached per session): without an override the first keyed image model wins; a mistyped override errors with recovery-style close matches and a malformed one gets the expected format (Pi 1.0 codemode-error philosophy). The prompt leaves only after the data-egress consent gate (`/multimodal-proxy consent yes` / `allowed-providers`). Results carry the generated image blocks (vision-capable base models see the pixels directly) plus a summary text naming each `image="<id>"` id, and an `outputSchema`-shaped `structuredContent` (provider, model, per-image id/mimeType/dimensions, `costUSD` when the provider reported usage; `usage` passed through on the tool result). Generated images are persisted in the per-session recall stores (`imageData`/`imageMeta`) with `generated-<hash8>.<ext>` filename hints, so the existing pipeline takes over: for text-only base models the `tool_result` hook auto-describes them via the vision model and appends `DescriptionEntry`s (making `#` recall work), and `analyze_image` can re-query or crop them by id regardless of base-model modality. Shares the per-turn tool-call budget with `analyze_image` (10/turn); annotated `readOnlyHint: false` (cost-incurring content creation) and namespaced `multimodal-proxy`. Registered only when the runtime exposes `generateImages` — pi < 1.0 never sees the tool.
|
|
26
|
+
- Five new tests for `resolveImageGenModelChoice` (default-first, exact override, close-match suggestions on typos, expected-format error, actionable hint when nothing is keyed) — 512 total, all green.
|
|
27
|
+
|
|
7
28
|
## [1.19.1] - 2026-10-02
|
|
8
29
|
|
|
9
30
|
### Changed
|
package/README.md
CHANGED
|
@@ -8,10 +8,35 @@ When **video or audio files** are detected, they are routed to a **multimodal mo
|
|
|
8
8
|
|
|
9
9
|
**YouTube links** are detected too: paste a URL (`youtube.com/watch?v=…`, `youtu.be/…`, `/shorts/…`, etc.) and the video is downloaded with [`yt-dlp`](https://github.com/yt-dlp/yt-dlp) and analyzed exactly like a local file.
|
|
10
10
|
|
|
11
|
+
**Compact tool rows (Pi ≥ 1.0.1)** — `analyze_image` and `generate_image` calls render as one tidy line each (`analyze_image "What does the legend say?" · 2 images (image="deadbeef…", shot.png) · crop bottom-right@0`) with themed result summaries (`✓ zai/glm-5.3-flash · cached · 1.2s`, `✓ 1 image via google/imagen-4 · $0.0311`), instead of the default JSON-args dump. Generated images still render inline from the result blocks; expand a row to see the full analysis fence.
|
|
12
|
+
|
|
13
|
+
## What's new in 1.21.0
|
|
14
|
+
|
|
15
|
+
Adopting Pi 1.0.1's new `pi.registerToolRenderer()` API:
|
|
16
|
+
|
|
17
|
+
- **Compact transcript rows for `analyze_image`** — the call line shows the question (truncated on a word boundary), the image refs (recall ids shortened to `image="deadbeef…"`, paths to their file name, `+N more` beyond two), a crop summary (`bottom-right@0`, `25%,13% 50%×50%@1`, `128,64 256×256@0`, or `crops ×N`), and any model override — instead of the raw JSON arguments. The result line summarizes `✓ provider/model · cached · latency` (or `✗ <error>`), and expanding the row reveals the full analysis fence.
|
|
18
|
+
- **Compact transcript rows for `generate_image`** — the call line shows the prompt (and any model override); the result line summarizes `✓ N images via provider/model · $cost · image="<id8>…"`. The generated images themselves keep rendering inline — pi draws image blocks from the result content next to the renderer output — so this is purely a better header/summary, now reliable across Kitty, Ghostty, WezTerm, and Warp thanks to pi 1.0.1's JPEG/WebP/GIF image fixes.
|
|
19
|
+
- **Works even when the tool isn't registered** — registering through `registerToolRenderer` (instead of inline `renderCall`/`renderResult` on the tool definitions) covers rows replayed from a session where the tool is now disabled (`/multimodal-proxy tool off`, mode off), resumed sessions, and HTML exports. `next()` chaining preserves renderers from other extensions for the keys we don't set.
|
|
20
|
+
- **Renderer state via tool `details`** — results now carry a display-sized `details` payload (provider/model, `cached`, `latencyMs`, image ids, `costUSD`; heavy fields stripped) that feeds the renderer but never reaches the model.
|
|
21
|
+
- Fully feature-gated (`typeof pi.registerToolRenderer === "function"`): pi < 1.0.1 hosts keep the default rendering, zero behavior change. pi-tui's `Text` component is loaded lazily with a plain-lines fallback so a missing package can never block extension load.
|
|
22
|
+
|
|
23
|
+
Also benefits from pi 1.0.1 host fixes (no code needed here): Anthropic prompt caches survive mid-session tool toggles (tools added/redefined mid-conversation are now defined inline), "model at capacity" errors are retried instead of ending the turn, and Bedrock Claude thinking-signature failures after tool/prompt changes are fixed.
|
|
24
|
+
|
|
25
|
+
## What's new in 1.20.0
|
|
26
|
+
|
|
27
|
+
Image generation, riding Pi 1.0's new `models.generateImages()` API:
|
|
28
|
+
|
|
29
|
+
- **`generate_image` tool** — agent-facing image creation with the session's image-generation models (feature-detected; silently absent on pi < 1.0). Prompt leaves only after the data-egress consent gate; the call reports usage/cost through the normal session accounting.
|
|
30
|
+
- **Generated images join the pipeline** — results return as image blocks (vision base models see the pixels) **and** are persisted in the recall stores, so a text-only base model gets an automatic vision-model description (same as any tool-returned image), `#` recall works once described, and `analyze_image` can re-query or crop them via their `image="<id>"` ids.
|
|
31
|
+
- **Codemode-friendly** — `outputSchema`-shaped `structuredContent` (provider, model, per-image ids/dimensions, `costUSD`) so scripts consume results programmatically; leans into Pi 1.0's leaner codemode with a compact description and recovery-style errors (close-match model suggestions, expected `provider/model-id` format, actionable `pi --login` hint when nothing is keyed).
|
|
32
|
+
- **Shared cost guard** — `generate_image` counts against the same per-turn tool-call budget as `analyze_image`.
|
|
33
|
+
|
|
11
34
|
## What's new in 1.19.0
|
|
12
35
|
|
|
13
36
|
Deep integration with recent Pi releases (0.87 / 0.99):
|
|
14
37
|
|
|
38
|
+
- **`generate_image` tool (Pi ≥ 1.0)** — the agent can now **create** images, not just understand them. Calls run through `models.generateImages()` with the session's credentials (usage counts toward session cost), the model defaults to the first keyed image-generation model (override per call with `model: "provider/model-id"`), results carry image blocks plus stable `image="<id>"` ids, and the existing pipeline takes over automatically: text-only base models get an auto-description via your vision model, and every generated image is recallable (`#`) and re-analyzable/croppable with `analyze_image` like any attachment. Data-egress consent gates the provider before the first call.
|
|
39
|
+
- **Pi 1.0 alignment (1.19.1)** — leaner `analyze_image` description (~40% smaller — codemode lists tools under a 3000-token budget and drops what doesn't fit), recovery-style errors that name close matches (model typos, recall ids) and expected shapes (crops), and a dynamic implicit-vision-model fallback via `getAvailableOfType()` when the curated chain is entirely unkeyed.
|
|
15
40
|
- **Virtual model — `Multimodal Auto`** (Pi ≥ 0.99, on by default): a new entry in `/model` that routes each request for you — turns whose pending input carries images go **natively** to your configured vision model (real pixels, no text fence), everything else stays on your base model (auto = whatever physical model handled the session before, or pin one with `/multimodal-proxy virtual-base`). Sticky routing keeps prompt caches valid within a turn; the footer shows the routed model and `/session` lists cost per physical model. Disable with `/multimodal-proxy virtual off`.
|
|
16
41
|
- **`analyze_image` speaks the 0.99 tool API** — errors now surface as real tool errors (`isError`) instead of fake text, results carry a machine-readable `structuredContent` (analysis text, provider/model, latency, grounding format, per-image ids/dimensions/crop) so **codemode scripts** can consume them programmatically, and the tool is annotated (`readOnlyHint`) and namespaced (`multimodal-proxy`) for codemode listings.
|
|
17
42
|
- **Model-aware upload downscale** (Pi ≥ 0.87) — when the configured vision model declares `inputLimits.images.resize` in its catalog entry, the proxy derives the downscale target, JPEG quality, and byte budget from it (tightening, never loosening, your `max-upload` settings). No more double-resizing against provider limits.
|
|
@@ -81,6 +81,7 @@ import {
|
|
|
81
81
|
selectVisionModels,
|
|
82
82
|
closeMatches,
|
|
83
83
|
pickDynamicVisionCandidate,
|
|
84
|
+
resolveImageGenModelChoice,
|
|
84
85
|
splitSubcommand,
|
|
85
86
|
stripImagePaths,
|
|
86
87
|
stripMediaPaths,
|
|
@@ -436,6 +437,56 @@ describe("pickDynamicVisionCandidate", () => {
|
|
|
436
437
|
});
|
|
437
438
|
});
|
|
438
439
|
|
|
440
|
+
describe("resolveImageGenModelChoice", () => {
|
|
441
|
+
const available = [
|
|
442
|
+
{ provider: "openrouter", modelId: "gpt-image-1" },
|
|
443
|
+
{ provider: "openrouter", modelId: "flux-schnell" },
|
|
444
|
+
{ provider: "google", modelId: "imagen-4" },
|
|
445
|
+
];
|
|
446
|
+
|
|
447
|
+
it("defaults to the first available model without an override", () => {
|
|
448
|
+
assert.deepEqual(resolveImageGenModelChoice(undefined, available), {
|
|
449
|
+
ok: true,
|
|
450
|
+
provider: "openrouter",
|
|
451
|
+
modelId: "gpt-image-1",
|
|
452
|
+
});
|
|
453
|
+
assert.deepEqual(resolveImageGenModelChoice("", available), {
|
|
454
|
+
ok: true,
|
|
455
|
+
provider: "openrouter",
|
|
456
|
+
modelId: "gpt-image-1",
|
|
457
|
+
});
|
|
458
|
+
});
|
|
459
|
+
|
|
460
|
+
it("accepts an exact override", () => {
|
|
461
|
+
assert.deepEqual(resolveImageGenModelChoice("google/imagen-4", available), {
|
|
462
|
+
ok: true,
|
|
463
|
+
provider: "google",
|
|
464
|
+
modelId: "imagen-4",
|
|
465
|
+
});
|
|
466
|
+
});
|
|
467
|
+
|
|
468
|
+
it("mistyped overrides carry recovery-style close matches", () => {
|
|
469
|
+
const out = resolveImageGenModelChoice("openrouter/gpt-imge-1", available);
|
|
470
|
+
assert.equal(out.ok, false);
|
|
471
|
+
if (!out.ok) {
|
|
472
|
+
assert.match(out.error, /not found/);
|
|
473
|
+
assert.ok(out.suggestions.includes("openrouter/gpt-image-1"));
|
|
474
|
+
}
|
|
475
|
+
});
|
|
476
|
+
|
|
477
|
+
it("rejects malformed model strings with the expected format", () => {
|
|
478
|
+
const out = resolveImageGenModelChoice("no-slash", available);
|
|
479
|
+
assert.equal(out.ok, false);
|
|
480
|
+
if (!out.ok) assert.match(out.error, /Expected format: provider\/model-id/);
|
|
481
|
+
});
|
|
482
|
+
|
|
483
|
+
it("errors with an actionable hint when nothing is keyed", () => {
|
|
484
|
+
const out = resolveImageGenModelChoice(undefined, []);
|
|
485
|
+
assert.equal(out.ok, false);
|
|
486
|
+
if (!out.ok) assert.match(out.error, /pi --login/);
|
|
487
|
+
});
|
|
488
|
+
});
|
|
489
|
+
|
|
439
490
|
describe("sanitizeProviderHeaders", () => {
|
|
440
491
|
it("drops null deletion markers", () => {
|
|
441
492
|
assert.deepEqual(
|
|
@@ -0,0 +1,250 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Unit tests for the tool-transcript rendering helpers (1.21.0).
|
|
3
|
+
*
|
|
4
|
+
* These back the pi ≥ 1.0.1 registerToolRenderer() integration: compact
|
|
5
|
+
* analyze_image / generate_image call and result lines. Pure data in, plain
|
|
6
|
+
* segments out — theming happens in vision-proxy.ts.
|
|
7
|
+
*
|
|
8
|
+
* Run:
|
|
9
|
+
* node --experimental-strip-types --test extensions/__tests__/tool-render.test.ts
|
|
10
|
+
*/
|
|
11
|
+
|
|
12
|
+
import { strict as assert } from "node:assert";
|
|
13
|
+
import { describe, it } from "node:test";
|
|
14
|
+
import {
|
|
15
|
+
analyzeDetailsForTranscript,
|
|
16
|
+
formatAnalyzeImageCall,
|
|
17
|
+
formatAnalyzeImageResult,
|
|
18
|
+
formatGenerateImageCall,
|
|
19
|
+
formatGenerateImageResult,
|
|
20
|
+
summarizeCrop,
|
|
21
|
+
summarizeForTranscript,
|
|
22
|
+
summarizeImageRef,
|
|
23
|
+
} from "../internal.ts";
|
|
24
|
+
|
|
25
|
+
const HASH32 = "deadbeefdeadbeefdeadbeefdeadbeef";
|
|
26
|
+
|
|
27
|
+
function plain(segments: ReadonlyArray<{ text: string }>): string {
|
|
28
|
+
return segments.map((s) => s.text).join("");
|
|
29
|
+
}
|
|
30
|
+
|
|
31
|
+
describe("summarizeForTranscript", () => {
|
|
32
|
+
it("passes short text through with collapsed whitespace", () => {
|
|
33
|
+
assert.equal(summarizeForTranscript(" hello world\n\tnext ", 50), "hello world next");
|
|
34
|
+
});
|
|
35
|
+
|
|
36
|
+
it("truncates long text on a word boundary", () => {
|
|
37
|
+
const out = summarizeForTranscript("alpha bravo charlie delta echo foxtrot golf hotel", 20);
|
|
38
|
+
assert.ok(out.length <= 20, `length ${out.length}: ${out}`);
|
|
39
|
+
assert.ok(out.endsWith("…"));
|
|
40
|
+
assert.ok(!out.endsWith(" …"), "no dangling space before ellipsis");
|
|
41
|
+
});
|
|
42
|
+
|
|
43
|
+
it("never splits a surrogate pair", () => {
|
|
44
|
+
const emoji = "😀".repeat(60); // each emoji is a surrogate pair
|
|
45
|
+
const out = summarizeForTranscript(emoji, 20); // odd max → cut lands mid-pair
|
|
46
|
+
assert.ok(out.length <= 20);
|
|
47
|
+
// No lone high surrogate anywhere: the string still round-trips as text.
|
|
48
|
+
// (With the guard removed, the hard cut leaves an unpaired high surrogate
|
|
49
|
+
// right before the ellipsis, which this regex detects.)
|
|
50
|
+
assert.ok(!/[\uD800-\uDBFF](?![\uDC00-\uDFFF])/.test(out));
|
|
51
|
+
});
|
|
52
|
+
});
|
|
53
|
+
|
|
54
|
+
describe("summarizeImageRef", () => {
|
|
55
|
+
it("shortens a quoted recall reference", () => {
|
|
56
|
+
assert.equal(summarizeImageRef(`image="${HASH32}"`), `image="deadbeef…"`);
|
|
57
|
+
});
|
|
58
|
+
|
|
59
|
+
it("shortens an unquoted recall reference and sha256:/#crop forms", () => {
|
|
60
|
+
assert.equal(summarizeImageRef(`image=${HASH32}`), `image="deadbeef…"`);
|
|
61
|
+
assert.equal(summarizeImageRef(`sha256:${HASH32}#x0y0`), `image="deadbeef…"`);
|
|
62
|
+
assert.equal(summarizeImageRef(HASH32), `image="deadbeef…"`);
|
|
63
|
+
});
|
|
64
|
+
|
|
65
|
+
it("reduces file paths to the file name (posix + windows)", () => {
|
|
66
|
+
assert.equal(summarizeImageRef("/home/u/pics/cat.png"), "cat.png");
|
|
67
|
+
assert.equal(summarizeImageRef("D:\\Users\\u\\Downloads\\shot (1).jpg"), "shot (1).jpg");
|
|
68
|
+
});
|
|
69
|
+
|
|
70
|
+
it("truncates very long file names", () => {
|
|
71
|
+
const long = "a".repeat(120) + ".png";
|
|
72
|
+
const out = summarizeImageRef(long);
|
|
73
|
+
assert.ok(out.length <= 48);
|
|
74
|
+
assert.ok(out.endsWith("…"));
|
|
75
|
+
});
|
|
76
|
+
|
|
77
|
+
it("leaves non-hash short junk as-is", () => {
|
|
78
|
+
assert.equal(summarizeImageRef("not-a-hash"), "not-a-hash");
|
|
79
|
+
});
|
|
80
|
+
});
|
|
81
|
+
|
|
82
|
+
describe("summarizeCrop", () => {
|
|
83
|
+
it("renders named regions with the image index", () => {
|
|
84
|
+
assert.equal(summarizeCrop({ image_index: 0, region: "bottom-right" }), "bottom-right@0");
|
|
85
|
+
});
|
|
86
|
+
|
|
87
|
+
it("renders normalized crops as percentages", () => {
|
|
88
|
+
assert.equal(
|
|
89
|
+
summarizeCrop({ image_index: 1, normalized: { x: 0.25, y: 0.133, width: 0.5, height: 0.5 } }),
|
|
90
|
+
"25%,13.3% 50%×50%@1",
|
|
91
|
+
);
|
|
92
|
+
});
|
|
93
|
+
|
|
94
|
+
it("renders pixel crops rounded", () => {
|
|
95
|
+
assert.equal(
|
|
96
|
+
summarizeCrop({ image_index: 0, pixels: { x: 128.4, y: 64.2, width: 256, height: 256 } }),
|
|
97
|
+
"128,64 256×256@0",
|
|
98
|
+
);
|
|
99
|
+
});
|
|
100
|
+
|
|
101
|
+
it("returns empty for unrecognized input", () => {
|
|
102
|
+
assert.equal(summarizeCrop(undefined), "");
|
|
103
|
+
assert.equal(summarizeCrop("nonsense"), "");
|
|
104
|
+
assert.equal(summarizeCrop({ image_index: 0 }), "crop@0");
|
|
105
|
+
});
|
|
106
|
+
});
|
|
107
|
+
|
|
108
|
+
describe("formatAnalyzeImageCall", () => {
|
|
109
|
+
it("builds a full call line", () => {
|
|
110
|
+
const segs = formatAnalyzeImageCall({
|
|
111
|
+
images: [`image="${HASH32}"`, "/tmp/chart.png"],
|
|
112
|
+
question: "What does the legend say?",
|
|
113
|
+
crop: [{ image_index: 0, region: "center" }],
|
|
114
|
+
model: "zai/glm-5.3-flash",
|
|
115
|
+
});
|
|
116
|
+
assert.equal(
|
|
117
|
+
plain(segs),
|
|
118
|
+
`"What does the legend say?" · 2 images (image="deadbeef…", chart.png) · crop center@0 · zai/glm-5.3-flash`,
|
|
119
|
+
);
|
|
120
|
+
assert.equal(segs[0]!.style, "accent");
|
|
121
|
+
});
|
|
122
|
+
|
|
123
|
+
it("collapses many images to two shown plus a count", () => {
|
|
124
|
+
const segs = formatAnalyzeImageCall({
|
|
125
|
+
images: ["/a.png", "/b.png", "/c.png", "/d.png"],
|
|
126
|
+
question: "q",
|
|
127
|
+
});
|
|
128
|
+
assert.ok(plain(segs).includes("4 images (a.png, b.png +2)"));
|
|
129
|
+
});
|
|
130
|
+
|
|
131
|
+
it("summarizes multiple crops as a count", () => {
|
|
132
|
+
const segs = formatAnalyzeImageCall({
|
|
133
|
+
images: ["/a.png", "/b.png"],
|
|
134
|
+
question: "q",
|
|
135
|
+
crop: [
|
|
136
|
+
{ image_index: 0, region: "top" },
|
|
137
|
+
{ image_index: 1, region: "bottom" },
|
|
138
|
+
],
|
|
139
|
+
});
|
|
140
|
+
assert.ok(plain(segs).includes("crops ×2"));
|
|
141
|
+
});
|
|
142
|
+
|
|
143
|
+
it("truncates long questions and tolerates garbage args", () => {
|
|
144
|
+
const long = formatAnalyzeImageCall({ images: ["/a.png"], question: "x".repeat(300) });
|
|
145
|
+
assert.ok(plain(long).length < 200);
|
|
146
|
+
|
|
147
|
+
assert.deepEqual(formatAnalyzeImageCall("junk"), [{ text: '"junk"', style: "muted" }]);
|
|
148
|
+
assert.deepEqual(formatAnalyzeImageCall(null), [{ text: "null", style: "muted" }]);
|
|
149
|
+
});
|
|
150
|
+
|
|
151
|
+
it("handles missing question/images defensively", () => {
|
|
152
|
+
const segs = formatAnalyzeImageCall({});
|
|
153
|
+
assert.equal(plain(segs), '""');
|
|
154
|
+
});
|
|
155
|
+
});
|
|
156
|
+
|
|
157
|
+
describe("formatGenerateImageCall", () => {
|
|
158
|
+
it("builds prompt + model line", () => {
|
|
159
|
+
const segs = formatGenerateImageCall({ prompt: "A red panda astronaut, watercolor", model: "google/imagen-4" });
|
|
160
|
+
assert.equal(plain(segs), '"A red panda astronaut, watercolor" · google/imagen-4');
|
|
161
|
+
});
|
|
162
|
+
|
|
163
|
+
it("prompt-only line and garbage fallback", () => {
|
|
164
|
+
assert.equal(plain(formatGenerateImageCall({ prompt: "cat" })), '"cat"');
|
|
165
|
+
assert.deepEqual(formatGenerateImageCall(42), [{ text: "42", style: "muted" }]);
|
|
166
|
+
});
|
|
167
|
+
});
|
|
168
|
+
|
|
169
|
+
describe("formatAnalyzeImageResult", () => {
|
|
170
|
+
it("summarizes a successful analysis", () => {
|
|
171
|
+
const segs = formatAnalyzeImageResult(
|
|
172
|
+
{ ok: true, provider: "zai", model: "glm-5.3-flash", cached: true, latencyMs: 1234 },
|
|
173
|
+
false,
|
|
174
|
+
);
|
|
175
|
+
assert.equal(plain(segs), "✓ zai/glm-5.3-flash · cached · 1.2s");
|
|
176
|
+
});
|
|
177
|
+
|
|
178
|
+
it("summarizes without optional fields and with model only", () => {
|
|
179
|
+
assert.equal(plain(formatAnalyzeImageResult({ ok: true }, false)), "✓");
|
|
180
|
+
assert.equal(plain(formatAnalyzeImageResult({ ok: true, model: "glm-5.3-flash" }, false)), "✓ glm-5.3-flash");
|
|
181
|
+
});
|
|
182
|
+
|
|
183
|
+
it("renders errors from details, isError flag, or a generic fallback", () => {
|
|
184
|
+
assert.equal(
|
|
185
|
+
plain(formatAnalyzeImageResult({ ok: false, error: "Error: consent required" }, true)),
|
|
186
|
+
"✗ Error: consent required",
|
|
187
|
+
);
|
|
188
|
+
// isError without details
|
|
189
|
+
assert.equal(plain(formatAnalyzeImageResult(undefined, true)), "✗ analysis failed");
|
|
190
|
+
// ok:false wins even without the flag (older rows)
|
|
191
|
+
assert.equal(plain(formatAnalyzeImageResult({ ok: false }, false)), "✗ analysis failed");
|
|
192
|
+
});
|
|
193
|
+
|
|
194
|
+
it("marks the summary segment styles", () => {
|
|
195
|
+
const segs = formatAnalyzeImageResult({ ok: true, provider: "zai", model: "m" }, false);
|
|
196
|
+
assert.equal(segs[0]!.style, "success");
|
|
197
|
+
assert.equal(segs[1]!.style, "muted");
|
|
198
|
+
});
|
|
199
|
+
});
|
|
200
|
+
|
|
201
|
+
describe("formatGenerateImageResult", () => {
|
|
202
|
+
it("summarizes generation with images, cost, and first id", () => {
|
|
203
|
+
const segs = formatGenerateImageResult(
|
|
204
|
+
{
|
|
205
|
+
ok: true,
|
|
206
|
+
provider: "google",
|
|
207
|
+
model: "imagen-4",
|
|
208
|
+
images: [
|
|
209
|
+
{ id: HASH32, mimeType: "image/png", width: 1024, height: 768 },
|
|
210
|
+
{ id: "f".repeat(32), mimeType: "image/png", width: 1024, height: 768 },
|
|
211
|
+
],
|
|
212
|
+
costUSD: 0.0311,
|
|
213
|
+
},
|
|
214
|
+
false,
|
|
215
|
+
);
|
|
216
|
+
assert.equal(plain(segs), `✓ 2 images via google/imagen-4 · $0.0311 · image="deadbeef…" +1`);
|
|
217
|
+
});
|
|
218
|
+
|
|
219
|
+
it("renders errors from details or generically", () => {
|
|
220
|
+
assert.equal(
|
|
221
|
+
plain(formatGenerateImageResult({ ok: false, error: "Error: no keyed image model" }, true)),
|
|
222
|
+
"✗ Error: no keyed image model",
|
|
223
|
+
);
|
|
224
|
+
assert.equal(plain(formatGenerateImageResult(undefined, true)), "✗ generation failed");
|
|
225
|
+
});
|
|
226
|
+
|
|
227
|
+
it("falls back to 'done' when details carry no image list", () => {
|
|
228
|
+
assert.equal(plain(formatGenerateImageResult({ ok: true, provider: "google", model: "imagen-4" }, false)), "✓ done via google/imagen-4");
|
|
229
|
+
});
|
|
230
|
+
});
|
|
231
|
+
|
|
232
|
+
describe("analyzeDetailsForTranscript", () => {
|
|
233
|
+
it("strips text and images but keeps display fields", () => {
|
|
234
|
+
const out = analyzeDetailsForTranscript({
|
|
235
|
+
ok: true,
|
|
236
|
+
text: "a".repeat(2000),
|
|
237
|
+
images: [{ id: HASH32 }],
|
|
238
|
+
provider: "zai",
|
|
239
|
+
model: "glm-5.3-flash",
|
|
240
|
+
latencyMs: 120,
|
|
241
|
+
});
|
|
242
|
+
assert.deepEqual(out, { ok: true, provider: "zai", model: "glm-5.3-flash", latencyMs: 120 });
|
|
243
|
+
});
|
|
244
|
+
|
|
245
|
+
it("does not mutate the input", () => {
|
|
246
|
+
const input = { ok: true, text: "keep-me", images: [1, 2] };
|
|
247
|
+
analyzeDetailsForTranscript(input);
|
|
248
|
+
assert.deepEqual(input, { ok: true, text: "keep-me", images: [1, 2] });
|
|
249
|
+
});
|
|
250
|
+
});
|
package/extensions/internal.ts
CHANGED
|
@@ -2435,6 +2435,50 @@ export function pickDynamicVisionCandidate<T extends DynamicCandidateLike>(model
|
|
|
2435
2435
|
return vision.find((m) => CHEAP_MODEL_RE.test(m.modelId)) ?? vision[0];
|
|
2436
2436
|
}
|
|
2437
2437
|
|
|
2438
|
+
/** Outcome of image-generation model resolution (generate_image tool). */
|
|
2439
|
+
export type ImageGenModelChoice =
|
|
2440
|
+
| { ok: true; provider: string; modelId: string }
|
|
2441
|
+
| { ok: false; error: string; suggestions: readonly string[] };
|
|
2442
|
+
|
|
2443
|
+
/**
|
|
2444
|
+
* Resolve the image-generation model for one `generate_image` call (Pi ≥ 1.0).
|
|
2445
|
+
* An explicit `provider/model-id` override must match an available (keyed)
|
|
2446
|
+
* image model exactly — otherwise the error carries recovery-style close
|
|
2447
|
+
* matches. Without an override the first available model wins.
|
|
2448
|
+
*/
|
|
2449
|
+
export function resolveImageGenModelChoice(
|
|
2450
|
+
override: string | undefined,
|
|
2451
|
+
available: ReadonlyArray<{ provider: string; modelId: string }>,
|
|
2452
|
+
): ImageGenModelChoice {
|
|
2453
|
+
const labels = available.map((m) => `${m.provider}/${m.modelId}`);
|
|
2454
|
+
if (override === undefined || override === "") {
|
|
2455
|
+
const first = available[0];
|
|
2456
|
+
if (!first) {
|
|
2457
|
+
return {
|
|
2458
|
+
ok: false,
|
|
2459
|
+
error: "no image-generation model is available (keyed) in this session — add a provider key (e.g. pi --login openrouter)",
|
|
2460
|
+
suggestions: [],
|
|
2461
|
+
};
|
|
2462
|
+
}
|
|
2463
|
+
return { ok: true, provider: first.provider, modelId: first.modelId };
|
|
2464
|
+
}
|
|
2465
|
+
const parsed = parseModelString(override);
|
|
2466
|
+
if (!parsed) {
|
|
2467
|
+
return {
|
|
2468
|
+
ok: false,
|
|
2469
|
+
error: `invalid model string "${override}". Expected format: provider/model-id`,
|
|
2470
|
+
suggestions: [],
|
|
2471
|
+
};
|
|
2472
|
+
}
|
|
2473
|
+
const exact = available.find((m) => m.provider === parsed.provider && m.modelId === parsed.modelId);
|
|
2474
|
+
if (exact) return { ok: true, provider: exact.provider, modelId: exact.modelId };
|
|
2475
|
+
return {
|
|
2476
|
+
ok: false,
|
|
2477
|
+
error: `model "${parsed.provider}/${parsed.modelId}" not found among available image-generation models`,
|
|
2478
|
+
suggestions: closeMatches(`${parsed.provider}/${parsed.modelId}`, labels, 3),
|
|
2479
|
+
};
|
|
2480
|
+
}
|
|
2481
|
+
|
|
2438
2482
|
export function shouldStripImages(config: VisionConfig, modelInput: readonly string[] | undefined): boolean {
|
|
2439
2483
|
if (config.mode === "off") return false;
|
|
2440
2484
|
if (config.mode === "always") return true;
|
|
@@ -3219,6 +3263,201 @@ export function buildToolCacheKey(
|
|
|
3219
3263
|
return `${sortedHashes.join("+")}${cropSig ? "#crop:" + cropSig : ""}?q=${questionHash}&m=${modelId}`;
|
|
3220
3264
|
}
|
|
3221
3265
|
|
|
3266
|
+
// ── Tool transcript rendering (pi ≥ 1.0.1 registerToolRenderer) ────────────
|
|
3267
|
+
|
|
3268
|
+
/**
|
|
3269
|
+
* Styling hints for one segment of a tool-transcript line, mapped to
|
|
3270
|
+
* theme.fg tokens by the renderer in vision-proxy.ts. Kept as plain data so
|
|
3271
|
+
* these helpers stay pure and unit-testable without a terminal theme.
|
|
3272
|
+
*/
|
|
3273
|
+
export type TranscriptStyle = "plain" | "accent" | "muted" | "dim" | "success" | "warning" | "error";
|
|
3274
|
+
|
|
3275
|
+
export interface TranscriptSegment {
|
|
3276
|
+
text: string;
|
|
3277
|
+
style: TranscriptStyle;
|
|
3278
|
+
}
|
|
3279
|
+
|
|
3280
|
+
/** Collapse whitespace and truncate on a word boundary (surrogate-pair safe). */
|
|
3281
|
+
export function summarizeForTranscript(text: string, max: number): string {
|
|
3282
|
+
const t = text.replace(/\s+/g, " ").trim();
|
|
3283
|
+
if (t.length <= max) return t;
|
|
3284
|
+
let cut = t.slice(0, Math.max(1, max - 1));
|
|
3285
|
+
// Never split a surrogate pair on a hard cut (lone high surrogate → U+FFFD).
|
|
3286
|
+
if (/[\uD800-\uDBFF]$/.test(cut)) cut = cut.slice(0, -1);
|
|
3287
|
+
const ws = cut.lastIndexOf(" ");
|
|
3288
|
+
if (ws > cut.length * 0.6) cut = cut.slice(0, ws);
|
|
3289
|
+
return `${cut.trimEnd()}…`;
|
|
3290
|
+
}
|
|
3291
|
+
|
|
3292
|
+
/** Defensive finite-number check for values coming from model-authored args. */
|
|
3293
|
+
function isFiniteNumber(v: unknown): v is number {
|
|
3294
|
+
return typeof v === "number" && Number.isFinite(v);
|
|
3295
|
+
}
|
|
3296
|
+
|
|
3297
|
+
/** Compact percentage for crop summaries: integral values stay integral. */
|
|
3298
|
+
function pct(v: number): string {
|
|
3299
|
+
const p = v * 100;
|
|
3300
|
+
return Math.abs(p - Math.round(p)) < 0.05 ? String(Math.round(p)) : String(Math.round(p * 10) / 10);
|
|
3301
|
+
}
|
|
3302
|
+
|
|
3303
|
+
/** Safe one-line JSON rendering of an arbitrary value (circular-safe). */
|
|
3304
|
+
function safeJson(value: unknown, max = 120): string {
|
|
3305
|
+
try {
|
|
3306
|
+
return summarizeForTranscript(JSON.stringify(value) ?? String(value), max);
|
|
3307
|
+
} catch {
|
|
3308
|
+
return String(value);
|
|
3309
|
+
}
|
|
3310
|
+
}
|
|
3311
|
+
|
|
3312
|
+
/**
|
|
3313
|
+
* Shorten one analyze_image image reference for display: a recall id
|
|
3314
|
+
* (`image="<hash>"`, `sha256:<hash>#…`, or a bare hash) becomes
|
|
3315
|
+
* `image="deadbeef…"`; a file path becomes its file name (Windows or POSIX).
|
|
3316
|
+
*/
|
|
3317
|
+
export function summarizeImageRef(ref: string): string {
|
|
3318
|
+
const s = ref.trim().replace(/^["']|["']$/g, "");
|
|
3319
|
+
const quoted = /image="?([a-f0-9]{16,64})"?/i.exec(s);
|
|
3320
|
+
if (quoted) return `image="${quoted[1]!.slice(0, 8)}…"`;
|
|
3321
|
+
const bare = parseRecallRef(s);
|
|
3322
|
+
if (bare) return `image="${bare.slice(0, 8)}…"`;
|
|
3323
|
+
const base = s.replace(/\\/g, "/").split("/").filter(Boolean).pop() ?? s;
|
|
3324
|
+
return summarizeForTranscript(base, 48);
|
|
3325
|
+
}
|
|
3326
|
+
|
|
3327
|
+
/**
|
|
3328
|
+
* Compact crop descriptor for the transcript: `bottom-right@0`,
|
|
3329
|
+
* `25%,13% 50%×50%@1`, or `128,64 256×256@0`. Empty string when nothing
|
|
3330
|
+
* recognizable.
|
|
3331
|
+
*/
|
|
3332
|
+
export function summarizeCrop(crop: unknown): string {
|
|
3333
|
+
if (typeof crop !== "object" || crop === null) return "";
|
|
3334
|
+
const c = crop as Record<string, unknown>;
|
|
3335
|
+
const idx =
|
|
3336
|
+
isFiniteNumber(c.image_index) ? `@${Math.trunc(c.image_index)}` : "";
|
|
3337
|
+
if (typeof c.region === "string" && c.region.trim()) return `${c.region.trim()}${idx}`;
|
|
3338
|
+
const box = (c.normalized ?? c.pixels) as Record<string, unknown> | undefined;
|
|
3339
|
+
if (
|
|
3340
|
+
typeof box === "object" &&
|
|
3341
|
+
box !== null &&
|
|
3342
|
+
isFiniteNumber(box.x) &&
|
|
3343
|
+
isFiniteNumber(box.y) &&
|
|
3344
|
+
isFiniteNumber(box.width) &&
|
|
3345
|
+
isFiniteNumber(box.height)
|
|
3346
|
+
) {
|
|
3347
|
+
if (c.normalized !== undefined) {
|
|
3348
|
+
return `${pct(box.x)}%,${pct(box.y)}% ${pct(box.width)}%×${pct(box.height)}%${idx}`;
|
|
3349
|
+
}
|
|
3350
|
+
return `${Math.round(box.x)},${Math.round(box.y)} ${Math.round(box.width)}×${Math.round(box.height)}${idx}`;
|
|
3351
|
+
}
|
|
3352
|
+
return idx ? `crop${idx}` : "";
|
|
3353
|
+
}
|
|
3354
|
+
|
|
3355
|
+
/** One-line transcript summary of an analyze_image call (pi ≥ 1.0.1 renderer). */
|
|
3356
|
+
export function formatAnalyzeImageCall(args: unknown): TranscriptSegment[] {
|
|
3357
|
+
if (typeof args !== "object" || args === null || Array.isArray(args)) {
|
|
3358
|
+
return [{ text: safeJson(args), style: "muted" }];
|
|
3359
|
+
}
|
|
3360
|
+
const a = args as Record<string, unknown>;
|
|
3361
|
+
const out: TranscriptSegment[] = [];
|
|
3362
|
+
const question = typeof a.question === "string" ? a.question : "";
|
|
3363
|
+
out.push({ text: `"${summarizeForTranscript(question, 120)}"`, style: "accent" });
|
|
3364
|
+
const images = Array.isArray(a.images)
|
|
3365
|
+
? a.images.filter((r): r is string => typeof r === "string" && r.trim() !== "")
|
|
3366
|
+
: [];
|
|
3367
|
+
if (images.length > 0) {
|
|
3368
|
+
const shown = images.slice(0, 2).map(summarizeImageRef);
|
|
3369
|
+
const more = images.length - shown.length;
|
|
3370
|
+
out.push({
|
|
3371
|
+
text:
|
|
3372
|
+
` · ${images.length} image${images.length === 1 ? "" : "s"}` +
|
|
3373
|
+
(shown.length ? ` (${shown.join(", ")}${more > 0 ? ` +${more}` : ""})` : ""),
|
|
3374
|
+
style: "muted",
|
|
3375
|
+
});
|
|
3376
|
+
}
|
|
3377
|
+
const crops = Array.isArray(a.crop) ? a.crop.map(summarizeCrop).filter((s) => s !== "") : [];
|
|
3378
|
+
if (crops.length === 1) out.push({ text: ` · crop ${crops[0]}`, style: "dim" });
|
|
3379
|
+
else if (crops.length > 1) out.push({ text: ` · crops ×${crops.length}`, style: "dim" });
|
|
3380
|
+
const model = typeof a.model === "string" && a.model.trim() ? a.model.trim() : undefined;
|
|
3381
|
+
if (model) out.push({ text: ` · ${summarizeForTranscript(model, 60)}`, style: "muted" });
|
|
3382
|
+
return out;
|
|
3383
|
+
}
|
|
3384
|
+
|
|
3385
|
+
/** One-line transcript summary of a generate_image call (pi ≥ 1.0.1 renderer). */
|
|
3386
|
+
export function formatGenerateImageCall(args: unknown): TranscriptSegment[] {
|
|
3387
|
+
if (typeof args !== "object" || args === null || Array.isArray(args)) {
|
|
3388
|
+
return [{ text: safeJson(args), style: "muted" }];
|
|
3389
|
+
}
|
|
3390
|
+
const a = args as Record<string, unknown>;
|
|
3391
|
+
const prompt = typeof a.prompt === "string" ? a.prompt : "";
|
|
3392
|
+
const out: TranscriptSegment[] = [{ text: `"${summarizeForTranscript(prompt, 120)}"`, style: "accent" }];
|
|
3393
|
+
const model = typeof a.model === "string" && a.model.trim() ? a.model.trim() : undefined;
|
|
3394
|
+
if (model) out.push({ text: ` · ${summarizeForTranscript(model, 60)}`, style: "muted" });
|
|
3395
|
+
return out;
|
|
3396
|
+
}
|
|
3397
|
+
|
|
3398
|
+
function recordOf(details: unknown): Record<string, unknown> | undefined {
|
|
3399
|
+
return typeof details === "object" && details !== null && !Array.isArray(details)
|
|
3400
|
+
? (details as Record<string, unknown>)
|
|
3401
|
+
: undefined;
|
|
3402
|
+
}
|
|
3403
|
+
|
|
3404
|
+
/** One-line transcript summary of an analyze_image result from its `details`. */
|
|
3405
|
+
export function formatAnalyzeImageResult(details: unknown, isError: boolean): TranscriptSegment[] {
|
|
3406
|
+
const d = recordOf(details);
|
|
3407
|
+
if (isError || d?.ok === false) {
|
|
3408
|
+
const msg = typeof d?.error === "string" && d.error ? d.error : "analysis failed";
|
|
3409
|
+
return [{ text: `✗ ${summarizeForTranscript(msg, 160)}`, style: "error" }];
|
|
3410
|
+
}
|
|
3411
|
+
const segments: TranscriptSegment[] = [{ text: "✓", style: "success" }];
|
|
3412
|
+
const provider = typeof d?.provider === "string" && d.provider ? d.provider : undefined;
|
|
3413
|
+
const modelId = typeof d?.model === "string" && d.model ? d.model : undefined;
|
|
3414
|
+
if (provider && modelId) segments.push({ text: ` ${summarizeForTranscript(`${provider}/${modelId}`, 80)}`, style: "muted" });
|
|
3415
|
+
else if (modelId) segments.push({ text: ` ${summarizeForTranscript(modelId, 80)}`, style: "muted" });
|
|
3416
|
+
if (d?.cached === true) segments.push({ text: " · cached", style: "dim" });
|
|
3417
|
+
if (isFiniteNumber(d?.latencyMs)) {
|
|
3418
|
+
segments.push({ text: ` · ${(Math.max(0, d.latencyMs) / 1000).toFixed(1)}s`, style: "dim" });
|
|
3419
|
+
}
|
|
3420
|
+
return segments;
|
|
3421
|
+
}
|
|
3422
|
+
|
|
3423
|
+
/** One-line transcript summary of a generate_image result from its `details`. */
|
|
3424
|
+
export function formatGenerateImageResult(details: unknown, isError: boolean): TranscriptSegment[] {
|
|
3425
|
+
const d = recordOf(details);
|
|
3426
|
+
if (isError || d?.ok === false) {
|
|
3427
|
+
const msg = typeof d?.error === "string" && d.error ? d.error : "generation failed";
|
|
3428
|
+
return [{ text: `✗ ${summarizeForTranscript(msg, 160)}`, style: "error" }];
|
|
3429
|
+
}
|
|
3430
|
+
const images = Array.isArray(d?.images)
|
|
3431
|
+
? (d!.images as unknown[]).filter((x): x is Record<string, unknown> => typeof x === "object" && x !== null)
|
|
3432
|
+
: [];
|
|
3433
|
+
const provider = typeof d?.provider === "string" && d.provider ? d.provider : undefined;
|
|
3434
|
+
const modelId = typeof d?.model === "string" && d.model ? d.model : undefined;
|
|
3435
|
+
const count = images.length > 0 ? `${images.length} image${images.length === 1 ? "" : "s"}` : "done";
|
|
3436
|
+
const segments: TranscriptSegment[] =
|
|
3437
|
+
provider && modelId
|
|
3438
|
+
? [{ text: `✓ ${count} via ${summarizeForTranscript(`${provider}/${modelId}`, 80)}`, style: "success" }]
|
|
3439
|
+
: [{ text: `✓ ${count}`, style: "success" }];
|
|
3440
|
+
if (isFiniteNumber(d?.costUSD)) segments.push({ text: ` · $${Math.max(0, d.costUSD).toFixed(4)}`, style: "dim" });
|
|
3441
|
+
const firstId = typeof images[0]?.id === "string" && images[0].id ? images[0].id : undefined;
|
|
3442
|
+
if (firstId) {
|
|
3443
|
+
const more = images.length > 1 ? ` +${images.length - 1}` : "";
|
|
3444
|
+
segments.push({ text: ` · image="${firstId.slice(0, 8)}…"${more}`, style: "dim" });
|
|
3445
|
+
}
|
|
3446
|
+
return segments;
|
|
3447
|
+
}
|
|
3448
|
+
|
|
3449
|
+
/**
|
|
3450
|
+
* Copy of a structured analyze_image result without the fields the
|
|
3451
|
+
* transcript renderer never reads — `details` must stay small because it is
|
|
3452
|
+
* persisted in the session log alongside the (already stored) text fence.
|
|
3453
|
+
*/
|
|
3454
|
+
export function analyzeDetailsForTranscript(structured: Record<string, unknown>): Record<string, unknown> {
|
|
3455
|
+
const rest: Record<string, unknown> = { ...structured };
|
|
3456
|
+
delete rest.text;
|
|
3457
|
+
delete rest.images;
|
|
3458
|
+
return rest;
|
|
3459
|
+
}
|
|
3460
|
+
|
|
3222
3461
|
// ── Fence builders ────────────────────────────────────────────────────────
|
|
3223
3462
|
|
|
3224
3463
|
export interface VideoDescriptionEntry {
|
|
@@ -256,6 +256,12 @@ import {
|
|
|
256
256
|
buildVideoProxySection,
|
|
257
257
|
extractXaiResponsesText,
|
|
258
258
|
formatXaiSttTranscript,
|
|
259
|
+
analyzeDetailsForTranscript,
|
|
260
|
+
formatAnalyzeImageCall,
|
|
261
|
+
formatAnalyzeImageResult,
|
|
262
|
+
formatGenerateImageCall,
|
|
263
|
+
formatGenerateImageResult,
|
|
264
|
+
type TranscriptSegment,
|
|
259
265
|
isTranscriptionRequest,
|
|
260
266
|
isXaiProvider,
|
|
261
267
|
bufferToPiAiImage,
|
|
@@ -284,6 +290,7 @@ import {
|
|
|
284
290
|
applyDefaultModelFallback,
|
|
285
291
|
closeMatches,
|
|
286
292
|
pickDynamicVisionCandidate,
|
|
293
|
+
resolveImageGenModelChoice,
|
|
287
294
|
DEFAULT_MODEL_FALLBACKS,
|
|
288
295
|
applyRecallCompletion,
|
|
289
296
|
buildRecallItems,
|
|
@@ -469,6 +476,52 @@ const TOOL_DESCRIPTION = [
|
|
|
469
476
|
/** Maximum analyze_image tool calls per agent turn. Prevents cost runaway. */
|
|
470
477
|
const MAX_TOOL_CALLS_PER_TURN = 10;
|
|
471
478
|
|
|
479
|
+
/**
|
|
480
|
+
* generate_image parameters (Pi ≥ 1.0). Lean on purpose: pi 1.0's codemode
|
|
481
|
+
* lists tool declarations under a 3000-token inline budget.
|
|
482
|
+
*/
|
|
483
|
+
const GenerateImageParams = Type.Object({
|
|
484
|
+
prompt: Type.String({
|
|
485
|
+
description: "Required, non-empty, max 4000 chars. What to draw; include style and aspect-ratio cues the provider supports.",
|
|
486
|
+
}),
|
|
487
|
+
model: Type.Optional(
|
|
488
|
+
Type.String({
|
|
489
|
+
description: "Optional; provider/model-id of an image-generation model. Defaults to the first available (keyed) image model.",
|
|
490
|
+
}),
|
|
491
|
+
),
|
|
492
|
+
reason: Type.Optional(Type.String({ description: "Optional; logged for analytics only" })),
|
|
493
|
+
});
|
|
494
|
+
|
|
495
|
+
/**
|
|
496
|
+
* outputSchema of generate_image: codemode scripts receive this
|
|
497
|
+
* structuredContent instead of the text + image blocks.
|
|
498
|
+
*/
|
|
499
|
+
const GenerateImageOutput = Type.Object({
|
|
500
|
+
ok: Type.Boolean({ description: "Whether generation succeeded." }),
|
|
501
|
+
error: Type.Optional(Type.String({ description: "Failure message, present when ok is false." })),
|
|
502
|
+
provider: Type.Optional(Type.String({ description: "Provider that generated the images." })),
|
|
503
|
+
model: Type.Optional(Type.String({ description: "Model id that generated the images." })),
|
|
504
|
+
images: Type.Optional(
|
|
505
|
+
Type.Array(
|
|
506
|
+
Type.Object({
|
|
507
|
+
id: Type.String({ description: "Stable image id usable for recall (image=\"...\") and analyze_image." }),
|
|
508
|
+
mimeType: Type.String(),
|
|
509
|
+
width: Type.Optional(Type.Number()),
|
|
510
|
+
height: Type.Optional(Type.Number()),
|
|
511
|
+
}),
|
|
512
|
+
),
|
|
513
|
+
),
|
|
514
|
+
costUSD: Type.Optional(Type.Number({ description: "Reported generation cost, when the provider returned usage." })),
|
|
515
|
+
});
|
|
516
|
+
|
|
517
|
+
const GENERATE_IMAGE_DESCRIPTION = [
|
|
518
|
+
"Generate images with the session's image-generation models (Pi ≥ 1.0, request-time credentials).",
|
|
519
|
+
"",
|
|
520
|
+
"`prompt` describes what to draw; `model` optionally picks an image-generation model (provider/model-id — defaults to the first available).",
|
|
521
|
+
"Returns image blocks attached to this result plus stable `image=\"<id>\"` ids that analyze_image can re-analyze or crop like any other image.",
|
|
522
|
+
"Each call costs money — use it when the user asks for an image to be created, not speculatively.",
|
|
523
|
+
].join("\n");
|
|
524
|
+
|
|
472
525
|
/** Default tool-result cache size; resized to config.cacheSize on first use. */
|
|
473
526
|
const DEFAULT_TOOL_CACHE_SIZE = 50;
|
|
474
527
|
|
|
@@ -508,6 +561,8 @@ interface SessionState {
|
|
|
508
561
|
keyProbe?: Map<string, boolean>;
|
|
509
562
|
/** Cached dynamic (registry-keyed) vision candidate; null = none or runtime without the API (Pi ≥ 1.0). */
|
|
510
563
|
dynamicVisionProbe?: { provider: string; modelId: string } | null;
|
|
564
|
+
/** Cached available (keyed) image-generation models (Pi ≥ 1.0). */
|
|
565
|
+
imageGenModels?: Array<{ provider: string; modelId: string }>;
|
|
511
566
|
/** Last implicit-default substitution already notified this session. */
|
|
512
567
|
notifiedDefaultSub?: string;
|
|
513
568
|
/**
|
|
@@ -1741,6 +1796,188 @@ function recallSuggestions(imageData: ImageDataStore, imageMeta: ImageMetaStore,
|
|
|
1741
1796
|
return closeMatches(query, candidates, limit);
|
|
1742
1797
|
}
|
|
1743
1798
|
|
|
1799
|
+
/** Content block shapes a tool result may carry (text and/or image). */
|
|
1800
|
+
type ToolContentBlock = { type: "text"; text: string } | { type: "image"; data: string; mimeType: string };
|
|
1801
|
+
|
|
1802
|
+
/** Structured outcome of one generate_image call (1.20.0). */
|
|
1803
|
+
interface GenerateImageOutcome {
|
|
1804
|
+
content: ToolContentBlock[];
|
|
1805
|
+
isError?: boolean;
|
|
1806
|
+
structured?: Record<string, unknown>;
|
|
1807
|
+
usage?: unknown;
|
|
1808
|
+
}
|
|
1809
|
+
|
|
1810
|
+
/** Wrap a failure message as a GenerateImageOutcome. */
|
|
1811
|
+
function generationError(message: string): GenerateImageOutcome {
|
|
1812
|
+
return { content: [{ type: "text", text: message }], isError: true, structured: { ok: false, error: message } };
|
|
1813
|
+
}
|
|
1814
|
+
|
|
1815
|
+
/**
|
|
1816
|
+
* Available (keyed) image-generation models via getAvailableOfType("image")
|
|
1817
|
+
* (Pi ≥ 1.0, feature-detected). Cached per session; probe failures cache as
|
|
1818
|
+
* an empty list so a transient error cannot wedge the tool.
|
|
1819
|
+
*/
|
|
1820
|
+
async function availableImageGenModels(ctx: ExtensionContext): Promise<Array<{ provider: string; modelId: string }>> {
|
|
1821
|
+
const state = getSessionState(ctx);
|
|
1822
|
+
if (state.imageGenModels) return state.imageGenModels;
|
|
1823
|
+
let out: Array<{ provider: string; modelId: string }> = [];
|
|
1824
|
+
const reg = ctx.modelRegistry as ModelRegistry & {
|
|
1825
|
+
getAvailableOfType?: (type: string) => Promise<ReadonlyArray<unknown>>;
|
|
1826
|
+
};
|
|
1827
|
+
if (typeof reg.getAvailableOfType === "function") {
|
|
1828
|
+
try {
|
|
1829
|
+
const models = (await reg.getAvailableOfType("image")) as ReadonlyArray<{ provider?: unknown; id?: unknown }>;
|
|
1830
|
+
out = models
|
|
1831
|
+
.filter((m): m is { provider: string; id: string } => typeof m.provider === "string" && typeof m.id === "string")
|
|
1832
|
+
.map((m) => ({ provider: m.provider, modelId: m.id }));
|
|
1833
|
+
} catch {
|
|
1834
|
+
out = [];
|
|
1835
|
+
}
|
|
1836
|
+
}
|
|
1837
|
+
state.imageGenModels = out;
|
|
1838
|
+
return out;
|
|
1839
|
+
}
|
|
1840
|
+
|
|
1841
|
+
/** Minimal response shape of ModelRegistry.generateImages() (Pi ≥ 1.0). */
|
|
1842
|
+
interface GenerateImagesResponseLike {
|
|
1843
|
+
output?: ReadonlyArray<{ type?: unknown; data?: unknown; mimeType?: unknown }>;
|
|
1844
|
+
usage?: { costUSD?: unknown; inputTokens?: unknown; outputTokens?: unknown };
|
|
1845
|
+
stopReason?: unknown;
|
|
1846
|
+
errorMessage?: unknown;
|
|
1847
|
+
}
|
|
1848
|
+
|
|
1849
|
+
/**
|
|
1850
|
+
* Generate one or more images with the session's image-generation models
|
|
1851
|
+
* (Pi ≥ 1.0 `models.generateImages()`): resolve the model (explicit override
|
|
1852
|
+
* with recovery-style suggestions, else the first keyed image model), gate
|
|
1853
|
+
* on data-egress consent (the prompt leaves), call the registry, and return
|
|
1854
|
+
* image blocks the conversation can see. Generated images are persisted in
|
|
1855
|
+
* the recall stores, so analyze_image (and `#` recall once described) work
|
|
1856
|
+
* on them exactly like attached images — including when the base model sees
|
|
1857
|
+
* the pixels directly and the tool_result auto-description path never fires.
|
|
1858
|
+
*/
|
|
1859
|
+
async function handleGenerateImage(
|
|
1860
|
+
params: { prompt: string; model?: string; reason?: string },
|
|
1861
|
+
ctx: ExtensionContext,
|
|
1862
|
+
config: VisionConfig,
|
|
1863
|
+
): Promise<GenerateImageOutcome> {
|
|
1864
|
+
const prompt = typeof params.prompt === "string" ? params.prompt.trim() : "";
|
|
1865
|
+
if (!prompt) return generationError("Error: prompt is required and must be non-empty.");
|
|
1866
|
+
if (prompt.length > 4000) return generationError("Error: prompt must be at most 4000 characters.");
|
|
1867
|
+
|
|
1868
|
+
// Resolve the image-generation model.
|
|
1869
|
+
const available = await availableImageGenModels(ctx);
|
|
1870
|
+
const choice = resolveImageGenModelChoice(params.model, available);
|
|
1871
|
+
if (!choice.ok) {
|
|
1872
|
+
return generationError(
|
|
1873
|
+
`Error: ${choice.error}.` +
|
|
1874
|
+
(choice.suggestions.length > 0 ? ` Close matches: ${choice.suggestions.join(", ")}.` : ""),
|
|
1875
|
+
);
|
|
1876
|
+
}
|
|
1877
|
+
|
|
1878
|
+
// Data-egress consent: the prompt leaves for the image provider.
|
|
1879
|
+
const entries = ctx.sessionManager.getEntries();
|
|
1880
|
+
if (!hasConsent(entries, choice.provider, config.allowedProviders, config.deniedProviders)) {
|
|
1881
|
+
return generationError(
|
|
1882
|
+
`Error: consent required before sending the prompt to ${choice.provider}. Please tell the user to run the following command and then retry:\n\n/multimodal-proxy consent yes\n\n(To pre-consent this provider permanently: /multimodal-proxy allowed-providers add ${choice.provider})`,
|
|
1883
|
+
);
|
|
1884
|
+
}
|
|
1885
|
+
|
|
1886
|
+
// Fetch the ImageModel object for the chosen pair (sync registry lookups).
|
|
1887
|
+
const reg = ctx.modelRegistry as ModelRegistry & {
|
|
1888
|
+
getModelsOfType?: (type: string) => ReadonlyArray<unknown>;
|
|
1889
|
+
findOfType?: (type: string, provider: string, modelId: string) => unknown;
|
|
1890
|
+
generateImages?: (
|
|
1891
|
+
model: unknown,
|
|
1892
|
+
context: { input: Array<{ type: "text"; text: string }> },
|
|
1893
|
+
options?: { signal?: AbortSignal },
|
|
1894
|
+
) => Promise<GenerateImagesResponseLike>;
|
|
1895
|
+
};
|
|
1896
|
+
if (typeof reg.generateImages !== "function") {
|
|
1897
|
+
return generationError("Error: this pi runtime does not expose models.generateImages() (needs pi ≥ 1.0). Update pi.");
|
|
1898
|
+
}
|
|
1899
|
+
let imageModel: unknown;
|
|
1900
|
+
if (typeof reg.getModelsOfType === "function") {
|
|
1901
|
+
const all = reg.getModelsOfType("image") as ReadonlyArray<{ provider?: unknown; id?: unknown }>;
|
|
1902
|
+
imageModel = all.find(
|
|
1903
|
+
(m) =>
|
|
1904
|
+
typeof m.provider === "string" &&
|
|
1905
|
+
typeof m.id === "string" &&
|
|
1906
|
+
m.provider === choice.provider &&
|
|
1907
|
+
m.id === choice.modelId,
|
|
1908
|
+
);
|
|
1909
|
+
}
|
|
1910
|
+
if (!imageModel && typeof reg.findOfType === "function") {
|
|
1911
|
+
imageModel = reg.findOfType("image", choice.provider, choice.modelId);
|
|
1912
|
+
}
|
|
1913
|
+
if (!imageModel) {
|
|
1914
|
+
return generationError(
|
|
1915
|
+
`Error: image model "${choice.provider}/${choice.modelId}" could not be resolved in the registry.`,
|
|
1916
|
+
);
|
|
1917
|
+
}
|
|
1918
|
+
|
|
1919
|
+
let response: GenerateImagesResponseLike;
|
|
1920
|
+
try {
|
|
1921
|
+
response = await reg.generateImages(imageModel, { input: [{ type: "text", text: prompt }] }, { signal: ctx.signal });
|
|
1922
|
+
} catch (err) {
|
|
1923
|
+
return generationError(`Error: image generation failed: ${err instanceof Error ? err.message : String(err)}`);
|
|
1924
|
+
}
|
|
1925
|
+
if (response.stopReason === "aborted") return generationError("Error: generation was cancelled.");
|
|
1926
|
+
if (response.stopReason === "error") {
|
|
1927
|
+
return generationError(
|
|
1928
|
+
`Error: image generation failed: ${typeof response.errorMessage === "string" ? response.errorMessage : "unknown provider error"}`,
|
|
1929
|
+
);
|
|
1930
|
+
}
|
|
1931
|
+
|
|
1932
|
+
const blocks = (response.output ?? []).filter(
|
|
1933
|
+
(b): b is { type: "image"; data: string; mimeType: string } =>
|
|
1934
|
+
b.type === "image" && typeof b.data === "string" && typeof b.mimeType === "string",
|
|
1935
|
+
);
|
|
1936
|
+
if (blocks.length === 0) {
|
|
1937
|
+
return generationError(
|
|
1938
|
+
"Error: the model returned no images (text-only response). Try a different prompt or model.",
|
|
1939
|
+
);
|
|
1940
|
+
}
|
|
1941
|
+
|
|
1942
|
+
// Persist recall data so analyze_image (and # recall, once a description
|
|
1943
|
+
// exists) work on generated images even when the base model sees the
|
|
1944
|
+
// pixels directly and the tool_result auto-description path never fires.
|
|
1945
|
+
const { imageMeta, imageData } = getSessionState(ctx);
|
|
1946
|
+
const structuredImages: Array<Record<string, unknown>> = [];
|
|
1947
|
+
for (const b of blocks) {
|
|
1948
|
+
const hash = hashImageData(b.data);
|
|
1949
|
+
const ext = b.mimeType.split("/")[1]?.replace(/[^a-z0-9]/gi, "") || "png";
|
|
1950
|
+
storeImageMeta(imageMeta, hash, b.data, `generated-${hash.slice(0, 8)}.${ext}`);
|
|
1951
|
+
storeImageData(imageData, hash, b.data, b.mimeType);
|
|
1952
|
+
const dims = imageMeta.get(hash);
|
|
1953
|
+
structuredImages.push({
|
|
1954
|
+
id: hash,
|
|
1955
|
+
mimeType: b.mimeType,
|
|
1956
|
+
...(dims ? { width: dims.width, height: dims.height } : {}),
|
|
1957
|
+
});
|
|
1958
|
+
}
|
|
1959
|
+
|
|
1960
|
+
const costUSD = typeof response.usage?.costUSD === "number" ? response.usage.costUSD : undefined;
|
|
1961
|
+
const ids = structuredImages.map((i) => `image="${i.id}"`).join(", ");
|
|
1962
|
+
const text =
|
|
1963
|
+
`[generate_image] ${pluralImages(blocks.length)} generated via ${choice.provider}/${choice.modelId}` +
|
|
1964
|
+
(ids ? ` (${ids})` : "") +
|
|
1965
|
+
(costUSD !== undefined ? ` — cost $${costUSD.toFixed(4)}` : "") +
|
|
1966
|
+
". The images are attached as image blocks; their ids can be re-analyzed or cropped with analyze_image.";
|
|
1967
|
+
|
|
1968
|
+
return {
|
|
1969
|
+
content: [{ type: "text", text }, ...blocks],
|
|
1970
|
+
structured: {
|
|
1971
|
+
ok: true,
|
|
1972
|
+
provider: choice.provider,
|
|
1973
|
+
model: choice.modelId,
|
|
1974
|
+
images: structuredImages,
|
|
1975
|
+
...(costUSD !== undefined ? { costUSD } : {}),
|
|
1976
|
+
},
|
|
1977
|
+
...(response.usage ? { usage: response.usage } : {}),
|
|
1978
|
+
};
|
|
1979
|
+
}
|
|
1980
|
+
|
|
1744
1981
|
/**
|
|
1745
1982
|
* Answer a targeted analyze_image question about one or more images: resolve
|
|
1746
1983
|
* references (session recall ids or file paths), validate and apply crops,
|
|
@@ -2105,6 +2342,7 @@ interface EditorAutocompleteProvider {
|
|
|
2105
2342
|
|
|
2106
2343
|
export default function (pi: ExtensionAPI) {
|
|
2107
2344
|
let _toolRegistered = false;
|
|
2345
|
+
let _generateToolRegistered = false;
|
|
2108
2346
|
let _autocompleteRegistered = false;
|
|
2109
2347
|
|
|
2110
2348
|
/**
|
|
@@ -2149,6 +2387,227 @@ export default function (pi: ExtensionAPI) {
|
|
|
2149
2387
|
);
|
|
2150
2388
|
}
|
|
2151
2389
|
|
|
2390
|
+
// ── Tool transcript renderers (pi ≥ 1.0.1) ─────────────────────────────
|
|
2391
|
+
|
|
2392
|
+
/** Minimal structural shape of the TUI component the renderers return. */
|
|
2393
|
+
interface ComponentLike {
|
|
2394
|
+
render(width: number): string[];
|
|
2395
|
+
invalidate?(): void;
|
|
2396
|
+
}
|
|
2397
|
+
|
|
2398
|
+
/** Theme subset the renderers use (pi's Theme helpers). */
|
|
2399
|
+
interface RenderThemeLike {
|
|
2400
|
+
fg(token: string, text: string): string;
|
|
2401
|
+
bold(text: string): string;
|
|
2402
|
+
}
|
|
2403
|
+
|
|
2404
|
+
/** ToolRenderContext subset available to renderCall/renderResult. */
|
|
2405
|
+
interface RenderContextLike {
|
|
2406
|
+
isError?: boolean;
|
|
2407
|
+
isPartial?: boolean;
|
|
2408
|
+
expanded?: boolean;
|
|
2409
|
+
showImages?: boolean;
|
|
2410
|
+
}
|
|
2411
|
+
|
|
2412
|
+
/** AgentToolResult subset the result renderer receives. */
|
|
2413
|
+
interface RenderResultLike {
|
|
2414
|
+
content?: ReadonlyArray<{ type?: string; text?: string }>;
|
|
2415
|
+
details?: unknown;
|
|
2416
|
+
}
|
|
2417
|
+
|
|
2418
|
+
type ToolRenderersLike = {
|
|
2419
|
+
renderCall?: (args: unknown, theme: RenderThemeLike, context: RenderContextLike) => ComponentLike;
|
|
2420
|
+
renderResult?: (
|
|
2421
|
+
result: RenderResultLike,
|
|
2422
|
+
options: { expanded: boolean; isPartial: boolean },
|
|
2423
|
+
theme: RenderThemeLike,
|
|
2424
|
+
context: RenderContextLike,
|
|
2425
|
+
) => ComponentLike;
|
|
2426
|
+
};
|
|
2427
|
+
|
|
2428
|
+
/**
|
|
2429
|
+
* pi-tui's Text (ANSI-aware word wrap), loaded lazily so an unresolvable
|
|
2430
|
+
* package can never block extension load — the crash-loop lesson from
|
|
2431
|
+
* 1.16.0. Warmed at registration time, long before the first tool row
|
|
2432
|
+
* renders.
|
|
2433
|
+
*/
|
|
2434
|
+
let _textCtor: (new (text: string, paddingX?: number, paddingY?: number) => ComponentLike) | undefined;
|
|
2435
|
+
let _textLoadTried = false;
|
|
2436
|
+
|
|
2437
|
+
function preloadTextComponent(): void {
|
|
2438
|
+
if (_textLoadTried) return;
|
|
2439
|
+
_textLoadTried = true;
|
|
2440
|
+
void import("@earendil-works/pi-tui")
|
|
2441
|
+
.then((m) => {
|
|
2442
|
+
const ctor = (
|
|
2443
|
+
m as { Text?: new (text: string, paddingX?: number, paddingY?: number) => ComponentLike }
|
|
2444
|
+
).Text;
|
|
2445
|
+
if (ctor) _textCtor = ctor;
|
|
2446
|
+
})
|
|
2447
|
+
.catch(() => {
|
|
2448
|
+
// pi-tui unreachable — the plain-lines fallback below keeps rendering
|
|
2449
|
+
});
|
|
2450
|
+
}
|
|
2451
|
+
|
|
2452
|
+
/**
|
|
2453
|
+
* Fixed-lines component used only when pi-tui's Text cannot load.
|
|
2454
|
+
* Truncates each line to the render width (ANSI-aware, rough wcwidth) —
|
|
2455
|
+
* pi's differential renderer stops on "rendered line exceeds terminal
|
|
2456
|
+
* width", so an overwide fallback line is worse than an ugly one.
|
|
2457
|
+
*/
|
|
2458
|
+
function plainLines(text: string): ComponentLike {
|
|
2459
|
+
return {
|
|
2460
|
+
render: (width: number) => text.split("\n").map((line) => truncateAnsiLine(line, Math.max(1, width))),
|
|
2461
|
+
invalidate: () => {},
|
|
2462
|
+
};
|
|
2463
|
+
}
|
|
2464
|
+
|
|
2465
|
+
/** Rough wcwidth: 2 for the common wide ranges, 0 for combining marks, else 1. */
|
|
2466
|
+
function codePointWidth(cp: number): number {
|
|
2467
|
+
if (cp >= 0x0300 && cp <= 0x036f) return 0; // combining diacritics
|
|
2468
|
+
if (
|
|
2469
|
+
(cp >= 0x1100 && cp <= 0x115f) || // Hangul Jamo
|
|
2470
|
+
(cp >= 0x2e80 && cp <= 0xa4cf) || // CJK radicals … Yi
|
|
2471
|
+
(cp >= 0xac00 && cp <= 0xd7a3) || // Hangul syllables
|
|
2472
|
+
(cp >= 0xf900 && cp <= 0xfaff) || // CJK compat ideographs
|
|
2473
|
+
(cp >= 0xfe30 && cp <= 0xfe4f) ||
|
|
2474
|
+
(cp >= 0xff00 && cp <= 0xff60) ||
|
|
2475
|
+
(cp >= 0xffe0 && cp <= 0xffe6) ||
|
|
2476
|
+
(cp >= 0x1f300 && cp <= 0x1faff) || // emoji blocks
|
|
2477
|
+
(cp >= 0x20000 && cp <= 0x3fffd) // CJK ext
|
|
2478
|
+
) {
|
|
2479
|
+
return 2;
|
|
2480
|
+
}
|
|
2481
|
+
return 1;
|
|
2482
|
+
}
|
|
2483
|
+
|
|
2484
|
+
/** ANSI-preserving truncation of one styled line to a visible width. */
|
|
2485
|
+
function truncateAnsiLine(line: string, width: number): string {
|
|
2486
|
+
let out = "";
|
|
2487
|
+
let w = 0;
|
|
2488
|
+
let i = 0;
|
|
2489
|
+
while (i < line.length) {
|
|
2490
|
+
if (line[i] === "\x1b") {
|
|
2491
|
+
const m = /^\x1b\[[0-9;]*m/.exec(line.slice(i));
|
|
2492
|
+
const seq = m ? m[0] : line[i]!;
|
|
2493
|
+
out += seq;
|
|
2494
|
+
i += seq.length;
|
|
2495
|
+
continue;
|
|
2496
|
+
}
|
|
2497
|
+
const cp = line.codePointAt(i)!;
|
|
2498
|
+
const s = String.fromCodePoint(cp);
|
|
2499
|
+
const cw = codePointWidth(cp);
|
|
2500
|
+
if (w + cw > width) break;
|
|
2501
|
+
out += s;
|
|
2502
|
+
w += cw;
|
|
2503
|
+
i += s.length;
|
|
2504
|
+
}
|
|
2505
|
+
return out;
|
|
2506
|
+
}
|
|
2507
|
+
|
|
2508
|
+
function textComponent(text: string): ComponentLike {
|
|
2509
|
+
return _textCtor ? new _textCtor(text, 0, 0) : plainLines(text);
|
|
2510
|
+
}
|
|
2511
|
+
|
|
2512
|
+
function styleSegments(segments: readonly TranscriptSegment[], theme: RenderThemeLike): string {
|
|
2513
|
+
let out = "";
|
|
2514
|
+
for (const seg of segments) {
|
|
2515
|
+
out += seg.style === "plain" ? seg.text : theme.fg(seg.style, seg.text);
|
|
2516
|
+
}
|
|
2517
|
+
return out;
|
|
2518
|
+
}
|
|
2519
|
+
|
|
2520
|
+
function textBlocks(content: ReadonlyArray<{ type?: string; text?: string }> | undefined): string[] {
|
|
2521
|
+
if (!content) return [];
|
|
2522
|
+
const out: string[] = [];
|
|
2523
|
+
for (const block of content) {
|
|
2524
|
+
if (block?.type === "text" && typeof block.text === "string" && block.text) out.push(block.text);
|
|
2525
|
+
}
|
|
2526
|
+
return out;
|
|
2527
|
+
}
|
|
2528
|
+
|
|
2529
|
+
function toolTitle(name: string, theme: RenderThemeLike): string {
|
|
2530
|
+
return theme.fg("toolTitle", theme.bold(`${name} `));
|
|
2531
|
+
}
|
|
2532
|
+
|
|
2533
|
+
/** Compact result component: summary line, plus the full text content when expanded. */
|
|
2534
|
+
function resultComponent(
|
|
2535
|
+
summary: readonly TranscriptSegment[],
|
|
2536
|
+
result: RenderResultLike,
|
|
2537
|
+
options: { expanded: boolean },
|
|
2538
|
+
theme: RenderThemeLike,
|
|
2539
|
+
): ComponentLike {
|
|
2540
|
+
let text = styleSegments(summary, theme);
|
|
2541
|
+
if (options.expanded) {
|
|
2542
|
+
// All text blocks: a generate_image result on a text-only base model
|
|
2543
|
+
// carries the header plus per-image description fences that replaced
|
|
2544
|
+
// the image blocks — the first block alone would hide them.
|
|
2545
|
+
const body = textBlocks(result.content).join("\n\n");
|
|
2546
|
+
if (body) {
|
|
2547
|
+
// Re-apply the style per line: pi resets styling after every line.
|
|
2548
|
+
text += `\n${body.split("\n").map((line) => theme.fg("toolOutput", line)).join("\n")}`;
|
|
2549
|
+
}
|
|
2550
|
+
}
|
|
2551
|
+
return textComponent(text);
|
|
2552
|
+
}
|
|
2553
|
+
|
|
2554
|
+
/**
|
|
2555
|
+
* Register transcript renderers for analyze_image/generate_image (pi ≥
|
|
2556
|
+
* 1.0.1). Registered through pi.registerToolRenderer — instead of inline
|
|
2557
|
+
* renderCall/renderResult on the tool definitions — so the compact rows
|
|
2558
|
+
* also cover tools that are *not registered* at render time: calls
|
|
2559
|
+
* replayed from a session where the tool is now disabled (`tool off` /
|
|
2560
|
+
* mode off), resumed sessions rendered before extension load, and HTML
|
|
2561
|
+
* exports. `next() ?? {}` keeps renderers from later resolvers / the
|
|
2562
|
+
* registered tool for the keys we don't set. Generated images keep
|
|
2563
|
+
* rendering inline regardless: pi draws image blocks from the result
|
|
2564
|
+
* content next to the renderer output, independent of renderResult.
|
|
2565
|
+
*/
|
|
2566
|
+
function registerToolRenderers() {
|
|
2567
|
+
const api = pi as unknown as {
|
|
2568
|
+
registerToolRenderer?: (
|
|
2569
|
+
resolver: (toolName: string, next: () => ToolRenderersLike | undefined) => ToolRenderersLike | undefined,
|
|
2570
|
+
) => void;
|
|
2571
|
+
};
|
|
2572
|
+
if (typeof api.registerToolRenderer !== "function") return; // pi < 1.0.1
|
|
2573
|
+
preloadTextComponent();
|
|
2574
|
+
api.registerToolRenderer((toolName, next) => {
|
|
2575
|
+
if (toolName === "analyze_image") {
|
|
2576
|
+
return {
|
|
2577
|
+
...(next() ?? {}),
|
|
2578
|
+
renderCall: (args, theme) =>
|
|
2579
|
+
textComponent(toolTitle("analyze_image", theme) + styleSegments(formatAnalyzeImageCall(args), theme)),
|
|
2580
|
+
renderResult: (result, options, theme, context) => {
|
|
2581
|
+
if (options.isPartial) return textComponent(theme.fg("warning", "Analyzing…"));
|
|
2582
|
+
return resultComponent(
|
|
2583
|
+
formatAnalyzeImageResult(result?.details, context?.isError === true),
|
|
2584
|
+
result ?? {},
|
|
2585
|
+
options,
|
|
2586
|
+
theme,
|
|
2587
|
+
);
|
|
2588
|
+
},
|
|
2589
|
+
};
|
|
2590
|
+
}
|
|
2591
|
+
if (toolName === "generate_image") {
|
|
2592
|
+
return {
|
|
2593
|
+
...(next() ?? {}),
|
|
2594
|
+
renderCall: (args, theme) =>
|
|
2595
|
+
textComponent(toolTitle("generate_image", theme) + styleSegments(formatGenerateImageCall(args), theme)),
|
|
2596
|
+
renderResult: (result, options, theme, context) => {
|
|
2597
|
+
if (options.isPartial) return textComponent(theme.fg("warning", "Generating…"));
|
|
2598
|
+
return resultComponent(
|
|
2599
|
+
formatGenerateImageResult(result?.details, context?.isError === true),
|
|
2600
|
+
result ?? {},
|
|
2601
|
+
options,
|
|
2602
|
+
theme,
|
|
2603
|
+
);
|
|
2604
|
+
},
|
|
2605
|
+
};
|
|
2606
|
+
}
|
|
2607
|
+
return next();
|
|
2608
|
+
});
|
|
2609
|
+
}
|
|
2610
|
+
|
|
2152
2611
|
// ── Virtual model (pi ≥ 0.99) ─────────────────────────────────────────────
|
|
2153
2612
|
|
|
2154
2613
|
const VIRTUAL_PROVIDER = "multimodal-proxy";
|
|
@@ -2339,8 +2798,12 @@ export default function (pi: ExtensionAPI) {
|
|
|
2339
2798
|
// unreadable config file — session_start surfaces the error path
|
|
2340
2799
|
});
|
|
2341
2800
|
|
|
2801
|
+
// Transcript renderers for analyze_image/generate_image rows (pi ≥ 1.0.1).
|
|
2802
|
+
// Registered once at load; the resolver serves any later session state.
|
|
2803
|
+
registerToolRenderers();
|
|
2804
|
+
|
|
2342
2805
|
/** Register or unregister the analyze_image tool based on config. */
|
|
2343
|
-
function syncToolRegistration(config: VisionConfig) {
|
|
2806
|
+
function syncToolRegistration(config: VisionConfig, runtimeCtx?: ExtensionContext) {
|
|
2344
2807
|
const shouldHaveTool = config.mode !== "off" && config.tool === "on";
|
|
2345
2808
|
if (shouldHaveTool && !_toolRegistered) {
|
|
2346
2809
|
pi.registerTool({
|
|
@@ -2398,11 +2861,93 @@ export default function (pi: ExtensionAPI) {
|
|
|
2398
2861
|
// codemode scripts. Older hosts ignore both fields.
|
|
2399
2862
|
...(outcome.isError ? { isError: true } : {}),
|
|
2400
2863
|
...(outcome.structured ? { structuredContent: outcome.structured } : {}),
|
|
2864
|
+
// `details` feeds the transcript renderer (pi ≥ 1.0.1) without
|
|
2865
|
+
// the model seeing it; heavy fields (text, images) are stripped.
|
|
2866
|
+
...(outcome.structured ? { details: analyzeDetailsForTranscript(outcome.structured) } : {}),
|
|
2401
2867
|
};
|
|
2402
2868
|
},
|
|
2403
2869
|
});
|
|
2404
2870
|
_toolRegistered = true;
|
|
2405
2871
|
}
|
|
2872
|
+
// generate_image (Pi ≥ 1.0 only): registered alongside analyze_image
|
|
2873
|
+
// when the runtime exposes ModelRegistry.generateImages; silently
|
|
2874
|
+
// skipped on older hosts so the tool never appears there.
|
|
2875
|
+
if (
|
|
2876
|
+
shouldHaveTool &&
|
|
2877
|
+
!_generateToolRegistered &&
|
|
2878
|
+
runtimeCtx &&
|
|
2879
|
+
typeof (runtimeCtx.modelRegistry as ModelRegistry & { generateImages?: unknown }).generateImages ===
|
|
2880
|
+
"function"
|
|
2881
|
+
) {
|
|
2882
|
+
pi.registerTool({
|
|
2883
|
+
name: "generate_image",
|
|
2884
|
+
label: "Generate Image",
|
|
2885
|
+
description: GENERATE_IMAGE_DESCRIPTION,
|
|
2886
|
+
promptSnippet: "Generate images with the session's image-generation models",
|
|
2887
|
+
promptGuidelines: [
|
|
2888
|
+
"Use generate_image when the user explicitly asks to create an image; each call costs money.",
|
|
2889
|
+
'Generated images carry image="<id>" ids and can be re-analyzed or cropped with analyze_image like any other image.',
|
|
2890
|
+
],
|
|
2891
|
+
// Cost-incurring content creation: not read-only. openWorldHint — the
|
|
2892
|
+
// prompt leaves for the configured image provider.
|
|
2893
|
+
annotations: {
|
|
2894
|
+
readOnlyHint: false,
|
|
2895
|
+
destructiveHint: false,
|
|
2896
|
+
idempotentHint: false,
|
|
2897
|
+
openWorldHint: true,
|
|
2898
|
+
},
|
|
2899
|
+
namespace: {
|
|
2900
|
+
name: "multimodal-proxy",
|
|
2901
|
+
description: "pi-multimodal-proxy media analysis (targeted image re-analysis with crop and grounding)",
|
|
2902
|
+
},
|
|
2903
|
+
outputSchema: GenerateImageOutput,
|
|
2904
|
+
parameters: GenerateImageParams,
|
|
2905
|
+
execute: async (_toolCallId, params, _signal, _onUpdate, extCtx) => {
|
|
2906
|
+
const entries = extCtx.sessionManager.getEntries();
|
|
2907
|
+
const config = await withModelFallback(resolveConfig(entries, process.env, _fileConfig), extCtx);
|
|
2908
|
+
|
|
2909
|
+
// Runtime check - tool may have been disabled mid-session
|
|
2910
|
+
if (config.tool !== "on" || config.mode === "off") {
|
|
2911
|
+
return {
|
|
2912
|
+
content: [
|
|
2913
|
+
{
|
|
2914
|
+
type: "text" as const,
|
|
2915
|
+
text: "Error: generate_image tool is currently disabled. Use /multimodal-proxy tool on to enable.",
|
|
2916
|
+
},
|
|
2917
|
+
],
|
|
2918
|
+
isError: true,
|
|
2919
|
+
};
|
|
2920
|
+
}
|
|
2921
|
+
|
|
2922
|
+
// Share the per-turn cost-runaway budget with analyze_image.
|
|
2923
|
+
const state = getSessionState(extCtx);
|
|
2924
|
+
state.toolCallCount++;
|
|
2925
|
+
if (state.toolCallCount > MAX_TOOL_CALLS_PER_TURN) {
|
|
2926
|
+
return {
|
|
2927
|
+
content: [
|
|
2928
|
+
{
|
|
2929
|
+
type: "text" as const,
|
|
2930
|
+
text: `Error: tool call limit reached (${MAX_TOOL_CALLS_PER_TURN} per turn). Try again in the next turn.`,
|
|
2931
|
+
},
|
|
2932
|
+
],
|
|
2933
|
+
isError: true,
|
|
2934
|
+
};
|
|
2935
|
+
}
|
|
2936
|
+
|
|
2937
|
+
const outcome = await handleGenerateImage(params, extCtx, config);
|
|
2938
|
+
return {
|
|
2939
|
+
content: outcome.content,
|
|
2940
|
+
...(outcome.isError ? { isError: true } : {}),
|
|
2941
|
+
...(outcome.structured ? { structuredContent: outcome.structured } : {}),
|
|
2942
|
+
// `details` feeds the transcript renderer (pi ≥ 1.0.1); the
|
|
2943
|
+
// structured payload is already display-sized (ids, dims, cost).
|
|
2944
|
+
...(outcome.structured ? { details: outcome.structured } : {}),
|
|
2945
|
+
...(outcome.usage !== undefined ? { usage: outcome.usage } : {}),
|
|
2946
|
+
};
|
|
2947
|
+
},
|
|
2948
|
+
});
|
|
2949
|
+
_generateToolRegistered = true;
|
|
2950
|
+
}
|
|
2406
2951
|
// Note: Pi's extension API doesn't have unregisterTool - tool registration
|
|
2407
2952
|
// persists for the session. The tool's execute handler checks the current
|
|
2408
2953
|
// config at runtime and returns an error if disabled.
|
|
@@ -2418,6 +2963,8 @@ export default function (pi: ExtensionAPI) {
|
|
|
2418
2963
|
clearImageData(state.imageData);
|
|
2419
2964
|
state.compaction = undefined;
|
|
2420
2965
|
state.keyProbe = undefined;
|
|
2966
|
+
state.dynamicVisionProbe = undefined;
|
|
2967
|
+
state.imageGenModels = undefined;
|
|
2421
2968
|
state.notifiedDefaultSub = undefined;
|
|
2422
2969
|
|
|
2423
2970
|
_fileConfig = await readPersistentFile();
|
|
@@ -2427,7 +2974,7 @@ export default function (pi: ExtensionAPI) {
|
|
|
2427
2974
|
ctx.ui.setStatus("multimodal-proxy", steadyStatusText(config, ctx.modelRegistry));
|
|
2428
2975
|
|
|
2429
2976
|
// Register tool if enabled
|
|
2430
|
-
syncToolRegistration(config);
|
|
2977
|
+
syncToolRegistration(config, ctx);
|
|
2431
2978
|
|
|
2432
2979
|
// Register the multimodal-proxy/auto virtual model (pi ≥ 0.99)
|
|
2433
2980
|
syncVirtualModelRegistration(config);
|
|
@@ -3319,7 +3866,7 @@ export default function (pi: ExtensionAPI) {
|
|
|
3319
3866
|
next.mode === "off" ? "warning" : "info",
|
|
3320
3867
|
);
|
|
3321
3868
|
// Sync tool registration on mode change
|
|
3322
|
-
syncToolRegistration(resolveConfig(ctx.sessionManager.getEntries(), process.env, _fileConfig));
|
|
3869
|
+
syncToolRegistration(resolveConfig(ctx.sessionManager.getEntries(), process.env, _fileConfig), ctx);
|
|
3323
3870
|
return;
|
|
3324
3871
|
}
|
|
3325
3872
|
|
|
@@ -3731,7 +4278,7 @@ Use "*" or "all" to grant consent for all providers globally.`,
|
|
|
3731
4278
|
}
|
|
3732
4279
|
if (valueLower === "on") {
|
|
3733
4280
|
const next = writePersisted({ ...persisted, tool: "on" });
|
|
3734
|
-
syncToolRegistration(resolveConfig(ctx.sessionManager.getEntries(), process.env, _fileConfig));
|
|
4281
|
+
syncToolRegistration(resolveConfig(ctx.sessionManager.getEntries(), process.env, _fileConfig), ctx);
|
|
3735
4282
|
ctx.ui.notify(`[multimodal-proxy] analyze_image tool: ON`, "info");
|
|
3736
4283
|
return;
|
|
3737
4284
|
}
|
|
@@ -4472,7 +5019,7 @@ Use "*" or "all" to grant consent for all providers globally.`,
|
|
|
4472
5019
|
if (modeChoice !== "fallback" && modeChoice !== "always" && modeChoice !== "off") return;
|
|
4473
5020
|
const next = writePersisted({ ...persisted, mode: modeChoice });
|
|
4474
5021
|
ctx.ui.notify(`Mode set to: ${next.mode}`, "info");
|
|
4475
|
-
syncToolRegistration(resolveConfig(ctx.sessionManager.getEntries(), process.env, _fileConfig));
|
|
5022
|
+
syncToolRegistration(resolveConfig(ctx.sessionManager.getEntries(), process.env, _fileConfig), ctx);
|
|
4476
5023
|
return;
|
|
4477
5024
|
}
|
|
4478
5025
|
|
|
@@ -4501,7 +5048,7 @@ Use "*" or "all" to grant consent for all providers globally.`,
|
|
|
4501
5048
|
}
|
|
4502
5049
|
const nextTool = effective.tool === "on" ? "off" : "on";
|
|
4503
5050
|
writePersisted({ ...persisted, tool: nextTool });
|
|
4504
|
-
syncToolRegistration(resolveConfig(ctx.sessionManager.getEntries(), process.env, _fileConfig));
|
|
5051
|
+
syncToolRegistration(resolveConfig(ctx.sessionManager.getEntries(), process.env, _fileConfig), ctx);
|
|
4505
5052
|
ctx.ui.notify(`Tool: ${nextTool}`, nextTool === "on" ? "info" : "warning");
|
|
4506
5053
|
return;
|
|
4507
5054
|
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-multimodal-proxy",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.21.0",
|
|
4
4
|
"description": "Automatic image, video and audio description for any model in Pi. Routes media to a multimodal model and injects descriptions into context.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"pi-package"
|