pi-vision-handoff 0.4.0 โ 0.4.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +43 -27
- package/package.json +1 -1
- package/src/dataloader.ts +63 -34
- package/src/describer.ts +18 -10
- package/src/index.ts +1 -1
- package/src/usage.ts +2 -2
- package/vision-handoff.ts +17 -10
package/README.md
CHANGED
|
@@ -10,6 +10,8 @@ _Describe images with a vision model you pick, then feed the text to models that
|
|
|
10
10
|
[](https://www.npmjs.com/package/pi-vision-handoff)
|
|
11
11
|
[](./LICENSE)
|
|
12
12
|
|
|
13
|
+
<img src="https://raw.githubusercontent.com/monotykamary/pi-vision-handoff/main/assets/vision-handoff.jpg" alt="Vision Handoff picker โ an interactive TUI listing every model, vision-capable ones marked with an eye, to choose the describer for text-only models" width="820">
|
|
14
|
+
|
|
13
15
|
</div>
|
|
14
16
|
|
|
15
17
|
---
|
|
@@ -35,9 +37,9 @@ No `settings.json` touched. No per-provider glue. Pick a describer once and ever
|
|
|
35
37
|
## Features
|
|
36
38
|
|
|
37
39
|
- **๐ฎ Interactive picker** โ `/vision-handoff` opens a TUI listing every model, vision-capable ones first (๐), to choose your describer.
|
|
38
|
-
- **๐ผ๏ธ DataLoader-batched descriptions** โ the `read` tools are `load()` callers: N parallel reads coalesce into ONE batched vision call (dispatched after the
|
|
40
|
+
- **๐ผ๏ธ DataLoader-batched descriptions** โ the `read` tools are `load()` callers: N parallel reads coalesce into ONE batched vision call (dispatched via `setImmediate` after the poll phase, so reads completing together batch instead of splitting), awaited during the tool-result phase (free time) so the agent's next turn never blocks on the describer. Descriptions are ready before `context` fires, so the swap is a non-blocking cache hit.
|
|
39
41
|
- **๐งน Hides pi's "model does not support images" note** โ on read results the extension strips pi's `[Current model does not support imagesโฆ]` note from the text block (it's misleading once the handoff delivers the image's content as text), while keeping the image block for kitty inline rendering and `/resume`.
|
|
40
|
-
- **๐ Provider-agnostic** โ uses pi's own model execution machinery (`@earendil-works/pi-ai`'s `
|
|
42
|
+
- **๐ Provider-agnostic** โ uses pi's own model execution machinery (`@earendil-works/pi-ai`'s `completeSimple()`), so it works with any provider/configured model, including custom provider extensions.
|
|
41
43
|
- **๐ง Automatic targets** โ by default, handoff applies to *every* model that lacks native vision. Opt out with `/vision-handoff auto off`.
|
|
42
44
|
- **๐๏ธ Explicit overrides** โ force handoff for specific models (e.g. a weak vision model) with `/vision-handoff add`.
|
|
43
45
|
- **โก Pre-warmed at paste-enter** โ the moment you press enter, `before_agent_start` scans the prompt for pasted clipboard image temp-file paths (pi writes pasted images to `<tmpdir>/pi-clipboard-<uuid>.<ext>` and inserts the path as text), reads them, and kicks off the ONE batched vision call concurrent with the agent's first response โ so by the time the agent reads the files, they're already cache hits.
|
|
@@ -137,7 +139,8 @@ pi -e ./vision-handoff.ts
|
|
|
137
139
|
The extension implements the **Facebook DataLoader pattern** for image
|
|
138
140
|
descriptions. The `read` tools are the `load()` callers; a per-image cache
|
|
139
141
|
memoizes promises; all `load()` calls in the same execution frame coalesce
|
|
140
|
-
into ONE batched vision call, dispatched after the
|
|
142
|
+
into ONE batched vision call, dispatched via `setImmediate` after the poll
|
|
143
|
+
phase (so reads completing together batch instead of splitting into N calls).
|
|
141
144
|
|
|
142
145
|
โ before_agent_start
|
|
143
146
|
โข captures this turn's user prompt (shared by every image description)
|
|
@@ -160,10 +163,12 @@ into ONE batched vision call, dispatched after the microtask cascade settles.
|
|
|
160
163
|
โข pi runs `read` calls in parallel (Promise.all); each read's
|
|
161
164
|
tool_result handler calls `loadDescription(img)` for its image
|
|
162
165
|
blocks and AWAITS the shared batch
|
|
163
|
-
โข DataLoader: all `load()` calls in
|
|
164
|
-
ONE batch โ ONE `
|
|
165
|
-
|
|
166
|
-
|
|
166
|
+
โข DataLoader: all `load()` calls in one event-loop poll iteration
|
|
167
|
+
land in ONE batch โ ONE `completeSimple()` vision call for the whole read
|
|
168
|
+
set. `enqueuePostPromiseJob` schedules dispatch via `setImmediate`
|
|
169
|
+
(the check phase, which runs AFTER the whole poll phase โ not
|
|
170
|
+
`process.nextTick`, which would drain between the reads' I/O
|
|
171
|
+
callbacks and split them into N single-image calls)
|
|
167
172
|
โข awaits the shared batch โ runs the describer during the tool-result
|
|
168
173
|
phase (free time: the agent is just waiting for tool results), so
|
|
169
174
|
the batch is COMPLETE before `context` fires โ `context` is a
|
|
@@ -191,24 +196,35 @@ A single frame's image set is described with **one** vision-model call, not
|
|
|
191
196
|
one per image. `loadDescription(img)` is synchronous: on a cache miss it
|
|
192
197
|
pushes the image's key (hash) into the current batch and returns a memoized
|
|
193
198
|
promise; on a cache hit it returns the existing (in-flight or resolved)
|
|
194
|
-
promise. `enqueuePostPromiseJob` schedules dispatch
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
single
|
|
199
|
-
|
|
200
|
-
`
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
image
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
|
|
199
|
+
promise. `enqueuePostPromiseJob` schedules dispatch via `setImmediate` (the
|
|
200
|
+
check phase, after the whole poll phase AND after the microtask queue
|
|
201
|
+
drains), so every `load()` caller โ whether from sync code, a `.then`
|
|
202
|
+
cascade, or a separate I/O callback in the same poll iteration โ registers
|
|
203
|
+
its key before the single vision call fires. This is why N parallel reads
|
|
204
|
+
coalesce into one call rather than splitting into N single-image calls:
|
|
205
|
+
`setImmediate` defers past the entire poll phase, whereas DataLoader's
|
|
206
|
+
classic `process.nextTick` would drain between the reads' I/O callbacks and
|
|
207
|
+
fire a dispatch after the first read but before the second. The batched call
|
|
208
|
+
sends every uncached image in a single user message with per-image
|
|
209
|
+
`<<<IMAGE k>>> โฆ <<<END>>>` delimiters; the response is parsed back into
|
|
210
|
+
per-image descriptions (keyed by `sha256(mime + base64)` in the per-image
|
|
211
|
+
cache). If the vision model ignores the delimiter format and the batched
|
|
212
|
+
response can't be split, each unparsed image falls back to its own
|
|
213
|
+
single-image `completeSimple()` call **in parallel** (no delimiters to cooperate
|
|
214
|
+
with) โ descriptions still arrive together. Only when a per-image call
|
|
215
|
+
itself genuinely fails (auth, timeout, empty) does that image degrade to
|
|
216
|
+
`[Image: description unavailable]`; one bad image never voids the rest.
|
|
217
|
+
|
|
218
|
+
Because pi runs parallel `read` tool calls via `Promise.all` and fires
|
|
219
|
+
each read's `tool_result` event as that read's I/O completes (poll phase) โ
|
|
220
|
+
via `agent.afterToolCall` โ `finalizeExecutedToolCall` โ the loader's
|
|
221
|
+
`setImmediate` dispatch defers to the check phase AFTER the whole poll
|
|
222
|
+
iteration, so reads completing together share ONE batch โ ONE vision call,
|
|
223
|
+
all resolving together. Reads completing in separate poll iterations get
|
|
224
|
+
separate calls, but always in parallel, never sequential. (The per-image
|
|
225
|
+
cache also dedupes a duplicate `load()` of the same image in one frame:
|
|
226
|
+
dispatch resolves every callback by hash, so a second load whose first cache
|
|
227
|
+
entry was evicted mid-frame still resolves โ it never hangs.)
|
|
212
228
|
|
|
213
229
|
**Failures are never cached.**
|
|
214
230
|
|
|
@@ -252,7 +268,7 @@ off mid-stream); earlier sections had `<<<END>>>` delimiters and are complete.
|
|
|
252
268
|
it before the LLM call), so a slow describer would also make aborting a turn
|
|
253
269
|
slow โ pi has to wait for the transform to return. The hook therefore wires
|
|
254
270
|
the turn's abort signal (`ctx.signal`, the active run's `AbortController`)
|
|
255
|
-
into the describer's `
|
|
271
|
+
into the describer's `completeSimple()` call, so a user cancel kills the in-flight
|
|
256
272
|
vision request immediately. A deliberate abort is not warned about (it's
|
|
257
273
|
not a vision-model failure) and the LLM-bound payload is left untouched since
|
|
258
274
|
the turn is being torn down anyway.
|
|
@@ -266,7 +282,7 @@ the turn is being torn down anyway.
|
|
|
266
282
|
| `context` (all messages) | `{ type: "input_image", image_url: "data:โฆ" }` (OpenAI Responses) | detected by shape โ replaced with `{ type: "input_text", text }` |
|
|
267
283
|
| `context` (all messages) | `{ type: "image", source: { type: "base64", media_type, data } }` (Anthropic Messages) | detected by shape โ replaced with `{ type: "text", text }` |
|
|
268
284
|
|
|
269
|
-
The describer call itself goes through pi's normal model machinery (`
|
|
285
|
+
The describer call itself goes through pi's normal model machinery (`completeSimple()`),
|
|
270
286
|
**not** the agent event loop โ so it never re-triggers `context` (no recursion).
|
|
271
287
|
The `read` tool result keeps its image block untouched (kitty inline + `/resume`);
|
|
272
288
|
only the `context`-cloned LLM-bound payload has images swapped for text.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-vision-handoff",
|
|
3
|
-
"version": "0.4.
|
|
3
|
+
"version": "0.4.2",
|
|
4
4
|
"description": "Give text-only pi models vision โ describe images with a vision model you pick via an interactive picker, then hand off the text description to non-vision models",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"author": "Tom X Nguyen",
|
package/src/dataloader.ts
CHANGED
|
@@ -4,19 +4,23 @@
|
|
|
4
4
|
* `loadDescription(img)` returns a memoized Promise for the description and
|
|
5
5
|
* pushes the image's key (hash) into the CURRENT batch. All `load()` calls in
|
|
6
6
|
* the same execution frame (+ its microtask cascade) coalesce into ONE batch
|
|
7
|
-
* object. Dispatch is scheduled
|
|
8
|
-
*
|
|
9
|
-
*
|
|
10
|
-
* with its description.
|
|
7
|
+
* object. Dispatch is scheduled via `setImmediate` (see `enqueuePostPromiseJob`),
|
|
8
|
+
* so every load in the frame โ and every load from a separate I/O callback in
|
|
9
|
+
* the same poll iteration โ lands in the single batch before the ONE vision
|
|
10
|
+
* call fires. Each load()'s promise then resolves with its description.
|
|
11
11
|
*
|
|
12
12
|
* Mapping: the `read` tools are the `load()` callers. Their `tool_result`
|
|
13
|
-
* handler awaits the shared batch โ so N parallel reads
|
|
14
|
-
*
|
|
15
|
-
*
|
|
16
|
-
*
|
|
17
|
-
*
|
|
18
|
-
*
|
|
19
|
-
*
|
|
13
|
+
* handler awaits the shared batch โ so N parallel reads coalesce into the SAME
|
|
14
|
+
* single vision call and all resolve together, the descriptions landing in the
|
|
15
|
+
* tool results BEFORE the agent's next turn. pi fires each read's `tool_result`
|
|
16
|
+
* as that read's I/O completes (poll phase); the loader's `setImmediate`
|
|
17
|
+
* dispatch defers to the check phase, AFTER the whole poll iteration, so reads
|
|
18
|
+
* completing together (the common case for cached local files) land in ONE
|
|
19
|
+
* batch โ and reads completing in separate iterations get separate calls, but
|
|
20
|
+
* always in parallel, never sequential. The agent's tool-result wait is free
|
|
21
|
+
* time, so this adds zero latency to the critical path. `context` then sees
|
|
22
|
+
* text-described tool results (no image blocks to swap); any remaining images
|
|
23
|
+
* (user-attached, custom-injected) are cache hits.
|
|
20
24
|
*
|
|
21
25
|
* All mutable state (batch, cache, turn context) lives on the instance โ no
|
|
22
26
|
* module-level globals โ and the class implements `Disposable` so a `using`
|
|
@@ -43,7 +47,12 @@ export const UNAVAILABLE = `${IMAGE_PLACEHOLDER_PREFIX}description unavailable${
|
|
|
43
47
|
interface DescriptionBatch {
|
|
44
48
|
keys: string[];
|
|
45
49
|
imgs: ExtractedImage[];
|
|
46
|
-
|
|
50
|
+
/** One per `loadDescription()` call. A duplicate load (same hash, but its
|
|
51
|
+
* first cache entry was evicted mid-frame so it couldn't short-circuit on
|
|
52
|
+
* the cache) pushes a second callback for an existing key โ so
|
|
53
|
+
* `callbacks.length` can exceed `keys.length`. `dispatchBatch` resolves by
|
|
54
|
+
* hash so every callback is reached. */
|
|
55
|
+
callbacks: { hash: string; resolve: (v: string) => void; reject: (e: Error) => void }[];
|
|
47
56
|
}
|
|
48
57
|
|
|
49
58
|
/** Engine-provided resolver for the configured vision model. */
|
|
@@ -59,16 +68,21 @@ export interface LoaderDeps extends DescriberDeps {
|
|
|
59
68
|
resolveVisionModel: VisionModelResolver;
|
|
60
69
|
}
|
|
61
70
|
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
*
|
|
66
|
-
*
|
|
67
|
-
*
|
|
68
|
-
*
|
|
69
|
-
*
|
|
71
|
+
/** Defer `fn` to the next check phase (`setImmediate`), so every
|
|
72
|
+
* `loadDescription()` โ whether called from sync code, a microtask cascade,
|
|
73
|
+
* or a separate I/O callback in the same poll iteration โ coalesces into one
|
|
74
|
+
* batch before the single vision call fires.
|
|
75
|
+
*
|
|
76
|
+
* Why `setImmediate` and not DataLoader's classic `process.nextTick`:
|
|
77
|
+
* nextTick drains between I/O callbacks in the poll phase, so dispatch would
|
|
78
|
+
* fire after the first parallel `read`'s `tool_result` but before the
|
|
79
|
+
* second's โ splitting N reads into N single-image calls. `setImmediate` runs
|
|
80
|
+
* in the check phase, AFTER the whole poll phase, so all `tool_result`
|
|
81
|
+
* handlers that fire in one poll iteration land in ONE batch. The check phase
|
|
82
|
+
* also runs after the microtask queue drains, so loads issued from a `.then`
|
|
83
|
+
* cascade (e.g. the clipboard pre-warm) still coalesce. */
|
|
70
84
|
function enqueuePostPromiseJob(fn: () => void): void {
|
|
71
|
-
|
|
85
|
+
setImmediate(fn);
|
|
72
86
|
}
|
|
73
87
|
|
|
74
88
|
export class DescriptionLoader implements Disposable {
|
|
@@ -119,12 +133,14 @@ export class DescriptionLoader implements Disposable {
|
|
|
119
133
|
batch.imgs.push(img);
|
|
120
134
|
idx = batch.keys.length - 1;
|
|
121
135
|
} else {
|
|
122
|
-
// Same image loaded twice in the frame
|
|
123
|
-
//
|
|
136
|
+
// Same image loaded twice in the frame (the first load's cache entry was
|
|
137
|
+
// evicted mid-frame, else the second load would have short-circuited on
|
|
138
|
+
// the cache). Share the one key/image slot but give this caller its own
|
|
139
|
+
// promise โ dispatch resolves it by hash alongside the first caller's.
|
|
124
140
|
batch.imgs[idx] = img;
|
|
125
141
|
}
|
|
126
142
|
const promise = new Promise<string>((resolve, reject) => {
|
|
127
|
-
batch.callbacks.push({ resolve, reject });
|
|
143
|
+
batch.callbacks.push({ hash, resolve, reject });
|
|
128
144
|
});
|
|
129
145
|
const cfg = this.deps.getConfig();
|
|
130
146
|
if (this.cache.size >= cfg.cacheMax) {
|
|
@@ -161,7 +177,14 @@ export class DescriptionLoader implements Disposable {
|
|
|
161
177
|
/** Dispatch the current batch: ONE batched `runBatch` vision call for every
|
|
162
178
|
* key collected this frame, then resolve each load()'s promise with its
|
|
163
179
|
* description (or {@link UNAVAILABLE} on failure). Failures are evicted from
|
|
164
|
-
* the cache so the next turn re-attempts.
|
|
180
|
+
* the cache so the next turn re-attempts.
|
|
181
|
+
*
|
|
182
|
+
* Results are resolved BY HASH and fanned out to EVERY callback: a batch can
|
|
183
|
+
* hold more callbacks than keys when the same image was loaded twice in one
|
|
184
|
+
* frame (its first cache entry was evicted mid-frame, so the second load
|
|
185
|
+
* pushed a second callback for the same hash). Indexing callbacks by key
|
|
186
|
+
* would skip those duplicates and hang their promises; iterating callbacks
|
|
187
|
+
* and looking up each one's hash fans the one result to all of them. */
|
|
165
188
|
private async dispatchBatch(): Promise<void> {
|
|
166
189
|
this.dispatchScheduled = false;
|
|
167
190
|
const batch = this.batch;
|
|
@@ -172,10 +195,8 @@ export class DescriptionLoader implements Disposable {
|
|
|
172
195
|
return;
|
|
173
196
|
}
|
|
174
197
|
if (this.turnSignal?.aborted) {
|
|
175
|
-
for (
|
|
176
|
-
|
|
177
|
-
batch.callbacks[i].resolve(UNAVAILABLE);
|
|
178
|
-
}
|
|
198
|
+
for (const key of batch.keys) this.cache.delete(key);
|
|
199
|
+
for (const cb of batch.callbacks) cb.resolve(UNAVAILABLE);
|
|
179
200
|
return;
|
|
180
201
|
}
|
|
181
202
|
this.deps.setLastError(null); // clear before a fresh attempt
|
|
@@ -190,19 +211,27 @@ export class DescriptionLoader implements Disposable {
|
|
|
190
211
|
this.deps,
|
|
191
212
|
this.turnSignal,
|
|
192
213
|
);
|
|
214
|
+
// Build per-hash results, then fan them out to every callback. This reaches
|
|
215
|
+
// duplicate-hash callbacks that a key-indexed loop would have skipped (and
|
|
216
|
+
// left hanging).
|
|
217
|
+
const results = new Map<string, string>();
|
|
193
218
|
for (let i = 0; i < batch.keys.length; i++) {
|
|
194
|
-
const
|
|
219
|
+
const key = batch.keys[i];
|
|
220
|
+
const raw = parsed.get(key);
|
|
195
221
|
if (raw) {
|
|
196
222
|
const final = wrapDescription(raw, cfg);
|
|
197
|
-
|
|
198
|
-
|
|
223
|
+
results.set(key, final);
|
|
224
|
+
// Cache the resolved value so later loads (this frame or next) hit.
|
|
225
|
+
this.cache.set(key, Promise.resolve(final));
|
|
199
226
|
} else {
|
|
200
227
|
// Genuine failure โ do NOT cache; next turn re-attempts (and surfaces
|
|
201
228
|
// the real error). Resolve (not reject) with UNAVAILABLE to match the
|
|
202
229
|
// graceful-degradation contract and avoid unhandled rejections.
|
|
203
|
-
this.cache.delete(
|
|
204
|
-
batch.callbacks[i].resolve(UNAVAILABLE);
|
|
230
|
+
this.cache.delete(key);
|
|
205
231
|
}
|
|
206
232
|
}
|
|
233
|
+
for (const cb of batch.callbacks) {
|
|
234
|
+
cb.resolve(results.get(cb.hash) ?? UNAVAILABLE);
|
|
235
|
+
}
|
|
207
236
|
}
|
|
208
237
|
}
|
package/src/describer.ts
CHANGED
|
@@ -1,6 +1,14 @@
|
|
|
1
1
|
/**
|
|
2
|
-
* The vision describer: calls a vision-capable model via pi-ai's
|
|
3
|
-
* to produce text descriptions of images.
|
|
2
|
+
* The vision describer: calls a vision-capable model via pi-ai's
|
|
3
|
+
* `completeSimple()` to produce text descriptions of images.
|
|
4
|
+
*
|
|
5
|
+
* `completeSimple` (not `complete`) is the path that translates the `reasoning`
|
|
6
|
+
* ThinkingLevel into each provider's `reasoningEffort`/budget. `complete()` โ
|
|
7
|
+
* `stream()` reads only the pre-translated `reasoningEffort` field and silently
|
|
8
|
+
* drops a bare `reasoning`, so the describer's thinking setting would be a no-op
|
|
9
|
+
* through `complete()` โ the bug this swap fixes. The agent loop, SDK, and
|
|
10
|
+
* compaction all route thinking through `completeSimple`/`streamSimple` for the
|
|
11
|
+
* same reason.
|
|
4
12
|
*
|
|
5
13
|
* Two entry points:
|
|
6
14
|
* - {@link runBatch}: ONE batched call describing N images at once (the
|
|
@@ -13,13 +21,13 @@
|
|
|
13
21
|
* Resource lifetimes (fetch interceptor, timeout timer, turn-abort wire) are
|
|
14
22
|
* managed with the `using` keyword via the {@link Disposable} guards in
|
|
15
23
|
* `dispose.ts`, replacing the manual `try`/`finally` cleanup the old code
|
|
16
|
-
* carried. Disposing is lexical and exception-safe: a thrown `
|
|
24
|
+
* carried. Disposing is lexical and exception-safe: a thrown `completeSimple()`
|
|
17
25
|
* still tears down the timer, uninstalls the interceptor, and detaches the
|
|
18
26
|
* abort listener.
|
|
19
27
|
*/
|
|
20
28
|
|
|
21
29
|
import type { Api, ImageContent, Message, Model, TextContent, ThinkingLevel } from "@earendil-works/pi-ai";
|
|
22
|
-
import {
|
|
30
|
+
import { completeSimple } from "@earendil-works/pi-ai/compat";
|
|
23
31
|
import type { ModelRegistry } from "@earendil-works/pi-coding-agent";
|
|
24
32
|
import {
|
|
25
33
|
batchUserPrompt,
|
|
@@ -52,7 +60,7 @@ import { abortWireGuard, fetchInterceptorGuard, timeoutGuard, type AbortWire } f
|
|
|
52
60
|
* model (e.g. 262144 context โ 8192 reserve = 253952 output budget). */
|
|
53
61
|
const INPUT_RESERVE_TOKENS = 8192;
|
|
54
62
|
|
|
55
|
-
/** Resolve the `maxTokens` to pass to `
|
|
63
|
+
/** Resolve the `maxTokens` to pass to `completeSimple()` for a describer call.
|
|
56
64
|
*
|
|
57
65
|
* - A configured `cfg.maxTokens` wins (explicit cost/latency cap), clamped to
|
|
58
66
|
* fit the context window.
|
|
@@ -76,7 +84,7 @@ export function resolveMaxTokens(
|
|
|
76
84
|
return Math.max(1, Math.min(requested, cap));
|
|
77
85
|
}
|
|
78
86
|
|
|
79
|
-
/** Resolve the `reasoning` (thinking) level to pass to `
|
|
87
|
+
/** Resolve the `reasoning` (thinking) level to pass to `completeSimple()`, or
|
|
80
88
|
* `undefined` to leave thinking off.
|
|
81
89
|
*
|
|
82
90
|
* - Returns `undefined` when thinking is disabled in config.
|
|
@@ -108,7 +116,7 @@ export interface DescriberDeps {
|
|
|
108
116
|
/** The result of a batched describer call: per-image raw descriptions keyed by hash. */
|
|
109
117
|
export type BatchResult = Map<string, string>;
|
|
110
118
|
|
|
111
|
-
/** Describe N images with ONE batched `
|
|
119
|
+
/** Describe N images with ONE batched `completeSimple()` call and split the response
|
|
112
120
|
* back into per-image descriptions. Returns a map keyed by image hash; an
|
|
113
121
|
* image whose section failed to parse is omitted (the caller treats omission
|
|
114
122
|
* as "description unavailable"). On a genuine call failure (auth, abort,
|
|
@@ -159,7 +167,7 @@ export async function runBatch(
|
|
|
159
167
|
const describeCtx: DescribeContext = { energyReader: undefined };
|
|
160
168
|
try {
|
|
161
169
|
const response = await describeAls.run(describeCtx, async () =>
|
|
162
|
-
|
|
170
|
+
completeSimple(
|
|
163
171
|
visionModel,
|
|
164
172
|
{ systemPrompt, messages: [userMessage] },
|
|
165
173
|
{ apiKey: auth.apiKey, headers: auth.headers, signal: controller.signal, maxTokens, reasoning },
|
|
@@ -231,7 +239,7 @@ export async function runBatch(
|
|
|
231
239
|
}
|
|
232
240
|
}
|
|
233
241
|
|
|
234
|
-
/** Describe a single image with one `
|
|
242
|
+
/** Describe a single image with one `completeSimple()` call and return the RAW
|
|
235
243
|
* description (no envelope, no truncation). Returns null on any genuine
|
|
236
244
|
* failure (auth, abort/error, empty) so the caller only caches `UNAVAILABLE`
|
|
237
245
|
* when a real describer attempt failed. */
|
|
@@ -273,7 +281,7 @@ export async function describeSingle(
|
|
|
273
281
|
const describeCtx: DescribeContext = { energyReader: undefined };
|
|
274
282
|
try {
|
|
275
283
|
const response = await describeAls.run(describeCtx, async () =>
|
|
276
|
-
|
|
284
|
+
completeSimple(
|
|
277
285
|
visionModel,
|
|
278
286
|
{ systemPrompt, messages: [userMessage] },
|
|
279
287
|
{ apiKey: auth.apiKey, headers: auth.headers, signal: controller.signal, maxTokens, reasoning },
|
package/src/index.ts
CHANGED
|
@@ -38,7 +38,7 @@ export const DEFAULT_USER_PROMPT_PREFIX = "The user's request about this image:
|
|
|
38
38
|
export const IMAGE_PLACEHOLDER_PREFIX = "[Image: ";
|
|
39
39
|
export const IMAGE_PLACEHOLDER_SUFFIX = "]";
|
|
40
40
|
|
|
41
|
-
/** Marker appended to a description whose `
|
|
41
|
+
/** Marker appended to a description whose `completeSimple()` call ended with
|
|
42
42
|
* `stopReason: "length"` โ i.e. the vision model hit a token limit (either the
|
|
43
43
|
* configured `maxTokens` or the provider's hard output cap) before finishing.
|
|
44
44
|
* A truncated description is still useful, but the agent must not mistake it
|
package/src/usage.ts
CHANGED
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
*
|
|
4
4
|
* One record is produced per REAL describer provider call (cache hits emit
|
|
5
5
|
* nothing):
|
|
6
|
-
* - model + tokens: from
|
|
6
|
+
* - model + tokens: from completeSimple()'s AssistantMessage.usage
|
|
7
7
|
* - energy + cost + raw MCR/energy/cost payloads: from Neuralwatt SSE comment
|
|
8
8
|
* lines parsed out of the teed response body (readEnergyFromTee). Present
|
|
9
9
|
* ONLY when the vision model is a Neuralwatt model โ non-Neuralwatt models
|
|
@@ -197,7 +197,7 @@ export function buildUsageRecord(
|
|
|
197
197
|
|
|
198
198
|
// โโ Concurrency-safe fetch interceptor โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
199
199
|
//
|
|
200
|
-
// The only body-interception point for
|
|
200
|
+
// The only body-interception point for completeSimple() is globalThis.fetch (pi-ai's
|
|
201
201
|
// StreamOptions.onResponse exposes headers only, not the body where the
|
|
202
202
|
// `: energy` SSE comments live). before_agent_start fires several describeImage()
|
|
203
203
|
// calls fire-and-forget, so a naรฏve save/patch/restore of globalThis.fetch would
|
package/vision-handoff.ts
CHANGED
|
@@ -210,7 +210,8 @@ export default function (pi: ExtensionAPI) {
|
|
|
210
210
|
//
|
|
211
211
|
// Both sources flow through the same loader: `load()` is synchronous and
|
|
212
212
|
// memoized, so all images in this frame (attached + clipboard-path)
|
|
213
|
-
// coalesce into ONE batch dispatched after the
|
|
213
|
+
// coalesce into ONE batch dispatched via `setImmediate` after the
|
|
214
|
+
// microtask cascade settles.
|
|
214
215
|
for (const image of event.images ?? []) {
|
|
215
216
|
if (!image || image.type !== "image" || !image.data) continue;
|
|
216
217
|
loader.loadDescription({ data: image.data, mimeType: image.mimeType || "image/png" }).catch(() => {});
|
|
@@ -244,12 +245,14 @@ export default function (pi: ExtensionAPI) {
|
|
|
244
245
|
// When the agent reads image files, this fires for each read result. It
|
|
245
246
|
// calls the loader's `loadDescription(img)` for every image block and
|
|
246
247
|
// AWAITS the shared batch โ so N parallel reads (pi runs `read` via
|
|
247
|
-
// Promise.all) coalesce into ONE batched vision call
|
|
248
|
-
//
|
|
249
|
-
//
|
|
250
|
-
//
|
|
251
|
-
//
|
|
252
|
-
//
|
|
248
|
+
// Promise.all) coalesce into ONE batched vision call: pi fires each read's
|
|
249
|
+
// `tool_result` as its I/O completes (poll phase), and the loader's
|
|
250
|
+
// `setImmediate` dispatch defers to the check phase AFTER the whole poll
|
|
251
|
+
// iteration, so reads completing together land in ONE batch and all resolve
|
|
252
|
+
// together. The descriptions replace the image blocks in the returned
|
|
253
|
+
// `content`, so by the time the agent's next turn starts the tool results
|
|
254
|
+
// already carry text โ the agent never sees raw image blocks it can't
|
|
255
|
+
// process.
|
|
253
256
|
//
|
|
254
257
|
// Why block here and not in `context`: the tool-result phase is free time
|
|
255
258
|
// (the agent is just waiting for tool results), so running the describer
|
|
@@ -277,9 +280,13 @@ export default function (pi: ExtensionAPI) {
|
|
|
277
280
|
}
|
|
278
281
|
|
|
279
282
|
// load() each image โ synchronous calls that push into the current batch
|
|
280
|
-
// and return memoized promises โ then await them all.
|
|
281
|
-
//
|
|
282
|
-
//
|
|
283
|
+
// and return memoized promises โ then await them all. pi fires each read's
|
|
284
|
+
// `tool_result` event as that read's I/O completes (poll phase); the
|
|
285
|
+
// loader's `setImmediate` dispatch defers to the check phase, AFTER the
|
|
286
|
+
// whole poll iteration, so reads completing together (the common case for
|
|
287
|
+
// cached local files) land in ONE batch โ ONE vision call for the whole
|
|
288
|
+
// read set, not N. Reads completing in separate iterations get separate
|
|
289
|
+
// calls, but always in parallel, never sequential. Awaiting here runs the
|
|
283
290
|
// describer during the tool-result phase (free time โ the agent is just
|
|
284
291
|
// waiting for tool results), so the batch is COMPLETE before `context`
|
|
285
292
|
// fires, making `context` a non-blocking cache hit instead of a cold miss
|