pi-vision-handoff 0.4.0 → 0.4.1

package/README.md

@@ -10,6 +10,8 @@ _Describe images with a vision model you pick, then feed the text to models that
 [![npm](https://img.shields.io/npm/v/pi-vision-handoff)](https://www.npmjs.com/package/pi-vision-handoff)
 [![license](https://img.shields.io/badge/license-MIT-blue)](./LICENSE)
 
+<img src="https://raw.githubusercontent.com/monotykamary/pi-vision-handoff/main/assets/vision-handoff.jpg" alt="Vision Handoff picker — an interactive TUI listing every model, vision-capable ones marked with an eye, to choose the describer for text-only models" width="820">
+
 </div>
 
 ---
@@ -35,7 +37,7 @@ No `settings.json` touched. No per-provider glue. Pick a describer once and ever
 ## Features
 
 - **🎮 Interactive picker** — `/vision-handoff` opens a TUI listing every model, vision-capable ones first (👁), to choose your describer.
-- **🖼️ DataLoader-batched descriptions** — the `read` tools are `load()` callers: N parallel reads coalesce into ONE batched vision call (dispatched after the microtask cascade settles), awaited during the tool-result phase (free time) so the agent's next turn never blocks on the describer. Descriptions are ready before `context` fires, so the swap is a non-blocking cache hit.
+- **🖼️ DataLoader-batched descriptions** — the `read` tools are `load()` callers: N parallel reads coalesce into ONE batched vision call (dispatched via `setImmediate` after the poll phase, so reads completing together batch instead of splitting), awaited during the tool-result phase (free time) so the agent's next turn never blocks on the describer. Descriptions are ready before `context` fires, so the swap is a non-blocking cache hit.
 - **🧹 Hides pi's "model does not support images" note** — on read results the extension strips pi's `[Current model does not support images…]` note from the text block (it's misleading once the handoff delivers the image's content as text), while keeping the image block for kitty inline rendering and `/resume`.
 - **🔌 Provider-agnostic** — uses pi's own model execution machinery (`@earendil-works/pi-ai`'s `complete()`), so it works with any provider/configured model, including custom provider extensions.
 - **🧠 Automatic targets** — by default, handoff applies to *every* model that lacks native vision. Opt out with `/vision-handoff auto off`.
@@ -137,7 +139,8 @@ pi -e ./vision-handoff.ts
 The extension implements the **Facebook DataLoader pattern** for image
 descriptions. The `read` tools are the `load()` callers; a per-image cache
 memoizes promises; all `load()` calls in the same execution frame coalesce
-into ONE batched vision call, dispatched after the microtask cascade settles.
+into ONE batched vision call, dispatched via `setImmediate` after the poll
+phase (so reads completing together batch instead of splitting into N calls).
 
     → before_agent_start
         • captures this turn's user prompt (shared by every image description)
@@ -160,10 +163,12 @@ into ONE batched vision call, dispatched after the microtask cascade settles.
         • pi runs `read` calls in parallel (Promise.all); each read's
           tool_result handler calls `loadDescription(img)` for its image
           blocks and AWAITS the shared batch
-        • DataLoader: all `load()` calls in the same microtask frame land in
-          ONE batch → ONE `complete()` vision call for the whole read set
-          (dispatched after the microtask cascade via enqueuePostPromiseJob:
-          `Promise.resolve().then(() => process.nextTick(dispatch))`)
+        • DataLoader: all `load()` calls in one event-loop poll iteration
+          land in ONE batch → ONE `complete()` vision call for the whole read
+          set. `enqueuePostPromiseJob` schedules dispatch via `setImmediate`
+          (the check phase, which runs AFTER the whole poll phase — not
+          `process.nextTick`, which would drain between the reads' I/O
+          callbacks and split them into N single-image calls)
         • awaits the shared batch — runs the describer during the tool-result
           phase (free time: the agent is just waiting for tool results), so
           the batch is COMPLETE before `context` fires → `context` is a
@@ -191,24 +196,35 @@ A single frame's image set is described with **one** vision-model call, not
 one per image. `loadDescription(img)` is synchronous: on a cache miss it
 pushes the image's key (hash) into the current batch and returns a memoized
 promise; on a cache hit it returns the existing (in-flight or resolved)
-promise. `enqueuePostPromiseJob` schedules dispatch after the microtask
-cascade settles, so every `load()` caller in the frame registers its key
-before the single vision call fires — exactly DataLoader's `getCurrentBatch`
-+ `enqueuePostPromiseJob`. The batched call sends every uncached image in a
-single user message with per-image `<<<IMAGE k>>> … <<<END>>>` delimiters; the
-response is parsed back into per-image descriptions (keyed by
-`sha256(mime + base64)` in the per-image cache). If the vision model ignores
-the delimiter format and the batched response can't be split, each unparsed
-image falls back to its own single-image `complete()` call **in parallel**
-(no delimiters to cooperate with) — descriptions still arrive together. Only
-when a per-image call itself genuinely fails (auth, timeout, empty) does that
-image degrade to `[Image: description unavailable]`; one bad image never
-voids the rest.
-
-Because pi runs parallel `read` tool calls via `Promise.all` and fires the
-`tool_result` event inside that `Promise.all` (via `agent.afterToolCall` →
-`finalizeExecutedToolCall`), N parallel reads' `load()` calls share ONE
-microtask frame → ONE batch → ONE vision call, all resolving together.
+promise. `enqueuePostPromiseJob` schedules dispatch via `setImmediate` (the
+check phase, after the whole poll phase AND after the microtask queue
+drains), so every `load()` caller — whether from sync code, a `.then`
+cascade, or a separate I/O callback in the same poll iteration — registers
+its key before the single vision call fires. This is why N parallel reads
+coalesce into one call rather than splitting into N single-image calls:
+`setImmediate` defers past the entire poll phase, whereas DataLoader's
+classic `process.nextTick` would drain between the reads' I/O callbacks and
+fire a dispatch after the first read but before the second. The batched call
+sends every uncached image in a single user message with per-image
+`<<<IMAGE k>>> … <<<END>>>` delimiters; the response is parsed back into
+per-image descriptions (keyed by `sha256(mime + base64)` in the per-image
+cache). If the vision model ignores the delimiter format and the batched
+response can't be split, each unparsed image falls back to its own
+single-image `complete()` call **in parallel** (no delimiters to cooperate
+with) — descriptions still arrive together. Only when a per-image call
+itself genuinely fails (auth, timeout, empty) does that image degrade to
+`[Image: description unavailable]`; one bad image never voids the rest.
+
+Because pi runs parallel `read` tool calls via `Promise.all` and fires
+each read's `tool_result` event as that read's I/O completes (poll phase) —
+via `agent.afterToolCall` → `finalizeExecutedToolCall` — the loader's
+`setImmediate` dispatch defers to the check phase AFTER the whole poll
+iteration, so reads completing together share ONE batch → ONE vision call,
+all resolving together. Reads completing in separate poll iterations get
+separate calls, but always in parallel, never sequential. (The per-image
+cache also dedupes a duplicate `load()` of the same image in one frame:
+dispatch resolves every callback by hash, so a second load whose first cache
+entry was evicted mid-frame still resolves — it never hangs.)
 
 **Failures are never cached.**
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-vision-handoff",
-  "version": "0.4.0",
+  "version": "0.4.1",
   "description": "Give text-only pi models vision — describe images with a vision model you pick via an interactive picker, then hand off the text description to non-vision models",
   "type": "module",
   "author": "Tom X Nguyen",

package/src/dataloader.ts

@@ -4,19 +4,23 @@
  * `loadDescription(img)` returns a memoized Promise for the description and
  * pushes the image's key (hash) into the CURRENT batch. All `load()` calls in
  * the same execution frame (+ its microtask cascade) coalesce into ONE batch
- * object. Dispatch is scheduled after the microtask queue settles (via
- * `enqueuePostPromiseJob`), so every load in the frame lands in the single
- * batch before the ONE vision call fires. Each load()'s promise then resolves
- * with its description.
+ * object. Dispatch is scheduled via `setImmediate` (see `enqueuePostPromiseJob`),
+ * so every load in the frame — and every load from a separate I/O callback in
+ * the same poll iteration — lands in the single batch before the ONE vision
+ * call fires. Each load()'s promise then resolves with its description.
  *
  * Mapping: the `read` tools are the `load()` callers. Their `tool_result`
- * handler awaits the shared batch — so N parallel reads (pi runs `read` via
- * `Promise.all`) block on the SAME single vision call and all resolve together,
- * the descriptions landing in the tool results BEFORE the agent's next turn.
- * The agent's tool-result wait is free time, so this adds zero latency to the
- * critical path. `context` then sees text-described tool results (no image
- * blocks to swap); any remaining images (user-attached, custom-injected) are
- * cache hits.
+ * handler awaits the shared batch — so N parallel reads coalesce into the SAME
+ * single vision call and all resolve together, the descriptions landing in the
+ * tool results BEFORE the agent's next turn. pi fires each read's `tool_result`
+ * as that read's I/O completes (poll phase); the loader's `setImmediate`
+ * dispatch defers to the check phase, AFTER the whole poll iteration, so reads
+ * completing together (the common case for cached local files) land in ONE
+ * batch — and reads completing in separate iterations get separate calls, but
+ * always in parallel, never sequential. The agent's tool-result wait is free
+ * time, so this adds zero latency to the critical path. `context` then sees
+ * text-described tool results (no image blocks to swap); any remaining images
+ * (user-attached, custom-injected) are cache hits.
  *
  * All mutable state (batch, cache, turn context) lives on the instance — no
  * module-level globals — and the class implements `Disposable` so a `using`
@@ -43,7 +47,12 @@ export const UNAVAILABLE = `${IMAGE_PLACEHOLDER_PREFIX}description unavailable${
 interface DescriptionBatch {
   keys: string[];
   imgs: ExtractedImage[];
-  callbacks: { resolve: (v: string) => void; reject: (e: Error) => void }[];
+  /** One per `loadDescription()` call. A duplicate load (same hash, but its
+   *  first cache entry was evicted mid-frame so it couldn't short-circuit on
+   *  the cache) pushes a second callback for an existing key — so
+   *  `callbacks.length` can exceed `keys.length`. `dispatchBatch` resolves by
+   *  hash so every callback is reached. */
+  callbacks: { hash: string; resolve: (v: string) => void; reject: (e: Error) => void }[];
 }
 
 /** Engine-provided resolver for the configured vision model. */
@@ -59,16 +68,21 @@ export interface LoaderDeps extends DescriberDeps {
   resolveVisionModel: VisionModelResolver;
 }
 
-const resolvedMicrotask = Promise.resolve();
-
-/** Defer `fn` until after the current microtask cascade settles, so every
- *  `loadDescription()` in the same frame coalesces into one batch before the
- *  single vision call fires. Mirrors DataLoader's enqueuePostPromiseJob: a
- *  Promise Job enqueues a global Job (process.nextTick), guaranteeing dispatch
- *  runs after "PromiseJobs" ends — after all `load()` callers in the frame
- *  have registered their keys. */
+/** Defer `fn` to the next check phase (`setImmediate`), so every
+ *  `loadDescription()` — whether called from sync code, a microtask cascade,
+ *  or a separate I/O callback in the same poll iteration — coalesces into one
+ *  batch before the single vision call fires.
+ *
+ *  Why `setImmediate` and not DataLoader's classic `process.nextTick`:
+ *  nextTick drains between I/O callbacks in the poll phase, so dispatch would
+ *  fire after the first parallel `read`'s `tool_result` but before the
+ *  second's — splitting N reads into N single-image calls. `setImmediate` runs
+ *  in the check phase, AFTER the whole poll phase, so all `tool_result`
+ *  handlers that fire in one poll iteration land in ONE batch. The check phase
+ *  also runs after the microtask queue drains, so loads issued from a `.then`
+ *  cascade (e.g. the clipboard pre-warm) still coalesce. */
 function enqueuePostPromiseJob(fn: () => void): void {
-  resolvedMicrotask.then(() => process.nextTick(fn));
+  setImmediate(fn);
 }
 
 export class DescriptionLoader implements Disposable {
@@ -119,12 +133,14 @@ export class DescriptionLoader implements Disposable {
       batch.imgs.push(img);
       idx = batch.keys.length - 1;
     } else {
-      // Same image loaded twice in the frame — share one cache slot, but give
-      // this caller its own promise.
+      // Same image loaded twice in the frame (the first load's cache entry was
+      // evicted mid-frame, else the second load would have short-circuited on
+      // the cache). Share the one key/image slot but give this caller its own
+      // promise — dispatch resolves it by hash alongside the first caller's.
       batch.imgs[idx] = img;
     }
     const promise = new Promise<string>((resolve, reject) => {
-      batch.callbacks.push({ resolve, reject });
+      batch.callbacks.push({ hash, resolve, reject });
     });
     const cfg = this.deps.getConfig();
     if (this.cache.size >= cfg.cacheMax) {
@@ -161,7 +177,14 @@ export class DescriptionLoader implements Disposable {
   /** Dispatch the current batch: ONE batched `runBatch` vision call for every
    *  key collected this frame, then resolve each load()'s promise with its
    *  description (or {@link UNAVAILABLE} on failure). Failures are evicted from
-   *  the cache so the next turn re-attempts. */
+   *  the cache so the next turn re-attempts.
+   *
+   *  Results are resolved BY HASH and fanned out to EVERY callback: a batch can
+   *  hold more callbacks than keys when the same image was loaded twice in one
+   *  frame (its first cache entry was evicted mid-frame, so the second load
+   *  pushed a second callback for the same hash). Indexing callbacks by key
+   *  would skip those duplicates and hang their promises; iterating callbacks
+   *  and looking up each one's hash fans the one result to all of them. */
   private async dispatchBatch(): Promise<void> {
     this.dispatchScheduled = false;
     const batch = this.batch;
@@ -172,10 +195,8 @@ export class DescriptionLoader implements Disposable {
       return;
     }
     if (this.turnSignal?.aborted) {
-      for (let i = 0; i < batch.keys.length; i++) {
-        this.cache.delete(batch.keys[i]);
-        batch.callbacks[i].resolve(UNAVAILABLE);
-      }
+      for (const key of batch.keys) this.cache.delete(key);
+      for (const cb of batch.callbacks) cb.resolve(UNAVAILABLE);
       return;
     }
     this.deps.setLastError(null); // clear before a fresh attempt
@@ -190,19 +211,27 @@ export class DescriptionLoader implements Disposable {
       this.deps,
       this.turnSignal,
     );
+    // Build per-hash results, then fan them out to every callback. This reaches
+    // duplicate-hash callbacks that a key-indexed loop would have skipped (and
+    // left hanging).
+    const results = new Map<string, string>();
     for (let i = 0; i < batch.keys.length; i++) {
-      const raw = parsed.get(batch.keys[i]);
+      const key = batch.keys[i];
+      const raw = parsed.get(key);
       if (raw) {
         const final = wrapDescription(raw, cfg);
-        this.cache.set(batch.keys[i], Promise.resolve(final));
-        batch.callbacks[i].resolve(final);
+        results.set(key, final);
+        // Cache the resolved value so later loads (this frame or next) hit.
+        this.cache.set(key, Promise.resolve(final));
       } else {
         // Genuine failure — do NOT cache; next turn re-attempts (and surfaces
         // the real error). Resolve (not reject) with UNAVAILABLE to match the
         // graceful-degradation contract and avoid unhandled rejections.
-        this.cache.delete(batch.keys[i]);
-        batch.callbacks[i].resolve(UNAVAILABLE);
+        this.cache.delete(key);
       }
     }
+    for (const cb of batch.callbacks) {
+      cb.resolve(results.get(cb.hash) ?? UNAVAILABLE);
+    }
   }
 }

package/vision-handoff.ts

@@ -210,7 +210,8 @@ export default function (pi: ExtensionAPI) {
     //
     // Both sources flow through the same loader: `load()` is synchronous and
     // memoized, so all images in this frame (attached + clipboard-path)
-    // coalesce into ONE batch dispatched after the microtask cascade.
+    // coalesce into ONE batch dispatched via `setImmediate` after the
+    // microtask cascade settles.
     for (const image of event.images ?? []) {
       if (!image || image.type !== "image" || !image.data) continue;
       loader.loadDescription({ data: image.data, mimeType: image.mimeType || "image/png" }).catch(() => {});
@@ -244,12 +245,14 @@ export default function (pi: ExtensionAPI) {
   // When the agent reads image files, this fires for each read result. It
   // calls the loader's `loadDescription(img)` for every image block and
   // AWAITS the shared batch — so N parallel reads (pi runs `read` via
-  // Promise.all) coalesce into ONE batched vision call (DataLoader: all
-  // load() calls in the same microtask frame share one batch, dispatched
-  // after the cascade settles) and all resolve together. The descriptions
-  // replace the image blocks in the returned `content`, so by the time the
-  // agent's next turn starts the tool results already carry text — the
-  // agent never sees raw image blocks it can't process.
+  // Promise.all) coalesce into ONE batched vision call: pi fires each read's
+  // `tool_result` as its I/O completes (poll phase), and the loader's
+  // `setImmediate` dispatch defers to the check phase AFTER the whole poll
+  // iteration, so reads completing together land in ONE batch and all resolve
+  // together. The descriptions replace the image blocks in the returned
+  // `content`, so by the time the agent's next turn starts the tool results
+  // already carry text — the agent never sees raw image blocks it can't
+  // process.
   //
   // Why block here and not in `context`: the tool-result phase is free time
   // (the agent is just waiting for tool results), so running the describer
@@ -277,9 +280,13 @@ export default function (pi: ExtensionAPI) {
     }
 
     // load() each image — synchronous calls that push into the current batch
-    // and return memoized promises — then await them all. Parallel reads'
-    // load() calls land in the SAME batch (same microtask frame), so this is
-    // ONE vision call for the whole read set, not N. Awaiting here runs the
+    // and return memoized promises — then await them all. pi fires each read's
+    // `tool_result` event as that read's I/O completes (poll phase); the
+    // loader's `setImmediate` dispatch defers to the check phase, AFTER the
+    // whole poll iteration, so reads completing together (the common case for
+    // cached local files) land in ONE batch — ONE vision call for the whole
+    // read set, not N. Reads completing in separate iterations get separate
+    // calls, but always in parallel, never sequential. Awaiting here runs the
     // describer during the tool-result phase (free time — the agent is just
     // waiting for tool results), so the batch is COMPLETE before `context`
     // fires, making `context` a non-blocking cache hit instead of a cold miss