pi-vision-handoff 0.2.0 → 0.3.0

package/README.md

@@ -26,8 +26,8 @@ The `pi-umans-provider` extension quietly solved this for GLM 5.1: a hardcoded "
 
 - **Pick any vision-capable model** from your registry via an interactive picker — OpenAI, Anthropic, Google, Ollama, or any custom provider pi knows about.
 - Your choice **persists** to `~/.pi/agent/extensions/pi-vision-handoff.json`.
-- For any model that doesn't declare image input (or any model you explicitly target), `pi-vision-handoff` describes the image with your chosen vision model **before** the request is sent, then **swaps the image block for the description** in the actual provider payload.
-- Works across all three request formats pi uses — OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages — detected by block shape.
+- For any model that doesn't declare image input (or any model you explicitly target), `pi-vision-handoff` describes the image with your chosen vision model and **inserts the description before the kept image** in the stored `read`-tool result — **before** pi-ai can strip the image for non-vision models. The image stays for kitty rendering; pi-ai later strips only the image block, leaving the description text for the model.
+- Works across all four image-block shapes pi uses — the three provider-transformed formats (OpenAI Chat Completions, OpenAI Responses, Anthropic Messages) plus pi-ai's internal `{ type: "image", data, mimeType }` emitted by the `read` tool — detected by shape.
 - Descriptions are **cached per image hash** (LRU), so the swap is instant by the time the request fires.
 
 No `settings.json` touched. No per-provider glue. Pick a describer once and every text-only model you own can suddenly see.
@@ -35,12 +35,14 @@ No `settings.json` touched. No per-provider glue. Pick a describer once and ever
 ## Features
 
 - **🎮 Interactive picker** — `/vision-handoff` opens a TUI listing every model, vision-capable ones first (👁), to choose your describer.
+- **🖼️ Read-tool hijack** — `read`-tool images are intercepted at `tool_result` time, before pi-ai can strip them for non-vision models. The kept image still renders inline (kitty); the description is inserted as a text block the text-only model consumes.
 - **🔌 Provider-agnostic** — uses pi's own model execution machinery (`@earendil-works/pi-ai`'s `complete()`), so it works with any provider/configured model, including custom provider extensions.
 - **🧠 Automatic targets** — by default, handoff applies to *every* model that lacks native vision. Opt out with `/vision-handoff auto off`.
 - **🗂️ Explicit overrides** — force handoff for specific models (e.g. a weak vision model) with `/vision-handoff add`.
 - **⚡ Cache-warmed** — `before_agent_start` describes attached images the moment you submit, so the request is rarely delayed.
 - **🛡️ Graceful degradation** — no API key? Describer unreachable? Aborted? The image is replaced with a clean `[Image: description unavailable]` placeholder instead of crashing your turn.
-- **🔧 Tunable** — cap description length and cache size in the config file.
+- **📊 Usage reporting** — every real describer call reports model + tokens (and Neuralwatt energy/cost when the vision model is a Neuralwatt model), via `pi.appendEntry` + a `vision-handoff:usage` event for live consumers.
+- **🔧 Tunable** — cap description length (`maxDescriptionLines`; unbounded by default, so `read`'s native `ctrl+o` collapse handles compactness) and cache size, in the config file.
 
 ## Usage
 
@@ -71,6 +73,7 @@ Created automatically at `~/.pi/agent/extensions/pi-vision-handoff.json` on firs
   "handoffModels": ["ollama/llava:13b"],
   "maxTokens": 1024,
   "cacheMax": 50,
+  "maxDescriptionLines": 0,
   "prompt": "Describe this image exhaustively…",
   "userPromptPrefix": "The user's request about this image: "
 }
@@ -84,6 +87,7 @@ Created automatically at `~/.pi/agent/extensions/pi-vision-handoff.json` on firs
 | `handoffModels` | `[]` | Extra `provider/id` refs that should also receive handoff. |
 | `maxTokens` | `1024` | Cap on a single description's output. |
 | `cacheMax` | `50` | Max described images kept in the in-memory cache per session. |
+| `maxDescriptionLines` | `0` | Cap on description lines (`0` = unbounded). Default keeps the full description so the `read` tool's native collapse (`ctrl+o`) handles compactness and the model gets complete context; setting `> 0` applies a lossy head-cap to both the TUI render and the model. |
 | `prompt` | _(built-in)_ | Override the describer system prompt. |
 | `userPromptPrefix` | _(built-in)_ | Override the prefix prepended to your original prompt. |
 
@@ -129,34 +133,75 @@ pi -e ./vision-handoff.ts
 
 ## How It Works
 
-```
-You submit a prompt + image, on a text-only model
-  → before_agent_start
-      • is this model a handoff target? (non-vision, or in handoffModels)
-      • for each attached image → describeImage() with your chosen vision model
-          - complete() via pi-ai (resolves API key/headers/baseUrl for you)
-          - cached by sha256(mime + base64)
-      • fire-and-forget — warms the cache
-  → before_provider_request
-      • walk the provider payload's messages[]
-      • for every image block (detected by shape) → swap for a text block:
-          "[Image: <cached description>]"
-      • returns the modified payload to pi
-
-Result: the text-only model receives a vivid text description in place of
-the image, and your turn proceeds normally.
-```
+Read-tool images (the common case — paste a screenshot, `read` an image file):
+
+    read tool returns an image block
+      → tool_result (toolName === "read")
+          • fires BEFORE pi-ai's transformMessages can strip the image for
+            non-vision models (the strip would leave only a "(tool image
+            omitted)" placeholder — too late to describe)
+          • is this model a handoff target? (non-vision, or in handoffModels)
+          • for each {type:"image",...} block → describeImage() with your
+            chosen vision model
+              - complete() via pi-ai (resolves API key/headers/baseUrl)
+              - cached by sha256(mime + base64)
+          • AUGMENTS the stored result: inserts "[Image: <description>]" as a
+            text block BEFORE the kept image, returns {content} to pi
+          • the image stays in content → kitty renders it inline
+          • at provider time, pi-ai strips ONLY the image block for non-vision
+            models — the inserted description text passes through to the model
+
+User-attached images (pasted into the prompt, not read via a tool):
+
+    → before_agent_start
+        • warms the description cache for attached images (fire-and-forget)
+    → before_provider_request
+        • walks the provider payload, swaps image blocks for "[Image: <desc>]"
+        • NOTE: for non-vision models, pi-ai strips image blocks BEFORE this
+          hook fires — so user-attached images on text-only models are not yet
+          described. Use the `read` tool on the image file instead, which routes
+          through the tool_result path above.
+
+Result: the text-only model receives a vivid text description alongside the
+(now-stripped) image reference, and your turn proceeds normally — while the
+terminal still renders the image inline via kitty.
 
 ### Image-block formats handled
 
-| API | Image block shape | Replacement |
-|-----|-------------------|-------------|
-| OpenAI Chat Completions | `{ type: "image_url", image_url: { url: "data:…" } }` | `{ type: "text", text }` |
-| OpenAI Responses | `{ type: "input_image", image_url: "data:…" }` | `{ type: "input_text", text }` |
-| Anthropic Messages | `{ type: "image", source: { type: "base64", media_type, data } }` | `{ type: "text", text }` |
+| Hook | Image block shape | Replacement |
+|------|-------------------|-------------|
+| `tool_result` (read) | `{ type: "image", data, mimeType }` (pi-ai internal) | description text block inserted **before** the kept image |
+| `before_provider_request` | `{ type: "image_url", image_url: { url: "data:…" } }` (OpenAI Chat Completions) | `{ type: "text", text }` |
+| `before_provider_request` | `{ type: "input_image", image_url: "data:…" }` (OpenAI Responses) | `{ type: "input_text", text }` |
+| `before_provider_request` | `{ type: "image", source: { type: "base64", media_type, data } }` (Anthropic Messages) | `{ type: "text", text }` |
 
 The describer call itself goes through pi's normal model machinery (`complete()`), **not** the agent event loop — so it never re-triggers `before_provider_request` (no recursion).
 
+### Usage reporting
+
+Every **real** describer call (cache misses only — cache hits emit nothing) reports its model + tokens so pi and other extensions can account for the handoff cost. When the vision model is a **Neuralwatt** model, the response's `: energy` / `: cost` / `: mcr-session` SSE comments are also captured (the OpenAI SDK discards comment lines, so the response body is teed and parsed — the same technique `pi-neuralwatt-provider` uses). For non-Neuralwatt vision models the energy fields are **omitted** (not zeroed), so consumers can distinguish "no energy" from "zero energy".
+
+Each record is published two ways, mirroring `pi-neuralwatt-provider`'s `neuralwatt:turn-energy` pattern:
+
+- **`pi.appendEntry("vision-handoff-usage", record)`** — persisted to the session log, so it replays on `/resume`, fork, and branch navigation.
+- **`pi.events.emit("vision-handoff:usage", record)`** — live event-bus channel a consumer (e.g. a `pi-tps`-style extension) can filter on to see tokens **and** energy in one payload.
+
+Record shape:
+
+```ts
+{
+  imageHash: string,            // sha256(mime + base64), first 32 hex chars
+  model: string, provider: string,
+  responseModel?: string, responseId?: string,
+  usage: Usage,                 // { input, output, cacheRead, cacheWrite, totalTokens, cost }
+  // Present ONLY when Neuralwatt SSE energy comments were captured:
+  energyJoules?: number, costUsd?: number,
+  energyRaw?: object, mcrSessionRaw?: object, costRaw?: object,
+}
+```
+
+Because `before_agent_start` fires several `describeImage()` calls fire-and-forget, the fetch interception is **refcounted** (installed only while ≥1 describe is in flight) and uses `AsyncLocalStorage` to route each teed response body to the describe call that issued it — so concurrent describes each attribute their own energy correctly without clobbering `globalThis.fetch`. When the vision model is a Neuralwatt model, `pi-neuralwatt-provider`'s own `streamNeuralwatt` tee nests on top and restores back to this interceptor; both tees read the same comment lines independently (the accepted duplication for easy filtering).
+
 ## Comparison with Alternatives
 
 | Approach | Pros | Cons |
@@ -171,7 +216,7 @@ The describer call itself goes through pi's normal model machinery (`complete()`
 
 ```bash
 pnpm install
-pnpm test          # Vitest unit tests (31 passing)
+pnpm test          # Vitest unit tests (77 passing)
 pnpm typecheck     # TypeScript validation
 pnpm lint:dead     # Dead code detection (knip)
 ```
@@ -182,11 +227,13 @@ pnpm lint:dead     # Dead code detection (knip)
 .
 ├── vision-handoff.ts            # Main extension: hooks, command, describer
 ├── src/
-│   ├── index.ts                  # Config schema, read/write, image-block helpers
+│   ├── index.ts                  # Config schema, read/write, image-block helpers (barrel)
+│   ├── usage.ts                  # Describer usage + Neuralwatt energy capture, fetch interceptor
 │   └── vision-model-selector.ts  # Interactive picker TUI component
 ├── __tests__/unit/
 │   ├── config-dir.test.ts        # Ensures getAgentDir() usage
-│   └── vision-handoff.test.ts    # Config, refs, image-block extraction, round-trip
+│   ├── usage.test.ts             # Energy parsing, usage records, concurrency-safe fetch routing
+│   └── vision-handoff.test.ts    # Config, refs, image-block extraction, insertion, truncation, round-trip
 ├── package.json
 ├── tsconfig.json
 ├── knip.json

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-vision-handoff",
-  "version": "0.2.0",
+  "version": "0.3.0",
   "description": "Give text-only pi models vision — describe images with a vision model you pick via an interactive picker, then hand off the text description to non-vision models",
   "type": "module",
   "author": "Tom X Nguyen",

package/src/index.ts

@@ -10,6 +10,8 @@ import { getAgentDir } from "@earendil-works/pi-coding-agent";
 import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
 import { join } from "node:path";
 
+export * from "./usage.js";
+
 /** Subdirectory under the pi agent dir where picker extensions store config. */
 const CONFIG_SUBDIR = "extensions";
 
@@ -57,6 +59,29 @@ export const DESCRIBE_TIMEOUT_MS = 30_000;
  */
 export const DEFAULT_MAX_DESCRIPTION_LINES = 0;
 
+/**
+ * The note pi core's `read` tool appends to image-result text blocks when the
+ * active model lacks image input (see `getNonVisionImageNote` in pi core).
+ *
+ * Once this extension inserts a description, the note becomes self-
+ * contradicting ("image will be omitted" vs. the vivid description that
+ * follows) and confuses the model. {@link stripNonVisionImageNote} removes it.
+ */
+export const NON_VISION_IMAGE_NOTE =
+  "[Current model does not support images. The image will be omitted from this request.]";
+
+/**
+ * Remove {@link NON_VISION_IMAGE_NOTE} from a text block.
+ *
+ * The read tool appends the note as `\n${NOTE}` (always the trailing segment of
+ * the metadata text block), so this also collapses the orphaned newline it
+ * leaves behind. Safe to call on text that does not contain the note (no-op).
+ */
+export function stripNonVisionImageNote(text: string): string {
+  if (!text.includes(NON_VISION_IMAGE_NOTE)) return text;
+  return text.split(NON_VISION_IMAGE_NOTE).join("").replace(/\n+$/, "");
+}
+
 export interface VisionHandoffConfig {
   /** Master switch. When false, no handoff occurs even if a vision model is configured. */
   enabled: boolean;
@@ -292,6 +317,12 @@ export interface ReplacedContent {
  * provider, registry, and API call. The extension wires its real `describeImage`
  * into this helper in its `tool_result` handler.
  *
+ * When at least one description is inserted, also strips pi core's
+ * {@link NON_VISION_IMAGE_NOTE} from text blocks — the note (appended by the
+ * read tool for non-vision models) would otherwise contradict the inserted
+ * description. Text blocks are reassigned (not mutated in place); the input
+ * array is left untouched.
+ *
  * Returns a new array and `changed: false` when there were no images, so
  * callers can short-circuit and avoid mutating pi's stored result unnecessarily.
  */
@@ -315,5 +346,18 @@ export async function insertImageDescriptions(
     next.push(block);
     changed = true;
   }
+  if (!changed) {
+    return { content: next, changed };
+  }
+  // We inserted at least one description. Strip the read tool's
+  // "[Current model does not support images...]" note from text blocks — it
+  // contradicts the description we just inserted ("image will be omitted" vs.
+  // the description that follows) and confuses the model.
+  for (let i = 0; i < next.length; i++) {
+    const block = next[i];
+    if (block.type === "text" && typeof block.text === "string" && block.text.includes(NON_VISION_IMAGE_NOTE)) {
+      next[i] = { type: "text", text: stripNonVisionImageNote(block.text) } satisfies TextContent;
+    }
+  }
   return { content: next, changed };
 }

package/src/usage.ts

@@ -0,0 +1,245 @@
+/**
+ * Describer usage + energy capture for pi-vision-handoff.
+ *
+ * One record is produced per REAL describer provider call (cache hits emit
+ * nothing):
+ *   - model + tokens: from complete()'s AssistantMessage.usage
+ *   - energy + cost + raw MCR/energy/cost payloads: from Neuralwatt SSE comment
+ *     lines parsed out of the teed response body (readEnergyFromTee). Present
+ *     ONLY when the vision model is a Neuralwatt model — non-Neuralwatt models
+ *     produce no comment lines, so the energy fields are OMITTED (not zeroed)
+ *     for easy downstream filtering ("no energy" vs "zero energy").
+ *
+ * The caller (vision-handoff.ts) persists the record via pi.appendEntry (replays
+ * on session resume/branch) AND emits it on pi.events so a live consumer can
+ * filter on the one channel for tokens AND energy.
+ *
+ * Split out of vision-handoff.ts so the pure pieces (readEnergyFromTee,
+ * buildUsageRecord) and the concurrency-safe fetch interceptor are unit-testable
+ * through the normal src/ import path — vision-handoff.ts runs readConfig() at
+ * module load and pulls in the TUI selector, so it is not unit-test-friendly.
+ */
+
+import { AsyncLocalStorage } from "node:async_hooks";
+import type { Api, AssistantMessage, Model, Usage } from "@earendil-works/pi-ai";
+
+/** Custom session-entry type persisted via pi.appendEntry. */
+export const USAGE_ENTRY_TYPE = "vision-handoff-usage";
+
+/** Event-bus channel emitted via pi.events. */
+export const USAGE_EVENT_CHANNEL = "vision-handoff:usage";
+
+/** Parsed Neuralwatt SSE-comment energy/cost/MCR data for one describer call. */
+export interface VisionHandoffEnergyCapture {
+  energyJoules: number;
+  costUsd: number;
+  energyRaw: Record<string, unknown> | null;
+  mcrSessionRaw: Record<string, unknown> | null;
+  costRaw: Record<string, unknown> | null;
+}
+
+/** Shared empty capture. Safe to share: readEnergyFromTee returns a fresh object
+ *  and never mutates this; buildUsageRecord only reads from its argument. */
+export const EMPTY_ENERGY_CAPTURE: VisionHandoffEnergyCapture = {
+  energyJoules: 0,
+  costUsd: 0,
+  energyRaw: null,
+  mcrSessionRaw: null,
+  costRaw: null,
+};
+
+/** A single describer-call usage record. Energy fields are present only when
+ *  Neuralwatt SSE energy comments were captured (omitted, not zeroed, otherwise). */
+export interface VisionHandoffUsageRecord {
+  imageHash: string;
+  model: string;
+  provider: string;
+  responseModel?: string;
+  responseId?: string;
+  usage: Usage;
+  /** Neuralwatt energy — present only when SSE energy comments were captured. */
+  energyJoules?: number;
+  costUsd?: number;
+  energyRaw?: Record<string, unknown> | null;
+  mcrSessionRaw?: Record<string, unknown> | null;
+  costRaw?: Record<string, unknown> | null;
+}
+
+/**
+ * Parse Neuralwatt SSE comment lines (`: energy`, `: cost`, `: mcr-session`)
+ * from a teed response body into a fresh capture object. Mirrors
+ * pi-neuralwatt-provider's readEnergyFromTee but returns a result instead of
+ * mutating module state — vision-handoff describes images concurrently (the
+ * before_agent_start warm-up fires several in parallel), so each call needs its
+ * own capture routed via {@link describeAls}. For non-Neuralwatt vision models
+ * no comment lines are present and the returned capture stays empty.
+ */
+export async function readEnergyFromTee(
+  body: ReadableStream<Uint8Array>,
+): Promise<VisionHandoffEnergyCapture> {
+  const result: VisionHandoffEnergyCapture = { ...EMPTY_ENERGY_CAPTURE };
+  const reader = body.getReader();
+  const decoder = new TextDecoder();
+  let buffer = "";
+
+  function processLine(line: string): void {
+    const trimmed = line.trim();
+    if (trimmed.startsWith(": energy ")) {
+      try {
+        const energy = JSON.parse(trimmed.slice(9));
+        result.energyJoules += energy.energy_joules || 0;
+        result.energyRaw = energy;
+      } catch {
+        // malformed energy comment — ignore
+      }
+    } else if (trimmed.startsWith(": mcr-session ")) {
+      try {
+        result.mcrSessionRaw = JSON.parse(trimmed.slice(14));
+      } catch {
+        // malformed mcr-session comment — ignore
+      }
+    } else if (trimmed.startsWith(": cost ")) {
+      try {
+        const cost = JSON.parse(trimmed.slice(7));
+        result.costUsd += cost.request_cost_usd || 0;
+        result.costRaw = cost;
+      } catch {
+        // malformed cost comment — ignore
+      }
+    }
+  }
+
+  try {
+    while (true) {
+      const { done, value } = await reader.read();
+      if (done) break;
+      buffer += decoder.decode(value, { stream: true });
+      const lines = buffer.split("\n");
+      buffer = lines.pop() || "";
+      for (const line of lines) processLine(line);
+    }
+  } catch {
+    // tee stream may error if the main stream is aborted — that's fine
+  }
+
+  const final = decoder.decode(new Uint8Array(0), { stream: false });
+  const remaining = (buffer + final).trim();
+  if (remaining) processLine(remaining);
+
+  try {
+    reader.releaseLock();
+  } catch {
+    // ignore
+  }
+  return result;
+}
+
+/**
+ * Build a usage record from a describer response + energy capture. Returns
+ * null when there is nothing meaningful to report (e.g. a provider-level
+ * failure with zero tokens and no energy) so the caller can skip emitting.
+ *
+ * Energy fields are OMITTED entirely (not zeroed) when no Neuralwatt SSE energy
+ * comments were captured, so consumers can distinguish "no energy" from
+ * "zero energy".
+ */
+export function buildUsageRecord(
+  response: AssistantMessage,
+  capture: VisionHandoffEnergyCapture,
+  visionModel: Model<Api>,
+  imageHash: string,
+): VisionHandoffUsageRecord | null {
+  const hasEnergy = !!(
+    capture.energyRaw ||
+    capture.costRaw ||
+    capture.mcrSessionRaw ||
+    capture.energyJoules > 0 ||
+    capture.costUsd > 0
+  );
+  const hasUsage =
+    !!response.usage &&
+    (response.usage.totalTokens > 0 || response.usage.input > 0 || response.usage.output > 0);
+  if (!hasUsage && !hasEnergy) return null;
+
+  const record: VisionHandoffUsageRecord = {
+    imageHash,
+    model: response.model || visionModel.id,
+    provider: response.provider || visionModel.provider,
+    responseModel: response.responseModel,
+    responseId: response.responseId,
+    usage: response.usage,
+  };
+  if (hasEnergy) {
+    record.energyJoules = capture.energyJoules;
+    record.costUsd = capture.costUsd;
+    record.energyRaw = capture.energyRaw;
+    record.mcrSessionRaw = capture.mcrSessionRaw;
+    record.costRaw = capture.costRaw;
+  }
+  return record;
+}
+
+// ── Concurrency-safe fetch interceptor ─────────────────────────────────────
+//
+// The only body-interception point for complete() is globalThis.fetch (pi-ai's
+// StreamOptions.onResponse exposes headers only, not the body where the
+// `: energy` SSE comments live). before_agent_start fires several describeImage()
+// calls fire-and-forget, so a naïve save/patch/restore of globalThis.fetch would
+// clobber under concurrency. The fix is a refcounted shared interceptor
+// (installed only while ≥1 describe is in flight, so non-describe fetches pass
+// through unmodified) + AsyncLocalStorage to route each teed response body to
+// the describe call that issued it. Nested patches from other extensions (e.g.
+// pi-neuralwatt-provider's streamNeuralwatt, which also tees for its own energy
+// display) chain on top and restore back to this interceptor, so both tees read
+// the same comment lines independently.
+
+/** Per-describer-call ALS slot carrying the energy tee reader. */
+export interface DescribeContext {
+  energyReader: Promise<VisionHandoffEnergyCapture> | undefined;
+}
+
+/** Routes each teed response body to the describe call that issued it. */
+export const describeAls = new AsyncLocalStorage<DescribeContext>();
+
+let fetchInterceptRefCount = 0;
+let savedRealFetch: typeof globalThis.fetch | null = null;
+
+/** Install the globalThis.fetch interceptor. Refcounted: the first caller
+ *  patches fetch; later callers just bump the count. Idempotent per install. */
+export function installFetchInterceptor(): void {
+  if (fetchInterceptRefCount === 0) {
+    savedRealFetch = globalThis.fetch;
+    globalThis.fetch = interceptedFetch;
+  }
+  fetchInterceptRefCount++;
+}
+
+/** Remove the interceptor. Refcounted: only the last caller restores fetch. */
+export function uninstallFetchInterceptor(): void {
+  if (fetchInterceptRefCount > 0) fetchInterceptRefCount--;
+  if (fetchInterceptRefCount === 0 && savedRealFetch) {
+    globalThis.fetch = savedRealFetch;
+    savedRealFetch = null;
+  }
+}
+
+/** Current refcount — 0 means the interceptor is not installed. Test hook. */
+export function fetchInterceptorRefcount(): number {
+  return fetchInterceptRefCount;
+}
+
+async function interceptedFetch(input: RequestInfo | URL, init?: RequestInit): Promise<Response> {
+  const real = savedRealFetch ?? globalThis.fetch;
+  const response = await real(input, init);
+  const store = describeAls.getStore();
+  // Outside a describer call (no ALS store) or for bodiless responses: pass
+  // through untouched.
+  if (!store || !response.body) return response;
+  const [bodyForSdk, bodyForEnergy] = response.body.tee();
+  store.energyReader = readEnergyFromTee(bodyForEnergy);
+  return new Response(bodyForSdk, {
+    headers: response.headers,
+    status: response.status,
+    statusText: response.statusText,
+  });
+}

package/vision-handoff.ts

@@ -37,21 +37,35 @@ import {
   HANDOFF_COMMAND_DESCRIPTION,
   IMAGE_PLACEHOLDER_PREFIX,
   IMAGE_PLACEHOLDER_SUFFIX,
+  USAGE_ENTRY_TYPE,
+  USAGE_EVENT_CHANNEL,
+  EMPTY_ENERGY_CAPTURE,
+  buildUsageRecord,
+  describeAls,
   extractImageFromBlock,
   formatModelRef,
   insertImageDescriptions,
+  installFetchInterceptor,
   isVisionModel,
   makeReplacementText,
   parseModelRef,
   readConfig,
   truncateDescription,
+  uninstallFetchInterceptor,
   writeConfig,
+  type DescribeContext,
   type VisionHandoffConfig,
+  type VisionHandoffEnergyCapture,
+  type VisionHandoffUsageRecord,
 } from "./src/index.js";
 import { VisionModelSelectorComponent, type VisionModelSelectorResult } from "./src/vision-model-selector.js";
 
 const UNAVAILABLE = `${IMAGE_PLACEHOLDER_PREFIX}description unavailable${IMAGE_PLACEHOLDER_SUFFIX}`;
 
+// Usage reporter; wired to pi.appendEntry + pi.events.emit in the default
+// export. No-op until then so describeImage is safe to call before wiring.
+let reportUsage: (record: VisionHandoffUsageRecord) => void = () => {};
+
 let config: VisionHandoffConfig = readConfig();
 
 /** User prompt for the current agent turn, captured from before_agent_start. */
@@ -125,17 +139,38 @@ async function describeImage(
 
     const controller = new AbortController();
     const timer = setTimeout(() => controller.abort(), DESCRIBE_TIMEOUT_MS);
+    // Energy/token capture for this describer call. The fetch interceptor is
+    // refcount-installed around the complete() window and routes the teed
+    // response body to this describe's AsyncLocalStorage slot — so concurrent
+    // describes (before_agent_start warm-up fires several in parallel) each
+    // get their own capture without clobbering globalThis.fetch. For non-
+    // Neuralwatt vision models no SSE energy comments are present and the
+    // capture stays empty (energy fields omitted from the record).
+    const describeCtx: DescribeContext = { energyReader: undefined };
+    installFetchInterceptor();
     try {
-      const response = await complete(
-        visionModel,
-        { systemPrompt, messages: [userMessage] },
-        {
-          apiKey: auth.apiKey,
-          headers: auth.headers,
-          signal: controller.signal,
-          maxTokens: cfg.maxTokens,
-        },
+      const response = await describeAls.run(describeCtx, async () =>
+        complete(
+          visionModel,
+          { systemPrompt, messages: [userMessage] },
+          {
+            apiKey: auth.apiKey,
+            headers: auth.headers,
+            signal: controller.signal,
+            maxTokens: cfg.maxTokens,
+          },
+        ),
       );
+      let capture: VisionHandoffEnergyCapture = EMPTY_ENERGY_CAPTURE;
+      if (describeCtx.energyReader) {
+        try {
+          capture = await describeCtx.energyReader;
+        } catch {
+          // tee aborted with the main stream — keep the empty capture
+        }
+      }
+      const record = buildUsageRecord(response, capture, visionModel, key);
+      if (record) reportUsage(record);
       if (response.stopReason === "aborted" || response.stopReason === "error") {
         return UNAVAILABLE;
       }
@@ -159,6 +194,8 @@ async function describeImage(
     } catch {
       return UNAVAILABLE;
     } finally {
+      if (describeCtx.energyReader) describeCtx.energyReader.catch(() => {});
+      uninstallFetchInterceptor();
       clearTimeout(timer);
     }
   })();
@@ -213,6 +250,26 @@ function notifyUnresolvedVisionModel(ctx: ExtensionContext, ref: string): void {
 export default function (pi: ExtensionAPI) {
   config = readConfig();
 
+  // Wire the usage reporter to pi's persistence + event bus. appendEntry
+  // persists the record so it replays on session resume/branch, and the event
+  // lets live consumers filter on one channel for tokens AND energy. Each call
+  // is independently guarded so a persistence/emit failure never breaks a
+  // describer turn. Re-assigned every factory invocation (pi re-runs the
+  // factory on /new, /resume, fork, /reload) so the closure always references
+  // the live pi.
+  reportUsage = (record: VisionHandoffUsageRecord) => {
+    try {
+      pi.appendEntry(USAGE_ENTRY_TYPE, record);
+    } catch {
+      // never break the describer on persistence failure
+    }
+    try {
+      pi.events?.emit(USAGE_EVENT_CHANNEL, record);
+    } catch {
+      // never break the describer on emit failure
+    }
+  };
+
   pi.on("session_start", async () => {
     // Reload in case the user edited the config on disk from another session.
     config = readConfig();