@cubicecho/agent-core 2.10.0 → 2.12.0

package/README.md

@@ -27,9 +27,11 @@ only, Node >=22.
 | `thinking` | Tells a scratchpad fenced inside `content` from the answer: `FenceSplitter` for a stream, `stripThinking` for a whole reply, and the fence tables both read. |
 | `side-task` | One-shot calls that support a run without being one — small prompt, short answer, no tools, never worth failing the run over. `askJson` holds the answer to a schema where the server can. |
 | `hooks` | The host's side of lifecycle hooks: `gather` before a request and `notify` after, the shared context budget, `withContext` to put what they add on the turn's question, `untrusted` to fence text nobody vouched for, and `turnMessages` to hand them a transcript. Running a hook is a runner the caller passes. |
-| `events` | The in-memory bus a watcher reads while a run happens: `emit`, `watch`, `history`, `fold`. A watcher's backlog is capped and reports its own gaps. |
+| `events` | The in-memory bus a watcher reads while a run happens: `emit`, `watch`, `history`, `fold`, and `runMetrics` for what a run cost. A watcher's backlog is capped and reports its own gaps. |
 | `client` | A pooled `OpenAI` client per endpoint, plus the context window: the served one where a local server says, the listed one otherwise, and their caches. |
 | `retry` | What to do when a request is lost, refused or too big: `isTransient`, `isModelLoading`, `backoffMs`, `ContextOverflow`, `EndpointSilent`, `requestTokens`. |
+| `calibration` | How many characters a token is worth on one model, learned from the prompt counts its endpoint reports: `charsPerTokenFor`, `calibrate`. |
+| `continuation` | `continueTurn`: carries on an answer the token ceiling cut off, by prefilling it as a trailing assistant message. |
 | `config` | The structural interfaces every function here asks for. |
 | `run-turn` | `runTurn`: one turn with the retry loop around the negotiation around the stream. The whole loop, for a caller that wants it rather than its parts. Sizes the request against an opt-in `contextLimit`. |
 | `agent-loop` | `runAgentLoop`: the loop above a turn — `runTurn` per step, the tools between, `load_tools` and preselection handled, until the model stops asking. Plus the parts it is made of: `buildBody`, `preselect`, `preview`, and `resolveApiKey` for a caller deciding which key an endpoint gets. |
@@ -87,6 +89,20 @@ truncated prose or — the case that bites — a tool call whose `arguments` sto
 caller meets a parse failure with nothing to attribute it to. `finishReason` is `"length"` there,
 `""` where the endpoint never said.
 
+`usage` is `prompt`, `completion`, `total` and `cached`, always there and zero where the server
+sent nothing. Everything else on it is there only when something measured it, and absent rather
+than zero otherwise:
+
+| Field | From |
+| --- | --- |
+| `uncached` | the prompt less `cached`, only where a cache count was reported, so a cold cache and no report read differently |
+| `reasoningTokens` | `completion_tokens_details.reasoning_tokens`; never estimated from the thinking stream |
+| `promptMs`, `promptTokensPerSecond`, `predictedMs`, `tokensPerSecond`, `draftTotal`, `draftAccepted` | llama.cpp's `timings`, which also stands in for `cached` (`cache_n`) where the usage has no cache count |
+| `firstTokenMs` | `streamTurn`, from the request to the first chunk that carried something |
+| `wallMs`, `retries`, `timeouts` | `runTurn`: the whole turn with its backoff, the lost requests sent again, and how many of those went silent |
+| `continuations` | `continueTurn` |
+| `cacheExpected`, `cacheBroken`, `cacheBreakReason`, `toolsDeclared`, `toolSchemaTokens` | `runAgentLoop`, below |
+
 `reasoning` is the scratchpad `onThinking` was told, kept because two common families want it
 back. gpt-oss and DeepSeek in thinking mode read the analysis behind a tool call off the assistant
 message on the next request: store it as `reasoning_content` on that message while it ends in a
@@ -150,6 +166,10 @@ and the newer spelling is exactly the one an older model or a llama.cpp-shaped e
 `modelCapabilitiesFor` starts a model at `legacyTokenLimit: true`, and an absent model has to read
 the same way it does.
 
+One more model flag is never answered by `negotiate`: `assistantPrefill`, whether the model
+carries on a trailing assistant message rather than answering afresh after it. Only `continueTurn`
+sends one, so only it latches the flag, and it is persisted in a snapshot like the rest.
+
 `runTurn` takes the same option and hands `request` the same second argument. Keying on
 `(endpoint, model)` rather than the model name alone is the part worth keeping: `gpt-4o` at
 OpenAI and `gpt-4o` behind a proxy need not be the same weights, and one that refused a reasoning
@@ -202,6 +222,21 @@ mean every property required and `additionalProperties: false`; a looser schema
 `preselect` is its first user, answering `{ tools: [...] }`, and `preselection` still takes the
 bare array an older prompt produced.
 
+### Showing a side task an image
+
+`ask` and `askJson` take the user turn as a string or as content parts, because an image reaches an
+OpenAI-compatible server only as an `image_url` part beside the text. The parts are sent as given
+and nothing here checks that the model can see: whether a text-only model rejects the image or
+answers without it is up to the server, so pick a vision model.
+
+```ts
+const page: SideTaskInput = [
+  { type: "text", text: "Transcribe this page." },
+  { type: "image_url", image_url: { url: `data:image/png;base64,${png}` } },
+];
+const text = await ask(config, visionModel, "You are an OCR engine.", page);
+```
+
 ## Sizing a request before sending it
 
 `runTurn` will refuse a request that cannot fit rather than spending a round trip finding out:
@@ -236,6 +271,30 @@ nothing, and the server gives the reply whatever the prompt leaves. A consumer t
 `maxTokens` from the limit itself before calling in no longer needs to, and doing both reserves
 the ceiling twice.
 
+How many tokens the body is, is a guess: characters over a divisor. Four is right for English prose
+and wrong for what a tool-using run is made of — JSON schemas and tool results pack closer to two or
+three characters a token — so a request the guard let through was refused anyway. The divisor is
+learned instead. Every turn `runTurn` answers comes back with the endpoint's exact prompt count for a
+body whose characters were already counted, and `calibrate` keeps that ratio per endpoint and model;
+the next request to the same model is sized by `charsPerTokenFor`, which is `CHARS_PER_TOKEN` (4)
+until a turn has reported one. It is the highest of the model's last four readings, which is the
+lowest token count: a count that comes out high refuses a run that would have fit, and one that
+comes out low only costs the round trip the guard was saving. A reading from a request carrying an
+image, or outside one to eight characters a token, is not taken.
+
+`requestTokens` and `messageTokens` take `{ charsPerToken }`, and so does `planCompaction`, so a
+caller sizing its own work can use the same number:
+
+```ts
+const charsPerToken = charsPerTokenFor(capabilitiesFor(config.baseUrl, config.apiKey), config.model);
+const plan = planCompaction(messages, { limit, used, charsPerToken });
+```
+
+The ratio is a measurement rather than a refusal, so it is not a capability latch and is not in
+`exportCapabilities`: it moves every turn, and a restarted process learns it again from its first
+one. llama.cpp's `/tokenize` would count exactly, but only after `/apply-template` renders the
+prompt — two round trips before every guarded request, on one server — so it is not used.
+
 The body is sized once, not per attempt: a downgraded request is strictly smaller than the one
 before it and the transcript does not change between retries. A `ContextOverflow` from this is
 neither a capability `negotiate` can answer nor something `isTransient` accepts, so it leaves
@@ -316,6 +375,55 @@ run. A preselection shapes the first step alone: those tools, no catalogue, no `
 a model with the menu still in front of it shops, reloading what it has or picking a sibling —
 and everything is back from the second step on.
 
+A turn cut off at `maxTokens` is said so as a notice, and with `maxContinuations` above zero it is
+continued first. `continueTurn` is the same thing for a caller with its own loop:
+
+```ts
+let turn = await runTurn(client, supports, build, options);
+turn = await continueTurn(client, supports, build, turn, { ...options, maxContinuations: 2 });
+```
+
+It sends the transcript again with the answer so far as a trailing assistant message, which
+llama.cpp renders as a prefill: the model carries on from the last token, and the cache holds all
+of the prompt and most of the reply. vLLM only does so given `continue_final_message: true` and
+`add_generation_prompt: false` on that request, which nothing here sends yet. The pieces come
+back as one turn — content and reasoning joined, token counts summed, `continuations` counting the
+extra requests, timings summed or weighted, a field only one piece reported dropped. Only an answer
+begun and cut off is continued. A turn cut off in its scratchpad is left alone, since llama.cpp
+refuses a prefill outright on a template with thinking on, and so is one ending in a tool call,
+whose truncated arguments `parseToolArguments` already reports.
+
+Whether it works is latched per model as `assistantPrefill`. A 400 or 422 for the request latches it
+off, and so does a continuation that starts the answer over word for word — how hosted OpenAI, which
+takes a trailing assistant message and ignores it, shows itself. Either way, and on any other
+failure short of a stop, the cut-off answer is kept with a notice. The cap is one continuation for
+`continueTurn` and none for the loop, since it spends a request, and on a server that does not
+continue, one to find out.
+
+Every turn ends in a `usage` event, whether or not the endpoint reported tokens. Its totals are the
+run's so far; its `turn` is that turn's own `TurnUsage` and `finishReason`. The loop adds what only
+it can know: `toolsDeclared` and `toolSchemaTokens` for the tool block it sent, and, from the second
+request on, `cacheExpected` (the previous prompt plus its reply) and — where a cache count was
+reported — `cacheBroken`, a hit short of 90% of the previous prompt. A broken cache is given a
+`cacheBreakReason` read off the request against the one before: `tools-changed`, `system-changed`,
+`history-rewritten` (a compaction or a prune in `beforeStep`), or `none-known` where the new request
+only appended, which points at the server — a slot evicted, a template that re-renders the tail.
+
+`runMetrics(events)` adds a run up from those events — tokens, cache hit ratio and breaks by
+reason, prefill, decode and tool time, the slowest turn, mean time to first token, draft
+acceptance, the largest prompt against a `contextLength`, turns cut off, tool errors by name, and
+an `outcome`. The loop returns it as `metrics`, with the `load_tools` counts only it can see
+(`toolsLoaded`, `redundantLoads`, `unknownToolNames`). It is a sibling of `fold` rather than part
+of it, since `fold`'s blocks are for display and a summary is not one:
+
+```ts
+const metrics = runMetrics(history(runId), { contextLength: config.contextLength });
+```
+
+Counts are always present; every other field is absent where no turn reported what it is made of.
+What it cannot say: whether a failed run was stopped, errored or ran out of tool iterations, and
+whether the host compacted — neither is in the events.
+
 `beforeStep` is handed the transcript before each request and may return a replacement, which is
 where compaction goes (below). Hooks are gathered once, onto the question, and never written into
 the transcript that comes back; `afterTurn` is told the reply without the run waiting on it.
@@ -364,7 +472,7 @@ once `used` (the last turn's prompt tokens, or the estimate) is past three quart
 The kept tail fills at most 35% of it and starts on a user message, since a transcript resuming
 mid-exchange is one servers refuse; leading system prompts are never folded, and an earlier
 summary is continued rather than summarised. `compactTranscript` writes the summary and tells
-`beforeCompact` hooks what is going while it does. A hook cannot veto it. Because the cut lands on
+`beforeCompact` hooks what is going while it does. Because the cut lands on
 a user message, one long tool run under a single question has nothing to fold — pruning is what
 keeps that one going.
 
@@ -381,6 +489,21 @@ beforeStep: async (messages, step) => {
 },
 ```
 
+A hook can ask for a compaction not to happen — its runner sets `veto` on the outcome — and by default
+nobody listens: the hooks run beside the summary, so a slow one costs the run nothing. Pass
+`honourVeto: true` with the hooks and they run first — the summary waits on them — and a veto from
+any hook that ran leaves the transcript as it was, with a note naming the hook. One that failed
+vetoes nothing. A compaction passed `forced: true`, because a request was already refused as too
+big, goes ahead regardless: a veto there only trades the summary for a `ContextOverflow`.
+`consult` is the same wait for a host that compacts its own way.
+
+```ts
+const compacted = await compactTranscript(messages, plan, summarise, {
+  hooks: { run, context, onNote, honourVeto: true },
+  forced: retryingAfterOverflow, // the last request came back as a ContextOverflow
+});
+```
+
 **Both rewrite the prefix.** A prompt cache matches from the first token, so a transcript whose
 early messages change is re-processed whole — on a local server that is the entire prefill, every
 time. Run them rarely and together, at the point `planCompaction` says the window is filling, so
@@ -564,7 +687,8 @@ with no timeout to give now leaves it out rather than inventing a `0`.
 ## What is kept for the life of the process
 
 Four caches outlive any one run: the `OpenAI` clients, the model listings, the latched
-capabilities, and `side-task`'s no-thinking hints. All four are module-level and keyed on the same
+capabilities, and `side-task`'s no-thinking hints. A fifth, the characters per token each model was
+measured at, is keyed on the endpoint's capabilities object, so it goes with them. All four are module-level and keyed on the same
 notion of an endpoint — its base URL and its API key — and the clients' key carries the request
 timeout as well, since that changes how a request is sent.
 
@@ -604,7 +728,8 @@ URL and the key, an absent key read as `NO_KEY` — rather than rebuilding that
 cannot drift. `endpointKey` holds the key in the clear; `endpointId`, its SHA-256 digest, is the one
 that is safe to write down.
 
-`resetAll` drops all four, and `reset.ts` names each seam separately for a test that wants one.
+`resetAll` drops all five, and `reset.ts` names each seam separately for a test that wants one —
+`resetCalibration` for the measured ratios.
 
 The latches can outlive the process as well, because otherwise every restart spends one refused
 request per endpoint and model learning the same facts again. `exportCapabilities` returns every

package/dist/agent-loop.d.ts

@@ -2,7 +2,7 @@ import type OpenAI from "openai";
 import { type Capabilities, type ModelCapabilities } from "./capabilities.ts";
 import type { CatalogServer } from "./catalog.ts";
 import type { Endpoint, ModelParams, RetryPolicy, ToolPolicy } from "./config.ts";
-import type { RunEventInput } from "./events.ts";
+import { type RunEventInput, type RunMetrics } from "./events.ts";
 import { type HookContext, type HookEvent, type HookNote, type HookRunner } from "./hooks.ts";
 import type { Turn, TurnUsage } from "./stream.ts";
 /**
@@ -168,6 +168,12 @@ export interface AgentLoopOptions {
     recoverToolCalls?: boolean;
     /** Each turn as it comes back, before its tools run. Recovered calls are in it as calls. */
     onTurn?: (turn: Turn, step: number) => void;
+    /**
+     * How many times an answer cut off at `maxTokens` is continued, zero — the default — for never.
+     * Opt-in because it spends another request, and on a server that does not continue a trailing
+     * assistant message it spends one to find that out. See `continueTurn`.
+     */
+    maxContinuations?: number;
 }
 /** What a finished loop hands back. */
 export interface AgentLoopResult {
@@ -185,6 +191,11 @@ export interface AgentLoopResult {
     used: string[];
     /** The hooks' notes from before the first request. */
     notes: HookNote[];
+    /**
+     * The run summed and derived: what `runMetrics` makes of the events this loop emitted, plus the
+     * `load_tools` findings only the loop sees, `wallMs` from the call to the return, and `outcome`.
+     */
+    metrics: RunMetrics;
 }
 /**
  * Runs a question to its answer: one `runTurn` per step, the tools it asks for between them,
@@ -195,7 +206,9 @@ export interface AgentLoopResult {
  * prompt unchanged from step to step, loaded tools are appended to the tool array in load order,
  * a catalogued tool called without being loaded is loaded and run rather than refused, and a
  * preselection shapes the first step. A turn cut off at `maxTokens` is said so as a notice,
- * because it otherwise reads exactly like a finished one.
+ * because it otherwise reads exactly like a finished one — or, given `maxContinuations`, is
+ * continued first. Every turn ends in a `usage` event carrying the turn's own report, with the
+ * cache compared against the request before it; see `TurnUsage` and `runMetrics`.
  *
  * @param options The config, transcript, tools and dispatcher, plus the optional hooks, events
  * and cancellation. See `AgentLoopOptions`.

package/dist/agent-loop.js

@@ -1,7 +1,11 @@
+import { charsPerTokenFor } from "./calibration.js";
 import { capabilitiesFor } from "./capabilities.js";
 import { firstTokenMs, getClient, NO_KEY, timeoutMs } from "./client.js";
+import { continueTurn } from "./continuation.js";
 import { errorMessage } from "./errors.js";
+import { runMetrics } from "./events.js";
 import { gather, notify, turnIndex, turnMessages, withContext, } from "./hooks.js";
+import { toolsChars } from "./retry.js";
 import { runTurn } from "./run-turn.js";
 import { relaxTools, sanitizeTools } from "./schema-compat.js";
 import { askJson, tryAsk } from "./side-task.js";
@@ -119,11 +123,60 @@ export async function preselect(config, model, catalog, prompt, { signal, onNoti
     const reply = await tryAsk("preselect", () => askJson(config, model, preselectSystem(maxPerLoad), preselectInput(catalog, prompt), PRESELECT_SCHEMA, { name: "preselection", maxTokens, signal, onNotice }), { onNotice });
     return preselection(reply, catalog, maxPerLoad);
 }
-/** Every numeric field of one usage added into another. */
+/**
+ * The four token counts of one usage added into another. Only those: the rest are measurements a
+ * sum of would mean nothing, or would mean something only `runMetrics` knows how to weigh.
+ */
 const accumulate = (total, turn) => {
-    for (const key of Object.keys(turn))
-        total[key] += turn[key] ?? 0;
+    total.prompt += turn.prompt;
+    total.completion += turn.completion;
+    total.total += turn.total;
+    total.cached += turn.cached;
+};
+/** Whether two messages say the same thing, by identity first since most of a transcript is. */
+const sameMessage = (a, b) => a === b || JSON.stringify(a) === JSON.stringify(b);
+/** The system messages a request opens with, which a template renders ahead of the history. */
+const leadingSystem = (messages) => {
+    const end = messages.findIndex((message) => message.role !== "system");
+    return messages.slice(0, end === -1 ? messages.length : end);
 };
+/**
+ * Where a request stopped matching the one before it, earliest in the rendered prompt first — the
+ * tool block, then the system prompt, then the history — or `none-known` where it only appended.
+ */
+function breakReason(previous, messages, tools) {
+    if (previous.tools.length !== tools.length ||
+        previous.tools.some((name, at) => name !== tools[at]))
+        return "tools-changed";
+    const before = leadingSystem(previous.messages);
+    const now = leadingSystem(messages);
+    if (before.length !== now.length || before.some((message, at) => !sameMessage(message, now[at])))
+        return "system-changed";
+    if (previous.messages.length > messages.length ||
+        previous.messages.some((message, at) => !sameMessage(message, messages[at])))
+        return "history-rewritten";
+    return "none-known";
+}
+/**
+ * The share of the previous prompt a cache that kept its prefix reports as hit. Short of it by
+ * more than this is a break, not the few tokens a template re-renders at the join.
+ */
+const CACHE_KEPT = 0.9;
+/**
+ * What one turn's cache should have been and whether it was, against the request before it.
+ * Nothing before the second request, or where the previous prompt was not reported.
+ */
+function cacheDiagnosis(previous, messages, tools, usage) {
+    if (!previous || !(previous.prompt > 0))
+        return {};
+    const found = { cacheExpected: previous.prompt + previous.completion };
+    if (usage.uncached === undefined)
+        return found;
+    found.cacheBroken = usage.cached < previous.prompt * CACHE_KEPT;
+    if (found.cacheBroken)
+        found.cacheBreakReason = breakReason(previous, messages, tools);
+    return found;
+}
 /**
  * Runs a question to its answer: one `runTurn` per step, the tools it asks for between them,
  * until a turn asks for none. Throws when `maxToolIterations` is spent, when stopped, and on
@@ -133,18 +186,43 @@ const accumulate = (total, turn) => {
  * prompt unchanged from step to step, loaded tools are appended to the tool array in load order,
  * a catalogued tool called without being loaded is loaded and run rather than refused, and a
  * preselection shapes the first step. A turn cut off at `maxTokens` is said so as a notice,
- * because it otherwise reads exactly like a finished one.
+ * because it otherwise reads exactly like a finished one — or, given `maxContinuations`, is
+ * continued first. Every turn ends in a `usage` event carrying the turn's own report, with the
+ * cache compared against the request before it; see `TurnUsage` and `runMetrics`.
  *
  * @param options The config, transcript, tools and dispatcher, plus the optional hooks, events
  * and cancellation. See `AgentLoopOptions`.
  */
 export async function runAgentLoop(options) {
     const { config, system = "", tools = [], catalog = [], dispatch, hooks, signal } = options;
-    const { onEvent, onTurn, beforeStep, parallel = false, recoverToolCalls: recover = true, } = options;
+    const { onTurn, beforeStep, parallel = false, recoverToolCalls: recover = true, maxContinuations = 0, } = options;
+    const started = Date.now();
+    // What the loop emitted, less the token deltas, for `runMetrics` at the end. Stamped here rather
+    // than by the bus, which the loop does not know about.
+    const recorded = [];
+    const record = (input) => {
+        if (input.kind === "thinking" || input.kind === "output")
+            return;
+        recorded.push({
+            runId: "",
+            seq: recorded.length + 1,
+            at: Date.now(),
+            text: "",
+            name: "",
+            step: "",
+            ok: null,
+            usage: null,
+            ...input,
+        });
+    };
+    const onEvent = (input) => {
+        record(input);
+        options.onEvent?.(input);
+    };
     const client = getClient(config);
     const supports = capabilitiesFor(config.baseUrl, config.apiKey);
     const maxRetries = Math.max(0, Number(config.maxRetries) || 0);
-    const notice = (text) => onEvent?.({ kind: "notice", text });
+    const notice = (text) => onEvent({ kind: "notice", text });
     const onDemand = config.toolDiscovery === "ondemand" && catalog.length > 0;
     const loaded = new Set(onDemand ? (options.loaded ?? []) : []);
     const preselected = onDemand ? [...(options.preselected ?? [])] : [];
@@ -175,12 +253,14 @@ export async function runAgentLoop(options) {
     const usage = { prompt: 0, completion: 0, total: 0, cached: 0 };
     const toolCalls = [];
     const answered = new Map();
+    const loads = { toolsLoaded: 0, redundantLoads: 0, unknownToolNames: 0 };
+    let previous;
     for (let step = 0; step < config.maxToolIterations; step++) {
         // A stop aborts the request in flight, but a tool call already handed off runs to its own
         // end — so the signal is read between steps as well.
         signal?.throwIfAborted();
         messages = (await beforeStep?.(messages, step)) ?? messages;
-        onEvent?.({ kind: "turn", text: `turn ${step + 1}` });
+        onEvent({ kind: "turn", text: `turn ${step + 1}` });
         const routed = preselected.length > 0 && step === 0;
         const declared = routed
             ? byName(new Set(preselected))
@@ -195,7 +275,8 @@ export async function runAgentLoop(options) {
             ...(prompt ? [{ role: "system", content: prompt }] : []),
             ...withContext(messages, question ? messages.indexOf(question) : -1, gathered.context, hooks?.preface),
         ];
-        const turn = await runTurn(client, supports, (supported, refused) => buildBody(config, supported, refused, request, declared), {
+        const build = (supported, refused) => buildBody(config, supported, refused, request, declared);
+        const turnOptions = {
             model: config.model,
             droppable: Object.keys(config.extraBody ?? {}),
             maxRetries,
@@ -207,20 +288,38 @@ export async function runAgentLoop(options) {
             idleMs: timeoutMs(config),
             firstChunkMs: firstTokenMs(config) ?? 0,
             onNotice: notice,
-            onThinking: (text) => onEvent?.({ kind: "thinking", text }),
-            onOutput: (text) => onEvent?.({ kind: "output", text }),
+            onThinking: (text) => onEvent({ kind: "thinking", text }),
+            onOutput: (text) => onEvent({ kind: "output", text }),
+        };
+        const first = await runTurn(client, supports, build, turnOptions);
+        const names = declared.map((tool) => (tool.type === "function" ? tool.function.name : ""));
+        // Compared before any continuation is joined on: the cache a request meets is the one its own
+        // prompt found, and a continuation's prompt is this request's plus the reply so far.
+        Object.assign(first.usage, cacheDiagnosis(previous, request, names, first.usage), {
+            toolsDeclared: declared.length,
+            toolSchemaTokens: Math.ceil(toolsChars(declared) / charsPerTokenFor(supports, config.model)),
         });
+        const firstPrompt = first.usage.prompt;
+        const turn = maxContinuations > 0
+            ? await continueTurn(client, supports, build, first, { ...turnOptions, maxContinuations })
+            : first;
+        previous = {
+            messages: request,
+            tools: names,
+            prompt: firstPrompt,
+            completion: turn.usage.completion,
+        };
         accumulate(usage, turn.usage);
-        if (turn.usage.total > 0 || turn.usage.prompt > 0 || turn.usage.completion > 0) {
-            onEvent?.({
-                kind: "usage",
-                usage: {
-                    promptTokens: usage.prompt,
-                    completionTokens: usage.completion,
-                    totalTokens: usage.total,
-                },
-            });
-        }
+        onEvent({
+            kind: "usage",
+            usage: {
+                promptTokens: usage.prompt,
+                completionTokens: usage.completion,
+                totalTokens: usage.total,
+                cachedTokens: usage.cached,
+                turn: { ...turn.usage, finishReason: turn.finishReason },
+            },
+        });
         if (turn.finishReason === "length") {
             notice(`the model stopped at maxTokens (${config.maxTokens}); this turn is cut short`);
         }
@@ -285,6 +384,7 @@ export async function runAgentLoop(options) {
                         : {}),
                 }, hooks.onNote);
             }
+            const metrics = runMetrics(recorded, { contextLength: config.contextLength });
             return {
                 turn: shown,
                 messages,
@@ -293,11 +393,17 @@ export async function runAgentLoop(options) {
                 loaded: [...loaded],
                 used: [...used],
                 notes: gathered.notes,
+                metrics: {
+                    ...metrics,
+                    ...(onDemand ? loads : {}),
+                    wallMs: Date.now() - started,
+                    outcome: turn.finishReason === "length" ? "truncated" : "answered",
+                },
             };
         }
         const run = async ({ call, args, error: unreadable, normal }) => {
             const { name, arguments: raw } = call.function;
-            onEvent?.({ kind: "tool-call", name, text: preview(raw) });
+            onEvent({ kind: "tool-call", name, text: preview(raw) });
             let content;
             let ok = true;
             try {
@@ -306,6 +412,9 @@ export async function runAgentLoop(options) {
                 if (onDemand && name === LOAD_TOOLS) {
                     const resolved = expandNames(requestedNames(args), catalog);
                     content = loadResult(resolved, catalog, loaded);
+                    for (const hit of resolved.matched)
+                        loads[loaded.has(hit) ? "redundantLoads" : "toolsLoaded"]++;
+                    loads.unknownToolNames += resolved.unknown.length;
                     for (const hit of resolved.matched)
                         loaded.add(hit);
                     ok = resolved.matched.length > 0;
@@ -328,7 +437,7 @@ export async function runAgentLoop(options) {
                 content = errorMessage(error);
                 ok = false;
             }
-            onEvent?.({ kind: "tool-result", name, ok, text: preview(content) });
+            onEvent({ kind: "tool-result", name, ok, text: preview(content) });
             return { id: call.id, name, ok, content };
         };
         const outcomes = [];

package/dist/calibration.d.ts

@@ -0,0 +1,33 @@
+import type OpenAI from "openai";
+import type { Capabilities } from "./capabilities.ts";
+/**
+ * The characters per token to size a request to this model with, `CHARS_PER_TOKEN` until a turn
+ * has reported one.
+ *
+ * The highest of the model's last few readings rather than their mean. The estimate guards a
+ * window, and `estimateTokens` says which side of wrong that should be on: a count that comes out
+ * high refuses a run that would have fit, one that comes out low only costs the round trip the
+ * guard was saving. The highest ratio is the lowest count, and the last few rather than all of
+ * them because a run's transcript grows by appending, so the latest requests are the best
+ * likeness of the next.
+ *
+ * @param supports The endpoint, as `capabilitiesFor` hands it over.
+ * @param model The name the endpoint knows the model as, as it goes in the body.
+ */
+export declare function charsPerTokenFor(supports: Capabilities, model: string): number;
+/**
+ * Takes one reading from a request that was answered, so the next one to this model is sized by it.
+ *
+ * `runTurn` calls it after every turn whose prompt was reported, so a caller using that has
+ * nothing to do. A request carrying an image or audio is not read: a vision model charges a
+ * picture hundreds of tokens its characters say nothing about. Neither is a reading outside what a
+ * tokenizer could produce, which is a miscount and not a tokenizer.
+ *
+ * @param supports The endpoint the request went to.
+ * @param body The request as it was last sent, which names the model.
+ * @param promptTokens The prompt count the endpoint reported for it. Zero or less is no report.
+ * @returns The ratio now in force for the model.
+ */
+export declare function calibrate(supports: Capabilities, body: OpenAI.ChatCompletionCreateParamsStreaming, promptTokens: number): number;
+/** Forgets every reading, so the next request is sized at `CHARS_PER_TOKEN` again. */
+export declare function resetCalibration(): void;

package/dist/calibration.js

@@ -0,0 +1,87 @@
+import { CHARS_PER_TOKEN, requestChars, toolsChars } from "./retry.js";
+/**
+ * How many characters a token is worth on one model, learned from what its endpoint reports.
+ *
+ * Four is right for English prose and wrong for everything a tool-using run is made of: JSON
+ * schemas and tool results pack closer to two or three, so the pre-flight guard let through
+ * requests the endpoint then refused. Every turn comes back with the exact prompt count for a
+ * request whose characters were already counted to size it, so the ratio is there for the taking.
+ *
+ * Kept apart from the capability latches, and out of `exportCapabilities`, on purpose. A latch is
+ * a refusal that holds until the process dies; this is a measurement that moves every turn and is
+ * learned again from the first one after a restart. `negotiate` compares every value on a
+ * `ModelCapabilities` to tell whether a flag moved under a request in flight, so a number that
+ * moves on every turn there would read as a latch changing and re-send refusals it should throw.
+ */
+/** How many of a model's latest readings the ratio is taken over. */
+const READINGS = 4;
+/**
+ * The ratios no tokenizer produces over a whole request. Below one is a request whose tokens are
+ * mostly somewhere the characters do not count — an image — and far above four is a server that
+ * reported the uncached part of the prompt as all of it.
+ */
+const PLAUSIBLE = { least: 1, most: 8 };
+/**
+ * The latest readings per model, under the endpoint's own `Capabilities` object — the identity
+ * `capabilitiesFor` already gives one server and key — so an endpoint forgotten by
+ * `resetCapabilities` takes its readings with it. Replaced rather than cleared by `resetCalibration`,
+ * since a `WeakMap` has no `clear`.
+ */
+let readings = new WeakMap();
+/**
+ * The characters per token to size a request to this model with, `CHARS_PER_TOKEN` until a turn
+ * has reported one.
+ *
+ * The highest of the model's last few readings rather than their mean. The estimate guards a
+ * window, and `estimateTokens` says which side of wrong that should be on: a count that comes out
+ * high refuses a run that would have fit, one that comes out low only costs the round trip the
+ * guard was saving. The highest ratio is the lowest count, and the last few rather than all of
+ * them because a run's transcript grows by appending, so the latest requests are the best
+ * likeness of the next.
+ *
+ * @param supports The endpoint, as `capabilitiesFor` hands it over.
+ * @param model The name the endpoint knows the model as, as it goes in the body.
+ */
+export function charsPerTokenFor(supports, model) {
+    const known = readings.get(supports)?.get(model);
+    return known?.length ? Math.max(...known) : CHARS_PER_TOKEN;
+}
+/** Whether any message carries a part whose tokens its characters do not count. */
+const hasMedia = (messages) => messages.some(({ content }) => Array.isArray(content) &&
+    content.some((part) => part.type !== "text" && part.type !== "refusal"));
+/**
+ * Takes one reading from a request that was answered, so the next one to this model is sized by it.
+ *
+ * `runTurn` calls it after every turn whose prompt was reported, so a caller using that has
+ * nothing to do. A request carrying an image or audio is not read: a vision model charges a
+ * picture hundreds of tokens its characters say nothing about. Neither is a reading outside what a
+ * tokenizer could produce, which is a miscount and not a tokenizer.
+ *
+ * @param supports The endpoint the request went to.
+ * @param body The request as it was last sent, which names the model.
+ * @param promptTokens The prompt count the endpoint reported for it. Zero or less is no report.
+ * @returns The ratio now in force for the model.
+ */
+export function calibrate(supports, body, promptTokens) {
+    if (!(promptTokens > 0) || hasMedia(body.messages))
+        return charsPerTokenFor(supports, body.model);
+    const ratio = (requestChars(body) + toolsChars(body.tools ?? [])) / promptTokens;
+    if (ratio < PLAUSIBLE.least || ratio > PLAUSIBLE.most) {
+        return charsPerTokenFor(supports, body.model);
+    }
+    let models = readings.get(supports);
+    if (!models) {
+        models = new Map();
+        readings.set(supports, models);
+    }
+    const known = models.get(body.model) ?? [];
+    known.push(ratio);
+    if (known.length > READINGS)
+        known.shift();
+    models.set(body.model, known);
+    return charsPerTokenFor(supports, body.model);
+}
+/** Forgets every reading, so the next request is sized at `CHARS_PER_TOKEN` again. */
+export function resetCalibration() {
+    readings = new WeakMap();
+}

package/dist/capabilities.d.ts

@@ -82,6 +82,14 @@ export interface ModelCapabilities {
      * endpoint's because one key reaches models that differ here, the way they differ on effort.
      */
     structuredOutput: boolean;
+    /**
+     * Continues a trailing assistant message rather than answering afresh, which `continueTurn`
+     * relies on. llama.cpp renders one as a prefill and picks up mid-word; hosted OpenAI takes the
+     * same request and writes a new reply after it, and some servers refuse it outright — llama.cpp
+     * itself does for a template with thinking enabled. Latched off by `continueTurn` on either,
+     * never by `negotiate`, since the refusal is about a request only a continuation sends.
+     */
+    assistantPrefill: boolean;
 }
 /**
  * What this endpoint is known not to support. The same object every time, so what `negotiate`

package/dist/capabilities.js

@@ -56,6 +56,7 @@ export function modelCapabilitiesFor(supports, model) {
             chosenTemperature: true,
             refusedFields: new Set(),
             structuredOutput: true,
+            assistantPrefill: true,
         };
         supports.models.set(model, known);
     }