@cubicecho/agent-core 2.6.0 → 2.7.0

package/README.md

@@ -27,8 +27,8 @@ only, Node >=22.
 | `side-task` | One-shot calls that support a run without being one — small prompt, short answer, no tools, never worth failing the run over. `askJson` holds the answer to a schema where the server can. |
 | `hooks` | The host's side of lifecycle hooks: `gather` before a request and `notify` after, the shared context budget, `withContext` to put what they add on the turn's question, `untrusted` to fence text nobody vouched for, and `turnMessages` to hand them a transcript. Running a hook is a runner the caller passes. |
 | `events` | The in-memory bus a watcher reads while a run happens: `emit`, `watch`, `history`, `fold`. A watcher's backlog is capped and reports its own gaps. |
-| `client` | A pooled `OpenAI` client per endpoint, plus the context-window listing and its cache. |
-| `retry` | What to do when a request is lost, refused or too big: `isTransient`, `backoffMs`, `ContextOverflow`, `EndpointSilent`, `requestTokens`. |
+| `client` | A pooled `OpenAI` client per endpoint, plus the context window: the served one where a local server says, the listed one otherwise, and their caches. |
+| `retry` | What to do when a request is lost, refused or too big: `isTransient`, `isModelLoading`, `backoffMs`, `ContextOverflow`, `EndpointSilent`, `requestTokens`. |
 | `config` | The structural interfaces every function here asks for. |
 | `run-turn` | `runTurn`: one turn with the retry loop around the negotiation around the stream. The whole loop, for a caller that wants it rather than its parts. Sizes the request against an opt-in `contextLimit`. |
 | `agent-loop` | `runAgentLoop`: the loop above a turn — `runTurn` per step, the tools between, `load_tools` and preselection handled, until the model stops asking. Plus the parts it is made of: `buildBody`, `preselect`, `preview`, and `resolveApiKey` for a caller deciding which key an endpoint gets. |
@@ -145,11 +145,21 @@ is worth another attempt. The content-free `{"role":"assistant"}` most servers o
 does not set it: nothing has been shown to anybody yet, so an endpoint that primes the stream and
 then wedges is retried like one that never answered at all.
 
-`idleMs` is silence, not a deadline: the timer is rearmed on every chunk, so a model that is
-still talking is never cut off however long it takes, and one that has stopped answering raises
+`idleMs` is silence, not a deadline: the timer is rearmed on every chunk, so a model that is still
+talking is never cut off however long it takes, and one that has stopped answering raises
 `EndpointSilent` rather than hanging the run. `timeoutMs(config)` returns `undefined` for a
-`requestTimeoutSeconds` of zero or absent, which waits forever — what a local model answering
-slowly needs.
+`requestTimeoutSeconds` of zero or absent, which waits forever — what a local model answering slowly
+needs.
+
+The first chunk gets its own allowance, `firstChunkMs`, because the first wait is prefill: tens of
+seconds for a long prompt on a local GPU, minutes on a CPU, and longer again when the server is
+loading the model on demand. It holds until a chunk carries something, so an empty
+`{"role":"assistant"}` sent before the prompt is read does not start the idle clock.
+`firstTokenMs(config)` reads `firstTokenSeconds` off the endpoint, and five times
+`requestTimeoutSeconds` where that is absent. With a watchdog armed, the SDK's own timer is switched
+off for the stream; it runs until the headers arrive, which is the end of prefill, and used to
+abandon one at the idle number. `requestTimeoutSeconds` still bounds calls that do not stream, side
+tasks and model listings, the same way.
 
 ## Structured side tasks
 
@@ -191,6 +201,16 @@ const turn = await runTurn(client, supports, build, {
 });
 ```
 
+`contextLimitFor` answers the operator's number when there is one. Otherwise it asks for the window
+the server is actually serving the model in (`servedWindow`): llama.cpp's `/props`
+(`default_generation_settings.n_ctx`) and LM Studio's `/api/v0/models` (`loaded_context_length`).
+That differs from the trained window in the case the guard exists for, a 256k model started at `-c
+16384`. A server with neither route is latched and not asked again. Failing both, it reads the
+`/v1/models` listing: `max_model_len` from vLLM, `context_length` from OpenRouter, and llama.cpp's
+`meta.n_ctx_train`, which is only the trained window. Ollama reports no window on any route this
+reads, and truncates an over-long prompt rather than refusing it, so on Ollama pass `contextLength`
+or there is no guard at all.
+
 What is weighed is the prompt plus the reply ceiling the body carries, under whichever spelling
 was chosen, because that is what the endpoint weighs: a 30k prompt into a 32k window with
 `max_tokens: 4096` is refused there, so it is refused here. A body with no ceiling reserves
@@ -209,6 +229,14 @@ refuses instead — one round trip later — and `runTurn` reads that refusal ba
 original error as `cause`. A rate limit borrows those words and means the opposite ("Request too
 large for gpt-4o ... on tokens per min"); that is ruled out and waited through as the 429 it is.
 
+A server still loading the model is waited for on its own clock. llama.cpp answers 503 `Loading
+model` (type `unavailable_error`) until the weights are mapped, thirty to ninety seconds for a large
+model from a cold cache, and a router build says the same while it swaps models; `backoffMs` would
+give up inside fifteen. `isModelLoading` recognises it, and `runTurn` polls every `LOADING_POLL_MS`
+for up to `loadingTimeoutMs` (two minutes by default, zero to turn it off) without spending
+`maxRetries`, with one notice at the start. `runAgentLoop` reads it as `loadingTimeoutSeconds` off
+the config. A 503 that says nothing about loading stays on the ordinary backoff.
+
 ## The loop
 
 `runAgentLoop` is the part of an agent that three servers had each written, and that had drifted
@@ -221,7 +249,7 @@ did. What it does not know is what the run is for — the prompt, the tools, and
 import { runAgentLoop, emit } from "@cubicecho/agent-core";
 
 const { turn, messages, usage, loaded } = await runAgentLoop({
-  config,                         // Endpoint & ModelParams & { maxToolIterations, toolDiscovery?, maxRetries?, contextLength? }
+  config,                         // Endpoint & ModelParams & { maxToolIterations, toolDiscovery?, maxRetries?, loadingTimeoutSeconds?, contextLength? }
   system,                         // sent as the first message; on-demand mode appends the catalogue
   messages: history,              // ending in the question; not written to
   tools,                          // every tool the run may reach

package/dist/agent-loop.js

@@ -1,5 +1,5 @@
 import { capabilitiesFor } from "./capabilities.js";
-import { getClient, NO_KEY, timeoutMs } from "./client.js";
+import { firstTokenMs, getClient, NO_KEY, timeoutMs } from "./client.js";
 import { errorMessage } from "./errors.js";
 import { gather, notify, turnIndex, turnMessages, withContext, } from "./hooks.js";
 import { runTurn } from "./run-turn.js";
@@ -187,9 +187,13 @@ export async function runAgentLoop(options) {
             model: config.model,
             droppable: Object.keys(config.extraBody ?? {}),
             maxRetries,
+            ...(config.loadingTimeoutSeconds === undefined
+                ? {}
+                : { loadingTimeoutMs: Math.max(0, config.loadingTimeoutSeconds) * 1000 }),
             contextLimit: config.contextLength ?? 0,
             signal,
             idleMs: timeoutMs(config),
+            firstChunkMs: firstTokenMs(config) ?? 0,
             onNotice: notice,
             onThinking: (text) => onEvent?.({ kind: "thinking", text }),
             onOutput: (text) => onEvent?.({ kind: "output", text }),

package/dist/client.d.ts

@@ -12,6 +12,14 @@ export declare const NO_KEY = "agent-core";
  * @param config Read for `requestTimeoutSeconds` alone.
  */
 export declare const timeoutMs: (config: Pick<Endpoint, "requestTimeoutSeconds">) => number | undefined;
+/** How many idle windows the first chunk gets when `firstTokenSeconds` is not given. */
+export declare const FIRST_TOKEN_FACTOR = 5;
+/**
+ * The wait for a streamed turn's first chunk, in the SDK's spelling: `undefined` is no limit.
+ *
+ * @param config Read for `firstTokenSeconds`, and `requestTimeoutSeconds` where that is absent.
+ */
+export declare const firstTokenMs: (config: Pick<Endpoint, "requestTimeoutSeconds" | "firstTokenSeconds">) => number | undefined;
 /**
  * The client for an endpoint, built once and kept.
  *
@@ -67,6 +75,23 @@ export declare const endpointId: (config: {
     baseUrl: string;
     apiKey?: string;
 }) => string;
+/**
+ * The window a local server is actually serving a model in, which its listing does not say.
+ *
+ * llama.cpp reports it on `/props` as `default_generation_settings.n_ctx`, per slot, and LM
+ * Studio on `/api/v0/models` as `loaded_context_length` while the model is loaded. Both differ
+ * from the trained window in the case that matters, a model started in a smaller one than it was
+ * built for, and reading the trained one lets an overflow through the guard meant to catch it.
+ * Ollama reports nothing on either; an operator there has to declare the window.
+ *
+ * Zero where no answer was found. A server without either route, which is every hosted API, is
+ * latched and not asked again; one that could not be reached is not remembered at all.
+ *
+ * @param config The endpoint, plus the model whose window is wanted.
+ */
+export declare function servedWindow(config: Endpoint & {
+    model: string;
+}): Promise<number>;
 /**
  * Asks an endpoint what it serves, and remembers the answer.
  *
@@ -81,10 +106,11 @@ export declare function listModels(config: Endpoint): Promise<ModelInfo[]>;
  * happily load a 256k model at `-c 16384` and go on listing it as 256k — and a run refused on
  * the honest-looking number is a run that fails at the endpoint instead.
  *
- * Otherwise the endpoint's listing is asked — once, and again whenever it does not name this
- * model, since a model can arrive after the first listing was taken. A server that will not
- * list models still has to be able to run a turn: a failure here is an unknown window, not a
- * failed run.
+ * Otherwise the window the server is actually serving the model in, where it has an API that
+ * says (`servedWindow`), and failing that the endpoint's listing — asked once, and again whenever
+ * it does not name this model, since a model can arrive after the first listing was taken. A
+ * server that will not list models still has to be able to run a turn: a failure here is an
+ * unknown window, not a failed run.
  *
  * @param config The endpoint, plus the model whose window is wanted.
  * @param declared The operator's own number. Above zero it wins and the endpoint is not asked.

package/dist/client.js

@@ -47,6 +47,20 @@ const clients = new Map();
  * constraint.
  */
 const MAX_CLIENTS = 32;
+/** How many idle windows the first chunk gets when `firstTokenSeconds` is not given. */
+export const FIRST_TOKEN_FACTOR = 5;
+/**
+ * The wait for a streamed turn's first chunk, in the SDK's spelling: `undefined` is no limit.
+ *
+ * @param config Read for `firstTokenSeconds`, and `requestTimeoutSeconds` where that is absent.
+ */
+export const firstTokenMs = (config) => {
+    if (config.firstTokenSeconds === undefined) {
+        const idle = timeoutMs(config);
+        return idle === undefined ? undefined : idle * FIRST_TOKEN_FACTOR;
+    }
+    return config.firstTokenSeconds > 0 ? config.firstTokenSeconds * 1000 : undefined;
+};
 /**
  * The client for an endpoint, built once and kept.
  *
@@ -89,12 +103,13 @@ export function getClient(config) {
     return client;
 }
 /**
- * The context window, spelled every way a server spells it.
+ * The context window, spelled every way a server spells it at the top of a listing entry.
  *
  * None of these is in the OpenAI listing schema, so every server that says anything says it as
- * an extra key of its own: `context_length` is llama.cpp and LM Studio, `max_model_len` vLLM,
- * `n_ctx` the raw llama bindings. Whichever turns up first is taken — a server reporting two
- * of them is reporting the same number twice.
+ * an extra key of its own: `max_model_len` is vLLM, `context_length` OpenRouter, `n_ctx` the raw
+ * llama bindings. Whichever turns up first is taken — a server reporting two of them is
+ * reporting the same number twice. llama.cpp puts its number under `meta` instead, and LM Studio
+ * and Ollama put none on this route at all; see `servedWindow`.
  */
 const CONTEXT_KEYS = [
     "context_length",
@@ -103,14 +118,19 @@ const CONTEXT_KEYS = [
     "context_window",
     "n_ctx",
 ];
+const positive = (value) => (typeof value === "number" && value > 0 ? value : 0);
 function contextLengthOf(model) {
     const record = model;
     for (const key of CONTEXT_KEYS) {
-        const value = record[key];
-        if (typeof value === "number" && value > 0)
+        const value = positive(record[key]);
+        if (value)
             return value;
     }
-    return 0;
+    // llama.cpp's, and the window the model was trained with rather than the one it is served in:
+    // a 256k model started at `-c 16384` lists 262144 here. Better than nothing, and why the
+    // served window is asked for first. `meta` is `null` while the model loads.
+    const meta = record.meta;
+    return positive(meta?.n_ctx_train);
 }
 /**
  * The last listing from each endpoint, so a run can size its window without a round trip.
@@ -172,6 +192,88 @@ export const endpointKey = (config) => JSON.stringify([config.baseUrl, config.ap
  * @param config Read for `baseUrl` and `apiKey` alone, as `endpointKey` reads it.
  */
 export const endpointId = (config) => createHash("sha256").update(endpointKey(config)).digest("hex");
+/**
+ * Served windows found by asking a server's own API, keyed on endpoint and model together.
+ *
+ * A model's entry stays until `resetClients`, like a listing; a server that answered without one
+ * is asked again after `LISTING_MISS_MS`, like a listing that did not name the model.
+ */
+const served = new Map();
+/** Endpoints that answered both probes with a refusal, and are not asked again. */
+const unserved = new Set();
+/** How long a probe may take before the window is taken from the listing instead. */
+const PROBE_TIMEOUT_MS = 10_000;
+/** The server root behind an OpenAI-compatible base URL, which is where the native APIs live. */
+const rootOf = (baseUrl) => baseUrl.replace(/\/+$/, "").replace(/\/v1$/, "");
+/** A route this server does not have, rather than one that failed to answer. */
+const NOT_THERE = new Set([404, 405, 501]);
+/**
+ * Asks one native endpoint. `missing` is a server without the route; `body` is absent for that
+ * and for any other refusal, and a server that could not be reached throws.
+ */
+async function probe(config, path) {
+    const apiKey = config.apiKey || undefined;
+    const response = await fetch(`${rootOf(config.baseUrl)}${path}`, {
+        headers: apiKey ? { authorization: `Bearer ${apiKey}` } : {},
+        signal: AbortSignal.timeout(timeoutMs(config) ?? PROBE_TIMEOUT_MS),
+    });
+    if (!response.ok)
+        return { missing: NOT_THERE.has(response.status) };
+    try {
+        return { missing: false, body: await response.json() };
+    }
+    catch {
+        return { missing: false };
+    }
+}
+/**
+ * The window a local server is actually serving a model in, which its listing does not say.
+ *
+ * llama.cpp reports it on `/props` as `default_generation_settings.n_ctx`, per slot, and LM
+ * Studio on `/api/v0/models` as `loaded_context_length` while the model is loaded. Both differ
+ * from the trained window in the case that matters, a model started in a smaller one than it was
+ * built for, and reading the trained one lets an overflow through the guard meant to catch it.
+ * Ollama reports nothing on either; an operator there has to declare the window.
+ *
+ * Zero where no answer was found. A server without either route, which is every hosted API, is
+ * latched and not asked again; one that could not be reached is not remembered at all.
+ *
+ * @param config The endpoint, plus the model whose window is wanted.
+ */
+export async function servedWindow(config) {
+    const endpoint = endpointKey(config);
+    if (unserved.has(endpoint))
+        return 0;
+    const key = JSON.stringify([endpoint, config.model]);
+    const known = served.get(key);
+    if (known && (known.window > 0 || Date.now() - known.at < LISTING_MISS_MS))
+        return known.window;
+    let window = 0;
+    try {
+        // Named, for a llama.cpp router serving several models; a single-model server ignores it.
+        const props = await probe(config, `/props?model=${encodeURIComponent(config.model)}`);
+        const settings = props.body
+            ?.default_generation_settings;
+        window = positive(settings?.n_ctx);
+        if (!window) {
+            const lmstudio = await probe(config, "/api/v0/models");
+            const { data } = (lmstudio.body ?? {});
+            const entry = Array.isArray(data)
+                ? data.find((model) => model?.id === config.model)
+                : undefined;
+            window = positive(entry?.loaded_context_length);
+            if (props.missing && lmstudio.missing) {
+                unserved.add(endpoint);
+                return 0;
+            }
+        }
+    }
+    catch {
+        return 0;
+    }
+    served.set(key, { window, at: Date.now() });
+    return window;
+}
 /**
  * Asks an endpoint what it serves, and remembers the answer.
  *
@@ -193,10 +295,11 @@ export async function listModels(config) {
  * happily load a 256k model at `-c 16384` and go on listing it as 256k — and a run refused on
  * the honest-looking number is a run that fails at the endpoint instead.
  *
- * Otherwise the endpoint's listing is asked — once, and again whenever it does not name this
- * model, since a model can arrive after the first listing was taken. A server that will not
- * list models still has to be able to run a turn: a failure here is an unknown window, not a
- * failed run.
+ * Otherwise the window the server is actually serving the model in, where it has an API that
+ * says (`servedWindow`), and failing that the endpoint's listing — asked once, and again whenever
+ * it does not name this model, since a model can arrive after the first listing was taken. A
+ * server that will not list models still has to be able to run a turn: a failure here is an
+ * unknown window, not a failed run.
  *
  * @param config The endpoint, plus the model whose window is wanted.
  * @param declared The operator's own number. Above zero it wins and the endpoint is not asked.
@@ -204,6 +307,9 @@ export async function listModels(config) {
 export async function contextLimitFor(config, declared = 0) {
     if (declared > 0)
         return declared;
+    const window = await servedWindow(config);
+    if (window > 0)
+        return window;
     const key = endpointKey(config);
     const listed = () => listings.get(key)?.find((model) => model.id === config.model);
     // The listing is asked for again when it does not name this model, rather than only when
@@ -245,4 +351,6 @@ export function resetClients() {
     clients.clear();
     listings.clear();
     misses.clear();
+    served.clear();
+    unserved.clear();
 }

package/dist/config.d.ts

@@ -18,7 +18,13 @@ export interface Endpoint {
     /** Empty is normal — a local server ignores it. See `getClient` for what is sent instead. */
     apiKey: string;
     /**
-     * Zero, less, or absent means no limit — what a local model answering slowly needs.
+     * How long an endpoint may go quiet. Zero, less, or absent means no limit — what a local model
+     * answering slowly needs.
+     *
+     * Not the whole request, which may take as long as the model keeps talking. On a streamed turn
+     * it is the silence allowed between chunks, the idle watchdog; the wait for the first chunk is
+     * `firstTokenSeconds`. On a call that does not stream, a side task or a model listing, it is
+     * the SDK's timer, which runs until the response headers arrive.
      *
      * Optional because a consumer that has no timeout to give should not have to invent one. Two
      * of the three servers this was extracted from carry no such field, and requiring it made
@@ -26,6 +32,18 @@ export interface Endpoint {
      * number standing in for an absent one.
      */
     requestTimeoutSeconds?: number;
+    /**
+     * How long a streamed turn may wait for its first chunk. Absent is five times
+     * `requestTimeoutSeconds`; zero or less is no limit.
+     *
+     * Its own number because the first wait is prefill, and on a local server prefill of a long
+     * prompt is tens of seconds on a GPU and minutes on a CPU, where the gap between tokens is a
+     * fraction of a second. A server loading the model on demand, Ollama after `keep_alive` or LM
+     * Studio just in time, holds the request open for the same reason. One number for both waits
+     * was either too slow to notice a wedged stream or tight enough to abandon a prefill, and a
+     * retry pays for that prefill again from nothing.
+     */
+    firstTokenSeconds?: number;
 }
 /** What to ask the model for. */
 export interface ModelParams {
@@ -68,6 +86,12 @@ export interface ToolPolicy {
 /** How many times a lost or refused request is worth sending again. See `retry.ts`. */
 export interface RetryPolicy {
     maxRetries: number;
+    /**
+     * How long to wait for a local server that says it is still loading the model, absent two
+     * minutes and zero not at all. Separate from `maxRetries`, which is sized for a request that
+     * was lost rather than for weights being read off a disk. See `isModelLoading`.
+     */
+    loadingTimeoutSeconds?: number;
 }
 /**
  * A whole agent configuration — every part, plus the two fields that belong to no group.

package/dist/index.d.ts

@@ -12,14 +12,14 @@
 export { type AgentLoopHooks, type AgentLoopOptions, type AgentLoopResult, buildBody, preselect, preview, resolveApiKey, runAgentLoop, type ToolCallOutcome, type ToolCallRequest, } from "./agent-loop.ts";
 export { type Capabilities, capabilitiesFor, type ModelCapabilities, modelCapabilitiesFor, type NegotiateOptions, negotiate, resetCapabilities, } from "./capabilities.ts";
 export type { CatalogServer } from "./catalog.ts";
-export { contextLimitFor, getClient, listModels, type ModelInfo, NO_KEY, resetClients, timeoutMs, } from "./client.ts";
+export { contextLimitFor, FIRST_TOKEN_FACTOR, firstTokenMs, getClient, listModels, type ModelInfo, NO_KEY, resetClients, servedWindow, timeoutMs, } from "./client.ts";
 export { COMPACT_AT, type CompactionOptions, type CompactionPlan, compactTranscript, KEEP_RATIO, type PruneOptions, planCompaction, pruneToolResults, SUMMARY_LEAD, SUMMARY_PROMPT, summariser, summaryInput, } from "./compaction.ts";
 export type { AgentConfig, Endpoint, ModelParams, RetryPolicy, ToolPolicy, } from "./config.ts";
 export { errorMessage } from "./errors.ts";
 export { configureEvents, type EventBusOptions, emit, endRun, fold, history, type RunEvent, type RunEventInput, type RunEventKind, type RunUsage, resetEvents, watch, } from "./events.ts";
 export { assembleContext, configureHooks, type Gathered, gather, HOOK_CONTEXT_TOKENS, HOOK_EVENTS, HOOK_PREFACE, type HookContext, type HookEvent, type HookMessage, type HookNote, type HookOptions, type HookOutcome, type HookRunner, INJECT_EVENTS, notify, resetHooks, turnIndex, turnMessages, UNTRUSTED_PREFACE, untrusted, withContext, } from "./hooks.ts";
 export { resetAll } from "./reset.ts";
-export { backoffMs, ContextOverflow, compact, EndpointSilent, isOverflow, isTransient, messageTokens, requestTokens, SMALLEST_LIKELY_WINDOW, sleep, } from "./retry.ts";
+export { backoffMs, ContextOverflow, compact, EndpointSilent, isModelLoading, isOverflow, isTransient, LOADING_POLL_MS, LOADING_TIMEOUT_MS, messageTokens, requestTokens, SMALLEST_LIKELY_WINDOW, sleep, } from "./retry.ts";
 export { type RunTurnOptions, runTurn } from "./run-turn.ts";
 export { isGrammarError, relaxTools, sanitizeTools } from "./schema-compat.ts";
 export { type AskJsonOptions, ask, askJson, clean, listLines, parseJson, resetHints, type SideTaskOptions, tryAsk, } from "./side-task.ts";

package/dist/index.js

@@ -11,13 +11,13 @@
  */
 export { buildBody, preselect, preview, resolveApiKey, runAgentLoop, } from "./agent-loop.js";
 export { capabilitiesFor, modelCapabilitiesFor, negotiate, resetCapabilities, } from "./capabilities.js";
-export { contextLimitFor, getClient, listModels, NO_KEY, resetClients, timeoutMs, } from "./client.js";
+export { contextLimitFor, FIRST_TOKEN_FACTOR, firstTokenMs, getClient, listModels, NO_KEY, resetClients, servedWindow, timeoutMs, } from "./client.js";
 export { COMPACT_AT, compactTranscript, KEEP_RATIO, planCompaction, pruneToolResults, SUMMARY_LEAD, SUMMARY_PROMPT, summariser, summaryInput, } from "./compaction.js";
 export { errorMessage } from "./errors.js";
 export { configureEvents, emit, endRun, fold, history, resetEvents, watch, } from "./events.js";
 export { assembleContext, configureHooks, gather, HOOK_CONTEXT_TOKENS, HOOK_EVENTS, HOOK_PREFACE, INJECT_EVENTS, notify, resetHooks, turnIndex, turnMessages, UNTRUSTED_PREFACE, untrusted, withContext, } from "./hooks.js";
 export { resetAll } from "./reset.js";
-export { backoffMs, ContextOverflow, compact, EndpointSilent, isOverflow, isTransient, messageTokens, requestTokens, SMALLEST_LIKELY_WINDOW, sleep, } from "./retry.js";
+export { backoffMs, ContextOverflow, compact, EndpointSilent, isModelLoading, isOverflow, isTransient, LOADING_POLL_MS, LOADING_TIMEOUT_MS, messageTokens, requestTokens, SMALLEST_LIKELY_WINDOW, sleep, } from "./retry.js";
 export { runTurn } from "./run-turn.js";
 export { isGrammarError, relaxTools, sanitizeTools } from "./schema-compat.js";
 export { ask, askJson, clean, listLines, parseJson, resetHints, tryAsk, } from "./side-task.js";

package/dist/retry.d.ts

@@ -93,6 +93,21 @@ export declare const SMALLEST_LIKELY_WINDOW = 8192;
  * @param error The rejection, as caught. What is not an SDK error is not transient.
  */
 export declare function isTransient(error: unknown): boolean;
+/**
+ * Whether a failure is a local server still loading the model, rather than one failing to serve.
+ *
+ * llama.cpp answers 503 `Loading model` with type `unavailable_error` from the moment it starts
+ * until the weights are mapped, and a router build says the same while it swaps models. That is
+ * thirty to ninety seconds for a large model from a cold page cache, and `backoffMs` gives up
+ * inside fifteen: sized for a busy host, not for one reading a file. A plain 503 is not this.
+ *
+ * @param error The rejection, as caught.
+ */
+export declare function isModelLoading(error: unknown): boolean;
+/** How long to wait between asking a loading server again. */
+export declare const LOADING_POLL_MS = 3000;
+/** How long `runTurn` waits for a model to load unless told otherwise. */
+export declare const LOADING_TIMEOUT_MS = 120000;
 /**
  * Exponential, with jitter so several tasks failing at once do not return in lockstep.
  *

package/dist/retry.js

@@ -214,6 +214,27 @@ export function isTransient(error) {
     const { status } = error;
     return status === 408 || status === 409 || status === 429 || (status ?? 0) >= 500;
 }
+/**
+ * Whether a failure is a local server still loading the model, rather than one failing to serve.
+ *
+ * llama.cpp answers 503 `Loading model` with type `unavailable_error` from the moment it starts
+ * until the weights are mapped, and a router build says the same while it swaps models. That is
+ * thirty to ninety seconds for a large model from a cold page cache, and `backoffMs` gives up
+ * inside fifteen: sized for a busy host, not for one reading a file. A plain 503 is not this.
+ *
+ * @param error The rejection, as caught.
+ */
+export function isModelLoading(error) {
+    if (!(error instanceof OpenAI.APIError) || error.status !== 503)
+        return false;
+    const body = error.error;
+    return (body?.type === "unavailable_error" ||
+        /loading model|model is loading|unavailable_error/i.test(`${error.message} ${body?.message ?? ""}`));
+}
+/** How long to wait between asking a loading server again. */
+export const LOADING_POLL_MS = 3000;
+/** How long `runTurn` waits for a model to load unless told otherwise. */
+export const LOADING_TIMEOUT_MS = 120_000;
 /**
  * Exponential, with jitter so several tasks failing at once do not return in lockstep.
  *

package/dist/run-turn.d.ts

@@ -63,6 +63,15 @@ export interface RunTurnOptions extends Omit<StreamTurnOptions, "produced"> {
      * of `extraBody`, ordinarily. See `NegotiateOptions.droppable`; it needs `model` too.
      */
     droppable?: Iterable<string>;
+    /**
+     * How long to wait on a server answering that the model is still loading, `LOADING_TIMEOUT_MS`
+     * unless given; zero gives up on the first such answer like any other 503.
+     *
+     * Polled every `LOADING_POLL_MS` without spending `maxRetries`, and announced once rather than
+     * per poll. A consumer that starts alongside its llama.cpp, or asks a router for a model it
+     * has to swap in, meets this on its first request every time.
+     */
+    loadingTimeoutMs?: number;
 }
 /**
  * `request` is a callback rather than a body because the body has to be rebuilt from whatever
@@ -77,4 +86,4 @@ export interface RunTurnOptions extends Omit<StreamTurnOptions, "produced"> {
  * @param options Retry budget, context limit, the model to negotiate for, notices, and the
  * stream's own callbacks.
  */
-export declare function runTurn(client: OpenAI, supports: Capabilities, request: (supports: Capabilities, model: ModelCapabilities | undefined) => OpenAI.ChatCompletionCreateParamsStreaming, { maxRetries, onNotice, contextLimit, model, droppable, ...stream }?: RunTurnOptions): Promise<Turn>;
+export declare function runTurn(client: OpenAI, supports: Capabilities, request: (supports: Capabilities, model: ModelCapabilities | undefined) => OpenAI.ChatCompletionCreateParamsStreaming, { maxRetries, onNotice, contextLimit, model, droppable, loadingTimeoutMs, ...stream }?: RunTurnOptions): Promise<Turn>;

package/dist/run-turn.js

@@ -1,6 +1,6 @@
 import { negotiate } from "./capabilities.js";
 import { errorMessage } from "./errors.js";
-import { backoffMs, ContextOverflow, compact, isOverflow, isTransient, requestTokens, SMALLEST_LIKELY_WINDOW, sleep, } from "./retry.js";
+import { backoffMs, ContextOverflow, compact, isModelLoading, isOverflow, isTransient, LOADING_POLL_MS, LOADING_TIMEOUT_MS, requestTokens, SMALLEST_LIKELY_WINDOW, sleep, } from "./retry.js";
 import { streamTurn } from "./stream.js";
 /**
  * `request` is a callback rather than a body because the body has to be rebuilt from whatever
@@ -15,7 +15,7 @@ import { streamTurn } from "./stream.js";
  * @param options Retry budget, context limit, the model to negotiate for, notices, and the
  * stream's own callbacks.
  */
-export async function runTurn(client, supports, request, { maxRetries = 0, onNotice, contextLimit = 0, model, droppable, ...stream } = {}) {
+export async function runTurn(client, supports, request, { maxRetries = 0, onNotice, contextLimit = 0, model, droppable, loadingTimeoutMs = LOADING_TIMEOUT_MS, ...stream } = {}) {
     // Sized once rather than per build. `request` is called again for every downgrade and every
     // retry, but a downgraded body is strictly smaller than the one before it and the transcript
     // does not change between attempts — so the first body is the one worth measuring, and
@@ -42,6 +42,9 @@ export async function runTurn(client, supports, request, { maxRetries = 0, onNot
         }
         return body;
     };
+    // When the server first said it was loading. Unset until then, and never reset: a model that
+    // loads, fails and loads again has had its allowance.
+    let loadingSince;
     for (let attempt = 0;; attempt++) {
         const produced = { any: false };
         try {
@@ -66,6 +69,20 @@ export async function runTurn(client, supports, request, { maxRetries = 0, onNot
             if (!(error instanceof ContextOverflow) && isOverflow(errorMessage(error))) {
                 throw new ContextOverflow(errorMessage(error), { cause: error });
             }
+            if (isModelLoading(error) && loadingTimeoutMs > 0) {
+                const now = Date.now();
+                if (loadingSince === undefined) {
+                    loadingSince = now;
+                    onNotice?.(`${model ?? "the model"} is still loading — waiting up to ${compact(loadingTimeoutMs / 1000)}s`);
+                }
+                if (now - loadingSince < loadingTimeoutMs) {
+                    // Not an attempt: the request was never looked at, and a two-minute load would
+                    // otherwise have to be bought with a retry budget meant for dropped connections.
+                    attempt--;
+                    await sleep(LOADING_POLL_MS, stream.signal);
+                    continue;
+                }
+            }
             if (attempt >= maxRetries || !isTransient(error))
                 throw error;
             const wait = backoffMs(attempt);

package/dist/stream.d.ts

@@ -75,6 +75,13 @@ export interface StreamTurnOptions {
      * needs.
      */
     idleMs?: number;
+    /**
+     * Silence allowed before the first chunk, `idleMs` unless given; zero waits forever.
+     *
+     * The first wait is prefill, or a server loading the model, and is routinely many times the
+     * gap between tokens. See `Endpoint.firstTokenSeconds` and `firstTokenMs`.
+     */
+    firstChunkMs?: number;
     /** Set by the first chunk that carries anything, so a failed call knows if it can be retried. */
     produced?: Produced;
     /** The model's scratchpad, as it arrives. */
@@ -98,4 +105,4 @@ export interface StreamTurnOptions {
  * @param body The request, which must set `stream: true`.
  * @param options Cancellation, the idle watchdog, and the token callbacks.
  */
-export declare function streamTurn(client: OpenAI, body: OpenAI.ChatCompletionCreateParamsStreaming, { signal, idleMs, produced, onThinking, onOutput }?: StreamTurnOptions): Promise<Turn>;
+export declare function streamTurn(client: OpenAI, body: OpenAI.ChatCompletionCreateParamsStreaming, { signal, idleMs, firstChunkMs, produced, onThinking, onOutput }?: StreamTurnOptions): Promise<Turn>;

package/dist/stream.js

@@ -1,4 +1,6 @@
 import { EndpointSilent } from "./retry.js";
+/** The largest delay a timer takes, which is as close to none as the SDK's timeout option goes. */
+const NO_SDK_TIMEOUT = 2 ** 31 - 1;
 /**
  * Runs one turn as a stream, reporting tokens as they arrive and assembling them back into a
  * message.
@@ -15,7 +17,7 @@ import { EndpointSilent } from "./retry.js";
  * @param body The request, which must set `stream: true`.
  * @param options Cancellation, the idle watchdog, and the token callbacks.
  */
-export async function streamTurn(client, body, { signal, idleMs, produced, onThinking, onOutput } = {}) {
+export async function streamTurn(client, body, { signal, idleMs, firstChunkMs, produced, onThinking, onOutput } = {}) {
     // Silence, not duration: the timer is rearmed on every chunk, so a model that is still
     // talking is never cut off however long it takes, and one that has stopped talking does not
     // hang the run until someone notices. A request that never answers at all is the same case
@@ -23,14 +25,20 @@ export async function streamTurn(client, body, { signal, idleMs, produced, onThi
     const watchdog = new AbortController();
     const linked = signal ? AbortSignal.any([signal, watchdog.signal]) : watchdog.signal;
     let idle;
-    const rearm = () => {
-        if (!idleMs)
-            return;
+    const first = firstChunkMs ?? idleMs;
+    // Until the model says something the wait is still prefill, however many empty chunks a
+    // server sends first: some open the stream with `{"role":"assistant"}` before they have read
+    // the prompt, and switching to the idle allowance on that abandoned the prefill it preceded.
+    let talking = false;
+    const rearm = (carried) => {
+        talking ||= carried;
+        const ms = talking ? idleMs : first;
         clearTimeout(idle);
-        idle = setTimeout(() => watchdog.abort(), idleMs);
+        if (ms)
+            idle = setTimeout(() => watchdog.abort(), ms);
     };
     try {
-        rearm();
+        rearm(false);
         return await collect();
     }
     catch (error) {
@@ -39,7 +47,9 @@ export async function streamTurn(client, body, { signal, idleMs, produced, onThi
         // this backwards records a stopped run as an endpoint fault, which nobody notices until
         // they read the row and disbelieve it.
         if (watchdog.signal.aborted && !signal?.aborted) {
-            throw new EndpointSilent(`the model endpoint sent nothing for ${(idleMs ?? 0) / 1000}s`);
+            const waited = talking ? idleMs : first;
+            const before = talking || first === idleMs ? "" : " before its first token";
+            throw new EndpointSilent(`the model endpoint sent nothing for ${(waited ?? 0) / 1000}s${before}`);
         }
         throw error;
     }
@@ -47,15 +57,23 @@ export async function streamTurn(client, body, { signal, idleMs, produced, onThi
         clearTimeout(idle);
     }
     async function collect() {
-        const stream = await client.chat.completions.create(body, { signal: linked });
+        // The SDK's own timer runs until the headers arrive, which for a stream is the end of
+        // prefill, and it was set from the idle number. Where a watchdog is armed it covers that
+        // wait already, with the allowance meant for it, so the SDK is told to leave it alone.
+        const watched = first || idleMs;
+        const stream = await client.chat.completions.create(body, {
+            signal: linked,
+            ...(watched ? { timeout: NO_SDK_TIMEOUT } : {}),
+        });
         const content = [];
         const calls = new Map();
         const usage = { prompt: 0, completion: 0, total: 0, cached: 0 };
         let finishReason = "";
         for await (const chunk of stream) {
             // Rearmed on every chunk, latched below on only some: a priming chunk is the endpoint
-            // being alive, which is all the watchdog is asking about.
-            rearm();
+            // being alive, which is all the watchdog is asking about. Which allowance it rearms with
+            // moves from the first chunk's to the idle one when a chunk carries something.
+            rearm(false);
             // Assigned rather than accumulated. `stream_options.include_usage` sends one final chunk
             // and the two agree there, but a server that reports cumulatively per chunk makes a sum
             // of sums out of an accumulator — and a token count wrong by a factor of the chunk count
@@ -87,7 +105,10 @@ export async function streamTurn(client, body, { signal, idleMs, produced, onThi
             // fragments count even though no callback reports them: a partial call is state the turn
             // has accumulated, and losing a retry is the safer half of that trade. Set before the
             // callbacks, so a watcher that throws mid-token cannot be told the same token twice.
-            if (produced && (thinking || delta.content || delta.tool_calls?.length))
+            const carried = Boolean(thinking || delta.content || delta.tool_calls?.length);
+            if (carried && !talking)
+                rearm(true);
+            if (produced && carried)
                 produced.any = true;
             if (thinking)
                 onThinking?.(thinking);

package/llms.txt

@@ -45,11 +45,14 @@ The contract between whatever holds the tools and the loop that offers them to a
 ### client
 
 - `contextLimitFor` — How much a model will read, in tokens.
+- `FIRST_TOKEN_FACTOR` — How many idle windows the first chunk gets when `firstTokenSeconds` is not given.
+- `firstTokenMs` — The wait for a streamed turn's first chunk, in the SDK's spelling: `undefined` is no limit.
 - `getClient` — The client for an endpoint, built once and kept.
 - `listModels` — Asks an endpoint what it serves, and remembers the answer.
 - `ModelInfo` (type) — A model an endpoint offers, and what it says the model will read.
 - `NO_KEY` — The SDK insists on a non-empty key even where the server will not look at it.
 - `resetClients` — Forgets every cached client and listing.
+- `servedWindow` — The window a local server is actually serving a model in, which its listing does not say.
 - `timeoutMs` — Zero, less, or absent means no limit, which the SDK spells as `undefined`.
 
 ### compaction
@@ -139,8 +142,11 @@ Everything about a request failing that is not about what the request said.
 - `ContextOverflow` — The request was bigger than the model will read.
 - `compact` — 1234 → "1.2k".
 - `EndpointSilent` — The endpoint stopped answering mid-request.
+- `isModelLoading` — Whether a failure is a local server still loading the model, rather than one failing to serve.
 - `isOverflow` — Whether a refusal means the request was too big, rather than merely refused.
 - `isTransient` — Whether a failed request is worth trying again.
+- `LOADING_POLL_MS` — How long to wait between asking a loading server again.
+- `LOADING_TIMEOUT_MS` — How long `runTurn` waits for a model to load unless told otherwise.
 - `messageTokens` — One message's estimated tokens, by the same count `requestTokens` sums for a whole request.
 - `requestTokens` — What this request will cost the window, in tokens, near enough.
 - `SMALLEST_LIKELY_WINDOW` — The smallest window worth believing in, and the floor under `runTurn`'s guard.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@cubicecho/agent-core",
-  "version": "2.6.0",
+  "version": "2.7.0",
   "description": "The endpoint-agnostic half of an OpenAI-compatible agent loop: tool-schema compatibility, on-demand tool loading, one-shot side tasks, run events, and a pooled client.",
   "keywords": [
     "openai",