@cubicecho/agent-core 4.8.0 → 4.9.0

package/README.md

@@ -867,10 +867,23 @@ again without it, with a notice naming it. A nested name is dropped at its top-l
 refused field nobody said was droppable is passed on, since dropping it would change the request
 behind the caller's back.
 
-On a llama.cpp server started with `--parallel`, pin each session to a slot with
-`extraBody: { id_slot: n }`. The slot keeps that session's KV cache warm, which is the difference
-between a cached prefill on every turn and a full one — but only while the prefix stays the same
-from turn to turn; see #63, and the caution on compaction below.
+On a llama.cpp server started with `--parallel`, give `runAgentLoop` a `session` — a run id, a
+conversation id — and the run is pinned to a slot of its own. `servedSlots` reads `total_slots` off
+`/props` (the request `servedWindow` already makes), `slotAllocator` hands each session a free
+slot, the one it last held where that is still free, and every request of the run carries
+`id_slot` and `cache_prompt: true` until the loop returns or throws. The slot keeps that session's
+KV cache warm against every other run on the server, which is the difference between a cached
+prefill on every turn and a full one — but only while the prefix stays the same from turn to turn;
+see #63, and the caution on compaction below.
+
+More sessions than slots run unpinned rather than wait, and so does a second loop under a key
+that holds a slot already. A notice says `pinned to slot 0 of 4`, or that no slot was free, and
+each turn of a session run on a server with slots reports `pinned`, which `runMetrics` counts as `unpinnedTurns`
+beside `cacheHitRatio`. A server that refuses `id_slot` is latched off it like any other
+`extraBody` field, and an `extraBody` that names `id_slot` itself is the host pinning by hand and
+is left alone. On a server without `total_slots` none of this happens. Whatever the server, the
+session also goes out as an `X-Session-Id` header (`SESSION_HEADER`), which llm-router's sticky
+affinity reads first, so a run behind a router keeps both its provider and its slot.
 
 Ollama's `options` object is not read on its OpenAI-compatible `/v1` route, so sampling set there
 does nothing; send the fields at the top level.
@@ -1379,6 +1392,13 @@ OpenRouter `:free` suffix, a typo) would otherwise fetch the listing on every ca
 same zero each time. Half a minute later it asks again, so a model pulled onto a box that has been
 up a week is still picked up without a restart.
 
+Which session holds which llama.cpp slot is kept as well, keyed on the base URL and the model but
+not the key: two keys reaching one server under two allocators would hand one slot to two runs,
+which is the collision the allocator is there to stop. It is bounded by the slots the server
+reported, and the slot count itself is asked for again every half minute, so a server restarted
+with another `--parallel` is not sent a slot it no longer has. `resetSlots` clears it, and
+`resetAll` with the rest.
+
 Both numbers are defaults, as is the ten seconds a window lookup is given
 (`lookupTimeoutMs`). `configureClients` moves them for the process, the way `configureEvents`
 and `configureHooks` do — a multi-tenant host raising the pool so tenants stop evicting each other,

package/dist/core/config.d.ts

@@ -89,7 +89,7 @@ export interface ModelParams {
      * What a model card asks for and nothing here names: `top_k`, `min_p`, `repeat_penalty` —
      * what actually stops a small model looping — and a server's own fields, such as llama.cpp's
      * `id_slot`, which pins a session to one slot of a `--parallel` server so its KV cache is
-     * still warm on the next turn. A local server ignores a field it does not know; OpenAI refuses
+     * still warm on the next turn — `runAgentLoop`'s `session` picks one itself. A local server ignores a field it does not know; OpenAI refuses
      * one by name, and `negotiate` drops that name for the model and sends the request again.
      * `model`, `messages`, `stream` and `tools` are the loop's and are not overridden from here.
      * Ollama's `options` object is not read on its `/v1` route, so sampling there has to go in as

package/dist/endpoint/client.d.ts

@@ -181,6 +181,22 @@ export interface LookupOptions {
 export declare function servedWindow(config: Endpoint & {
     model: string;
 }, { signal }?: LookupOptions): Promise<number>;
+/**
+ * How many slots a llama.cpp server runs a model in, for pinning a session to one of them.
+ *
+ * @param config - The endpoint, plus the model the requests will name.
+ * @param [options] - Cancellation. See `LookupOptions`.
+ * @returns `total_slots` from `/props`, or zero where the server has no such route or did not say.
+ * Rejects only with the reason of `options.signal`: a server that could not be reached answers zero.
+ *
+ * @remarks
+ * The same request `servedWindow` makes, so a run that has sized its window has the count without
+ * a second one. A count stands for `listingMissMs`, and a server without `/props` is not asked again
+ * until `resetClients`. Zero is what every server but llama.cpp answers, and leaves a run unpinned.
+ */
+export declare function servedSlots(config: Endpoint & {
+    model: string;
+}, { signal }?: LookupOptions): Promise<number>;
 /**
  * Asks an endpoint what it serves, and remembers the answer.
  *

package/dist/endpoint/client.js

@@ -322,6 +322,32 @@ export const endpointId = (config) => digestOf(endpointKey(config));
  * is asked again after `listingMissMs`, like a listing that did not name the model.
  */
 const served = scoped(() => new Map());
+/**
+ * Slot counts read off llama.cpp's `/props`, keyed on endpoint and model as `served` is.
+ *
+ * @remarks
+ * `gone` is a server without the route, which is not asked again until `resetClients`. Any other
+ * answer, a count included, stands for `listingMissMs`: a server restarted with another
+ * `--parallel` must not be sent a slot it no longer has for the life of the process.
+ */
+const slotCounts = scoped(() => new Map());
+/**
+ * Remembers what a `/props` answer said of its slots.
+ *
+ * @param key - The endpoint and model, by `modelKey`.
+ * @param props - How `/props` answered, as `probe` hands it over.
+ */
+function noteSlots(key, props) {
+    const { total_slots } = (props.body ?? {});
+    slotCounts().set(key, { slots: positive(total_slots), at: Date.now(), gone: props.missing });
+}
+/**
+ * The path `/props` is asked on, naming the model for a llama.cpp router serving several.
+ *
+ * @param model - The model's name as the endpoint knows it.
+ * @returns The path and its query. A single-model server ignores the query.
+ */
+const propsPath = (model) => `/props?model=${encodeURIComponent(model)}`;
 /** Endpoints with neither native route, which are not asked for either again. */
 const unserved = scoped(() => new Set());
 /**
@@ -452,8 +478,8 @@ async function readServed(config, within) {
     }
     let window = 0;
     try {
-        // Named, for a llama.cpp router serving several models; a single-model server ignores it.
-        const props = await probe(config, `/props?model=${encodeURIComponent(config.model)}`, within);
+        const props = await probe(config, propsPath(config.model), within);
+        noteSlots(key, props);
         const settings = props.body?.default_generation_settings;
         window = positive(settings?.n_ctx);
         if (!window) {
@@ -483,6 +509,49 @@ async function readServed(config, within) {
     served().set(key, { window, at: Date.now() });
     return window;
 }
+/**
+ * How many slots a llama.cpp server runs a model in, for pinning a session to one of them.
+ *
+ * @param config - The endpoint, plus the model the requests will name.
+ * @param [options] - Cancellation. See `LookupOptions`.
+ * @returns `total_slots` from `/props`, or zero where the server has no such route or did not say.
+ * Rejects only with the reason of `options.signal`: a server that could not be reached answers zero.
+ *
+ * @remarks
+ * The same request `servedWindow` makes, so a run that has sized its window has the count without
+ * a second one. A count stands for `listingMissMs`, and a server without `/props` is not asked again
+ * until `resetClients`. Zero is what every server but llama.cpp answers, and leaves a run unpinned.
+ */
+export async function servedSlots(config, { signal } = {}) {
+    const slots = await readSlots(config, lookupSignal(config, signal));
+    signal?.throwIfAborted();
+    return slots;
+}
+/**
+ * `servedSlots` under a lookup's signal.
+ *
+ * @param config - The endpoint, plus the model the requests will name.
+ * @param within - The lookup's signal. Fired, the answer is zero and nothing is remembered.
+ * @returns The slot count, or zero. Never rejects.
+ */
+async function readSlots(config, within) {
+    const endpoint = endpointKey(config);
+    if (unserved().has(endpoint)) {
+        return 0;
+    }
+    const key = modelKey(endpoint, config.model);
+    const known = slotCounts().get(key);
+    if (known && (known.gone || Date.now() - known.at < poolLimits().listingMissMs)) {
+        return known.slots;
+    }
+    try {
+        noteSlots(key, await probe(config, propsPath(config.model), within));
+    }
+    catch {
+        return 0;
+    }
+    return slotCounts().get(key)?.slots ?? 0;
+}
 /**
  * Asks an endpoint what it serves, and remembers the answer.
  *
@@ -618,6 +687,7 @@ export function resetClients() {
     misses().clear();
     trainedOnly().clear();
     served().clear();
+    slotCounts().clear();
     unserved().clear();
     refusals().clear();
 }

package/dist/endpoint/slots.d.ts

@@ -0,0 +1,54 @@
+/**
+ * Which slot of a multi-slot llama.cpp server each session is on, so concurrent runs stop
+ * evicting each other's prompt cache.
+ *
+ * `llama-server --parallel N` keeps one KV cache per slot, and a turn that lands on a slot holding
+ * another run's prefix is prefilled again from the top. The server's own choice is a guess by
+ * prompt similarity; this hands each session a slot of its own for as long as it holds it, and
+ * the same one again next time where it is still free.
+ */
+/** The request field that names a llama.cpp slot. */
+export declare const SLOT_FIELD: 'id_slot';
+/** The request field that asks llama.cpp to reuse the slot's cached prefix. On by default there. */
+export declare const CACHE_PROMPT_FIELD: 'cache_prompt';
+/** The header llm-router's sticky affinity reads first, which keeps a session on one provider. */
+export declare const SESSION_HEADER: 'X-Session-Id';
+/** Hands one endpoint's slots to sessions. See `slotAllocator`. */
+export interface SlotAllocator {
+    /** How many slots the endpoint said it has. */
+    readonly total: number;
+    /**
+     * Gives a session a slot for as long as it holds it.
+     *
+     * @param session - The run's own key: a run id, a conversation id.
+     * @returns The slot, from zero: the one this session last held where it is free, else a free
+     * one. `undefined` where every slot is held, or this session holds one already.
+     */
+    acquire(session: string): number | undefined;
+    /**
+     * Frees the slot a session holds. A session holding none is a no-op.
+     *
+     * @param session - The key it was acquired under.
+     */
+    release(session: string): void;
+}
+/**
+ * The slot allocator for one endpoint and model, sized to what its `/props` said.
+ *
+ * @param endpoint - The server, read for `baseUrl` alone.
+ * @param model - The model the requests name.
+ * @param total - The slots it has, as `servedSlots` reads them. Zero or less leaves none to hand out.
+ * @returns The live allocator: two calls for one endpoint and model share every slot. Slots past a
+ * smaller `total` than last time are dropped, and a holder of one releases into nothing.
+ *
+ * @remarks
+ * A free slot is chosen in this order: the one the session last held, then one nobody has held,
+ * then the one released longest ago, so a session that has just finished keeps its cache for its
+ * next step as long as a free slot can be found elsewhere. More sessions than slots get no pin and
+ * are not queued; llama.cpp places them by its own guess, which is what every run had before.
+ */
+export declare function slotAllocator(endpoint: {
+    baseUrl: string;
+}, model: string, total: number): SlotAllocator;
+/** Forgets every endpoint's slots, held ones included. For tests, and `resetAll`. */
+export declare function resetSlots(): void;

package/dist/endpoint/slots.js

@@ -0,0 +1,88 @@
+import { scoped } from '../core/scope.js';
+import { modelKey } from './client.js';
+/**
+ * Which slot of a multi-slot llama.cpp server each session is on, so concurrent runs stop
+ * evicting each other's prompt cache.
+ *
+ * `llama-server --parallel N` keeps one KV cache per slot, and a turn that lands on a slot holding
+ * another run's prefix is prefilled again from the top. The server's own choice is a guess by
+ * prompt similarity; this hands each session a slot of its own for as long as it holds it, and
+ * the same one again next time where it is still free.
+ */
+/** The request field that names a llama.cpp slot. */
+export const SLOT_FIELD = 'id_slot';
+/** The request field that asks llama.cpp to reuse the slot's cached prefix. On by default there. */
+export const CACHE_PROMPT_FIELD = 'cache_prompt';
+/** The header llm-router's sticky affinity reads first, which keeps a session on one provider. */
+export const SESSION_HEADER = 'X-Session-Id';
+/** Each endpoint and model's slots, and the counter releases are ordered by. */
+const pools = scoped(() => ({ slots: new Map(), clock: 0 }));
+/**
+ * The slots one endpoint has, as the allocator keeps them.
+ *
+ * @param endpoint - The server, read for `baseUrl` alone.
+ * @param model - The model, which a llama.cpp router serves from a process with slots of its own.
+ * @returns The key: the base URL less its trailing slashes, and the model.
+ *
+ * @remarks
+ * The API key is left out where every other per-endpoint cache keys on it. Two keys reaching
+ * different servers through one URL cost only a slot left unused; two keys reaching the same
+ * server under two allocators hand one slot to two runs, which is the collision this exists to stop.
+ */
+const poolKey = (endpoint, model) => modelKey(endpoint.baseUrl.trim().replace(/\/+$/, ''), model);
+/**
+ * The slot allocator for one endpoint and model, sized to what its `/props` said.
+ *
+ * @param endpoint - The server, read for `baseUrl` alone.
+ * @param model - The model the requests name.
+ * @param total - The slots it has, as `servedSlots` reads them. Zero or less leaves none to hand out.
+ * @returns The live allocator: two calls for one endpoint and model share every slot. Slots past a
+ * smaller `total` than last time are dropped, and a holder of one releases into nothing.
+ *
+ * @remarks
+ * A free slot is chosen in this order: the one the session last held, then one nobody has held,
+ * then the one released longest ago, so a session that has just finished keeps its cache for its
+ * next step as long as a free slot can be found elsewhere. More sessions than slots get no pin and
+ * are not queued; llama.cpp places them by its own guess, which is what every run had before.
+ */
+export function slotAllocator(endpoint, model, total) {
+    const key = poolKey(endpoint, model);
+    const pool = pools();
+    const size = Math.max(0, Math.floor(total));
+    const known = pool.slots.get(key) ?? [];
+    // Resized onto the same slot objects, so an allocator handed out before the resize still
+    // releases what it acquired.
+    const slots = known.length === size ? known : Array.from({ length: size }, (_, index) => known[index] ?? { released: 0 });
+    pool.slots.set(key, slots);
+    const acquire = (session) => {
+        const holdsOne = slots.some((slot) => slot.holder === session);
+        if (holdsOne) {
+            return undefined;
+        }
+        const free = slots.map((slot, index) => ({ slot, index })).filter(({ slot }) => slot.holder === undefined);
+        const own = free.find(({ slot }) => slot.last === session);
+        const unused = free.find(({ slot }) => slot.last === undefined);
+        const oldest = free.reduce((best, entry) => (best === undefined || entry.slot.released < best.slot.released ? entry : best), undefined);
+        const chosen = own ?? unused ?? oldest;
+        if (!chosen) {
+            return undefined;
+        }
+        chosen.slot.holder = session;
+        chosen.slot.last = session;
+        return chosen.index;
+    };
+    const release = (session) => {
+        const held = slots.find((slot) => slot.holder === session);
+        if (!held) {
+            return;
+        }
+        held.holder = undefined;
+        pool.clock++;
+        held.released = pool.clock;
+    };
+    return { total: size, acquire, release };
+}
+/** Forgets every endpoint's slots, held ones included. For tests, and `resetAll`. */
+export function resetSlots() {
+    pools.reset();
+}

package/dist/index.d.ts

@@ -14,8 +14,9 @@ export { estimateFrom, type LedgerEntry, type LedgerEstimateOptions, type Ledger
 export { type AgentConfig, type Endpoint, type ModelParams, type RetryPolicy, ToolDiscovery, type ToolPolicy, } from './core/config.ts';
 export { calibrate, charsPerTokenFor, resetCalibration } from './endpoint/calibration.ts';
 export { type Capabilities, capabilitiesFor, EFFORT_LADDER, EFFORT_NONE, EFFORT_OFF, effortFor, expireCapabilities, type ModelCapabilities, modelCapabilitiesFor, type NegotiateOptions, negotiate, resetCapabilities, } from './endpoint/capabilities.ts';
-export { type ClientPoolOptions, configureClients, contextLimitFor, endpointId, endpointKey, firstTokenMs, getClient, type LookupOptions, listModels, type ModelInfo, NO_KEY, resetClients, resolveApiKey, servedWindow, timeoutMs, } from './endpoint/client.ts';
+export { type ClientPoolOptions, configureClients, contextLimitFor, endpointId, endpointKey, firstTokenMs, getClient, type LookupOptions, listModels, type ModelInfo, NO_KEY, resetClients, resolveApiKey, servedSlots, servedWindow, timeoutMs, } from './endpoint/client.ts';
 export { isGrammarError, relaxSchema, relaxTools, sanitizeSchema, sanitizeTools, } from './endpoint/schema-compat.ts';
+export { CACHE_PROMPT_FIELD, resetSlots, SESSION_HEADER, SLOT_FIELD, type SlotAllocator, slotAllocator, } from './endpoint/slots.ts';
 export { HOOK_EVENTS, HookEvent, INJECT_EVENTS } from './hooks/hook-events.ts';
 export { assembleContext, configureHooks, consult, type Gathered, gather, HOOK_CONTEXT_TOKENS, HOOK_PREFACE, type HookContext, type HookMessage, type HookNote, type HookOptions, type HookOutcome, type HookRunner, notify, resetHooks, turnIndex, turnMessages, UNTRUSTED_PREFACE, untrusted, withContext, } from './hooks/hooks.ts';
 export { type AgentLoopHooks, type AgentLoopOptions, type AgentLoopRequest, type AgentLoopResult, PRESELECT_APPEND, PRESELECT_EXCLUSIVE, PRESELECTED_NOTICE, runAgentLoop, type StepWindow, } from './run/agent-loop.ts';

package/dist/index.js

@@ -14,8 +14,9 @@ export { estimateFrom, rebaseLedger, recordRequest, tokensBetween, } from './con
 export { ToolDiscovery, } from './core/config.js';
 export { calibrate, charsPerTokenFor, resetCalibration } from './endpoint/calibration.js';
 export { capabilitiesFor, EFFORT_LADDER, EFFORT_NONE, EFFORT_OFF, effortFor, expireCapabilities, modelCapabilitiesFor, negotiate, resetCapabilities, } from './endpoint/capabilities.js';
-export { configureClients, contextLimitFor, endpointId, endpointKey, firstTokenMs, getClient, listModels, NO_KEY, resetClients, resolveApiKey, servedWindow, timeoutMs, } from './endpoint/client.js';
+export { configureClients, contextLimitFor, endpointId, endpointKey, firstTokenMs, getClient, listModels, NO_KEY, resetClients, resolveApiKey, servedSlots, servedWindow, timeoutMs, } from './endpoint/client.js';
 export { isGrammarError, relaxSchema, relaxTools, sanitizeSchema, sanitizeTools, } from './endpoint/schema-compat.js';
+export { CACHE_PROMPT_FIELD, resetSlots, SESSION_HEADER, SLOT_FIELD, slotAllocator, } from './endpoint/slots.js';
 export { HOOK_EVENTS, HookEvent, INJECT_EVENTS } from './hooks/hook-events.js';
 export { assembleContext, configureHooks, consult, gather, HOOK_CONTEXT_TOKENS, HOOK_PREFACE, notify, resetHooks, turnIndex, turnMessages, UNTRUSTED_PREFACE, untrusted, withContext, } from './hooks/hooks.js';
 export { PRESELECT_APPEND, PRESELECT_EXCLUSIVE, PRESELECTED_NOTICE, runAgentLoop, } from './run/agent-loop.js';

package/dist/run/agent-loop.d.ts

@@ -333,6 +333,19 @@ export interface AgentLoopOptions {
     ledger?: TokenLedger;
     /** Stops the run: the request in flight, and between steps and calls. */
     signal?: AbortSignal;
+    /**
+     * The run's own key — a run id, a conversation id — which keeps its prompt cache warm across
+     * steps and beside other runs. Every request carries it as `SESSION_HEADER`, for a router's
+     * sticky affinity.
+     *
+     * @remarks
+     * On a llama.cpp server that reports `total_slots`, the run also holds one slot from its first
+     * request to its return or throw, sent as `id_slot` with `cache_prompt: true`, and a notice says
+     * which. The slot is the one this key held last where it is free; with every slot held, or one
+     * held by this key already, the run goes unpinned. An `extraBody` naming `id_slot` is the host
+     * pinning for itself and is left alone, and a server that refuses the field is latched off it.
+     */
+    session?: string;
     /** Told what the run is doing, as the events a watcher reads. */
     onEvent?: (event: RunEventInput) => void;
     /**

package/dist/run/agent-loop.js

@@ -4,7 +4,8 @@ import { ToolDiscovery } from '../core/config.js';
 import { counted } from '../core/guards.js';
 import { charsPerTokenFor } from '../endpoint/calibration.js';
 import { capabilitiesFor, modelCapabilitiesFor, } from '../endpoint/capabilities.js';
-import { firstTokenMs, getClient, loadingMs, timeoutMs } from '../endpoint/client.js';
+import { firstTokenMs, getClient, loadingMs, servedSlots, timeoutMs } from '../endpoint/client.js';
+import { CACHE_PROMPT_FIELD, SESSION_HEADER, SLOT_FIELD, slotAllocator } from '../endpoint/slots.js';
 import { HookEvent } from '../hooks/hook-events.js';
 import { configureHooks, gather, notify, turnIndex, turnMessages, withContext, } from '../hooks/hooks.js';
 import { serverInstructionsPrompt } from '../tools/instructions.js';
@@ -111,6 +112,7 @@ function cacheDiagnosis(previous, next, usage) {
  * run's `temperature` or `maxTokens`.
  */
 export async function runAgentLoop(options) {
+    let pin;
     // Until the steps have set themselves up there is only what the caller handed over.
     const standing = {
         read: () => ({
@@ -125,7 +127,8 @@ export async function runAgentLoop(options) {
     };
     let result;
     try {
-        result = await runSteps(options, standing);
+        pin = await pinSession(options);
+        result = await runSteps(options, standing, pin);
     }
     catch (error) {
         // Wrapped rather than annotated: an abort's reason is one object shared by every run under
@@ -135,11 +138,56 @@ export async function runAgentLoop(options) {
             ? new AgentLoopOverflow(error.message, standing.read(), { cause: error })
             : new AgentLoopError(errorMessage(error), standing.read(), { cause: error });
     }
+    finally {
+        pin?.release();
+    }
     if (result) {
         return result;
     }
     throw new ToolIterationLimit(`Stopped after ${options.config.maxToolIterations} tool iterations.`, standing.read());
 }
+/**
+ * Pins a run to a llama.cpp slot where it has a session and the server has slots to hand out.
+ *
+ * @param options - The run's options. Read for `session`, the config and the signal; a notice goes
+ * to `onEvent`.
+ * @returns The pin, unpinned on a server without slots, one that refused `id_slot` or whose slots
+ * are all held. `undefined` without a session. Rejects only with the reason of `signal`.
+ */
+async function pinSession(options) {
+    const { config, session, signal } = options;
+    if (!session) {
+        return undefined;
+    }
+    const headers = { [SESSION_HEADER]: session };
+    const unpinned = { fields: {}, headers, slotted: false, release: () => { } };
+    const pinsItself = config.extraBody !== undefined && SLOT_FIELD in config.extraBody;
+    if (pinsItself) {
+        return unpinned;
+    }
+    const total = await servedSlots(config, { signal });
+    if (total === 0) {
+        return unpinned;
+    }
+    const notice = (text) => options.onEvent?.({ kind: RunEventKind.Notice, text });
+    const refused = modelCapabilitiesFor(capabilitiesFor(config.baseUrl, config.apiKey), config.model);
+    if (refused.refusedFields.has(SLOT_FIELD)) {
+        return { ...unpinned, slotted: true };
+    }
+    const slots = slotAllocator(config, config.model, total);
+    const slot = slots.acquire(session);
+    if (slot === undefined) {
+        notice(`no free slot of ${total} for this session; running unpinned`);
+        return { ...unpinned, slotted: true };
+    }
+    notice(`pinned to slot ${slot} of ${total}`);
+    return {
+        fields: { [SLOT_FIELD]: slot, [CACHE_PROMPT_FIELD]: true },
+        headers,
+        slotted: true,
+        release: () => slots.release(session),
+    };
+}
 /**
  * A message's words, as the preselector is asked about them: its string, or its text parts.
  *
@@ -422,10 +470,11 @@ function turnCalls(turn, recoverable, notice) {
  * @param options - What `runAgentLoop` was given.
  * @param standing - Written to: once the steps have set themselves up, its `read` is theirs, so the
  * caller can read the run as it stands whether they return or throw.
+ * @param [pin] - What the run's session adds to each request. Absent adds nothing.
  * @returns The result of the turn that asked for no tools, or `undefined` when `maxToolIterations`
  * is spent.
  */
-async function runSteps(options, standing) {
+async function runSteps(options, standing, pin) {
     const { config, system = '', tools = [], catalog = [], dispatch, hooks, signal } = options;
     const { onRequest, onTurn, onMessage, onToolCall, onToolResult, beforeStep, parallel = false, recoverToolCalls: recover = true, maxContinuations = 0, toolOrder = true, dedupeToolCalls = true, preselectRouting = PRESELECT_EXCLUSIVE, serverInstructions = false, } = options;
     const dedupable = typeof dedupeToolCalls === 'function' ? dedupeToolCalls : () => dedupeToolCalls;
@@ -435,6 +484,9 @@ async function runSteps(options, standing) {
     const supports = capabilitiesFor(config.baseUrl, config.apiKey);
     const maxRetries = Math.max(0, Number(config.maxRetries) || 0);
     const notice = (text) => onEvent({ kind: RunEventKind.Notice, text });
+    // The host's own fields win over the pin's, so an explicit `cache_prompt: false` stays false.
+    const extraBody = pin ? { ...pin.fields, ...config.extraBody } : config.extraBody;
+    const sending = { ...config, extraBody };
     const { proxied, onDemand, loaded, preselected } = await discovery(options, notice);
     const instructions = serverInstructions === false
         ? ''
@@ -577,10 +629,11 @@ async function runSteps(options, standing) {
             ...(prompt ? [{ role: Role.System, content: prompt }] : []),
             ...withContext(messages, questionAt(), gathered.context, preface),
         ];
-        const build = (supported, refused) => buildBody(config, supported, refused, request, declared, toolOrder);
+        const build = (supported, refused) => buildBody(sending, supported, refused, request, declared, toolOrder);
         const turnOptions = {
             model: config.model,
-            droppable: Object.keys(config.extraBody ?? {}),
+            droppable: Object.keys(extraBody ?? {}),
+            ...(pin ? { headers: pin.headers } : {}),
             maxRetries,
             loadingTimeoutMs: loadingMs(config),
             contextLimit: config.contextLength ?? 0,
@@ -604,7 +657,13 @@ async function runSteps(options, standing) {
         // The breakdown measures the array `toolSchemaTokens` does, so where no prompt was reported
         // the two give one number for the tool block rather than two.
         const charsPerToken = charsPerTokenFor(supports, config.model);
-        Object.assign(first.usage, cacheDiagnosis(previous, { messages: request, tools: names }, first.usage), {
+        // Read after the turn, so a refusal answered by dropping `id_slot` counts this turn unpinned.
+        const pinned = pin?.slotted
+            ? {
+                pinned: SLOT_FIELD in pin.fields && !modelCapabilitiesFor(supports, config.model).refusedFields.has(SLOT_FIELD),
+            }
+            : {};
+        Object.assign(first.usage, cacheDiagnosis(previous, { messages: request, tools: names }, first.usage), pinned, {
             toolsDeclared: declared.length,
             toolSchemaTokens: Math.ceil(toolsChars(declared) / charsPerToken),
             context: contextTokens({ model: config.model, stream: true, messages: request, tools: declared }, { charsPerToken, promptTokens: first.usage.prompt }),

package/dist/run/events.d.ts

@@ -337,6 +337,11 @@ export interface RunMetrics {
     cacheBreakReasons: Partial<Record<CacheBreakReason, number>>;
     /** Turns cut off at `maxTokens` after any continuation. */
     truncatedTurns: number;
+    /**
+     * Turns of a `session` run served on no llama.cpp slot of their own, which `cacheHitRatio` reads
+     * against. Absent where no turn reported `pinned`.
+     */
+    unpinnedTurns?: number;
     /** From the first event to the last, which a still-running run keeps moving. */
     wallMs?: number;
     /** Prefill time summed over the turns. */

package/dist/run/events.js

@@ -576,6 +576,9 @@ export function runMetrics(events, { contextLength } = {}) {
             if (turn.finishReason === FinishReason.Length) {
                 metrics.truncatedTurns++;
             }
+            if (turn.pinned !== undefined) {
+                add('unpinnedTurns', turn.pinned ? 0 : 1);
+            }
             if (turn.wallMs !== undefined) {
                 metrics.slowestTurnMs = Math.max(metrics.slowestTurnMs ?? 0, turn.wallMs);
             }

package/dist/runtime/reset.d.ts

@@ -2,13 +2,14 @@
  * Forgets everything this package remembers between calls.
  *
  * @remarks
- * Six modules here keep state for the life of the runtime — the process's, unless this is called
+ * Seven modules here keep state for the life of the runtime — the process's, unless this is called
  * as a method of one `createRuntime` made, which clears that one alone — each for a good reason
  * and each with its own seam: the pooled clients and their model listings, the endpoints that turned
  * out not to take `stream_options` or a grammar, each model's measured characters per token,
- * the models that refused the no-thinking hints, the event bus, and the hooks' configured
- * budget and preface. `resetClients`, `resetCapabilities`, `resetCalibration`, `resetHints`,
- * `resetEvents` and `resetHooks` stay exported, because a test that means to clear one thing
+ * the models that refused the no-thinking hints, the event bus, the hooks' configured
+ * budget and preface, and which session holds which llama.cpp slot. `resetClients`,
+ * `resetCapabilities`, `resetCalibration`, `resetHints`, `resetEvents`, `resetHooks` and
+ * `resetSlots` stay exported, because a test that means to clear one thing
  * should say so.
  *
  * This is for the other case, which is every teardown. What they hold is *latched
@@ -18,7 +19,7 @@
  * one that reads the latch fails only when it happens to run second. `tests/side-task-hints.test.ts`
  * was written that way and only passed because every case had been handed a hostname of its own.
  *
- * It is also the seam that does not need finding again. A seventh module with a cache is a seventh
+ * It is also the seam that does not need finding again. An eighth module with a cache is an eighth
  * line here, rather than an edit to the teardown of three consumers who will not all notice.
  *
  * A test that builds a `createRuntime` per case has nothing to tear down, and is the better

package/dist/runtime/reset.js

@@ -1,6 +1,7 @@
 import { resetCalibration } from '../endpoint/calibration.js';
 import { resetCapabilities } from '../endpoint/capabilities.js';
 import { resetClients } from '../endpoint/client.js';
+import { resetSlots } from '../endpoint/slots.js';
 import { resetHooks } from '../hooks/hooks.js';
 import { resetEvents } from '../run/events.js';
 import { resetHints } from '../turn/side-task.js';
@@ -8,13 +9,14 @@ import { resetHints } from '../turn/side-task.js';
  * Forgets everything this package remembers between calls.
  *
  * @remarks
- * Six modules here keep state for the life of the runtime — the process's, unless this is called
+ * Seven modules here keep state for the life of the runtime — the process's, unless this is called
  * as a method of one `createRuntime` made, which clears that one alone — each for a good reason
  * and each with its own seam: the pooled clients and their model listings, the endpoints that turned
  * out not to take `stream_options` or a grammar, each model's measured characters per token,
- * the models that refused the no-thinking hints, the event bus, and the hooks' configured
- * budget and preface. `resetClients`, `resetCapabilities`, `resetCalibration`, `resetHints`,
- * `resetEvents` and `resetHooks` stay exported, because a test that means to clear one thing
+ * the models that refused the no-thinking hints, the event bus, the hooks' configured
+ * budget and preface, and which session holds which llama.cpp slot. `resetClients`,
+ * `resetCapabilities`, `resetCalibration`, `resetHints`, `resetEvents`, `resetHooks` and
+ * `resetSlots` stay exported, because a test that means to clear one thing
  * should say so.
  *
  * This is for the other case, which is every teardown. What they hold is *latched
@@ -24,7 +26,7 @@ import { resetHints } from '../turn/side-task.js';
  * one that reads the latch fails only when it happens to run second. `tests/side-task-hints.test.ts`
  * was written that way and only passed because every case had been handed a hostname of its own.
  *
- * It is also the seam that does not need finding again. A seventh module with a cache is a seventh
+ * It is also the seam that does not need finding again. An eighth module with a cache is an eighth
  * line here, rather than an edit to the teardown of three consumers who will not all notice.
  *
  * A test that builds a `createRuntime` per case has nothing to tear down, and is the better
@@ -37,4 +39,5 @@ export function resetAll() {
     resetHints();
     resetEvents();
     resetHooks();
+    resetSlots();
 }

package/dist/runtime/runtime.d.ts

@@ -1,7 +1,8 @@
 import { compactTranscript, runCompaction, summariser } from '../context/compaction.ts';
 import { calibrate, charsPerTokenFor, resetCalibration } from '../endpoint/calibration.ts';
 import { capabilitiesFor, expireCapabilities, resetCapabilities } from '../endpoint/capabilities.ts';
-import { type ClientPoolOptions, configureClients, contextLimitFor, getClient, listModels, resetClients, servedWindow } from '../endpoint/client.ts';
+import { type ClientPoolOptions, configureClients, contextLimitFor, getClient, listModels, resetClients, servedSlots, servedWindow } from '../endpoint/client.ts';
+import { resetSlots, slotAllocator } from '../endpoint/slots.ts';
 import { assembleContext, configureHooks, consult, gather, type HookOptions, notify, resetHooks, withContext } from '../hooks/hooks.ts';
 import { runAgentLoop } from '../run/agent-loop.ts';
 import { configureEvents, type EventBusOptions, emit, endRun, history, resetEvents, watch } from '../run/events.ts';
@@ -31,6 +32,8 @@ declare const STATEFUL: {
     listModels: typeof listModels;
     contextLimitFor: typeof contextLimitFor;
     servedWindow: typeof servedWindow;
+    servedSlots: typeof servedSlots;
+    slotAllocator: typeof slotAllocator;
     configureClients: typeof configureClients;
     capabilitiesFor: typeof capabilitiesFor;
     expireCapabilities: typeof expireCapabilities;
@@ -55,6 +58,7 @@ declare const STATEFUL: {
     resetHints: typeof resetHints;
     resetEvents: typeof resetEvents;
     resetHooks: typeof resetHooks;
+    resetSlots: typeof resetSlots;
     resetAll: typeof resetAll;
 };
 /** What a runtime starts out configured with. Every field optional; see `createRuntime`. */

package/dist/runtime/runtime.js

@@ -2,7 +2,8 @@ import { compactTranscript, runCompaction, summariser } from '../context/compact
 import { inScope, rootScope } from '../core/scope.js';
 import { calibrate, charsPerTokenFor, resetCalibration } from '../endpoint/calibration.js';
 import { capabilitiesFor, expireCapabilities, resetCapabilities } from '../endpoint/capabilities.js';
-import { configureClients, contextLimitFor, getClient, listModels, resetClients, servedWindow, } from '../endpoint/client.js';
+import { configureClients, contextLimitFor, getClient, listModels, resetClients, servedSlots, servedWindow, } from '../endpoint/client.js';
+import { resetSlots, slotAllocator } from '../endpoint/slots.js';
 import { assembleContext, configureHooks, consult, gather, notify, resetHooks, withContext, } from '../hooks/hooks.js';
 import { runAgentLoop } from '../run/agent-loop.js';
 import { configureEvents, emit, endRun, history, resetEvents, watch } from '../run/events.js';
@@ -32,6 +33,8 @@ const STATEFUL = {
     listModels,
     contextLimitFor,
     servedWindow,
+    servedSlots,
+    slotAllocator,
     configureClients,
     capabilitiesFor,
     expireCapabilities,
@@ -56,6 +59,7 @@ const STATEFUL = {
     resetHints,
     resetEvents,
     resetHooks,
+    resetSlots,
     resetAll,
 };
 /**

package/dist/wire/stream.d.ts

@@ -112,6 +112,12 @@ export interface TurnUsage {
     cacheBreakReason?: CacheBreakReason;
     /** How many tools the request declared, filled by `runAgentLoop`. */
     toolsDeclared?: number;
+    /**
+     * The turn was served on the llama.cpp slot its run holds, filled by `runAgentLoop`. Present only
+     * for a run given a `session` on a server that reports slots: `false` is one that could not be
+     * pinned, every slot being held or the server refusing `id_slot`.
+     */
+    pinned?: boolean;
     /**
      * The declared tool block's estimated tokens. A chat template renders it ahead of the system
      * prompt, so this is what every change to the tool list costs the cache.
@@ -250,6 +256,11 @@ export interface StreamTurnOptions {
     onThinking?: (delta: string) => void;
     /** The model's answer, as it arrives. */
     onOutput?: (delta: string) => void;
+    /**
+     * Headers sent with the request, beside the client's own — `SESSION_HEADER`, for a router that
+     * keeps a session on one provider. Never refused, so never negotiated.
+     */
+    headers?: Record<string, string>;
 }
 /**
  * Runs one turn as a stream, reporting tokens as they arrive and assembling them back into a
@@ -271,4 +282,4 @@ export interface StreamTurnOptions {
  * which run it is: `step` is the caller's flow concept, and wrapping these into an `emit` is one
  * line at the call site.
  */
-export declare function streamTurn(client: OpenAI, body: OpenAI.ChatCompletionCreateParamsStreaming, { signal, idleMs, firstChunkMs, produced, fences, startInReasoning, onThinking, onOutput, }?: StreamTurnOptions): Promise<Turn>;
+export declare function streamTurn(client: OpenAI, body: OpenAI.ChatCompletionCreateParamsStreaming, { signal, idleMs, firstChunkMs, produced, fences, startInReasoning, onThinking, onOutput, headers, }?: StreamTurnOptions): Promise<Turn>;

package/dist/wire/stream.js

@@ -208,7 +208,7 @@ const NO_SDK_TIMEOUT = 2_147_483_647;
  * which run it is: `step` is the caller's flow concept, and wrapping these into an `emit` is one
  * line at the call site.
  */
-export async function streamTurn(client, body, { signal, idleMs, firstChunkMs, produced, fences = DEFAULT_FENCES, startInReasoning, onThinking, onOutput, } = {}) {
+export async function streamTurn(client, body, { signal, idleMs, firstChunkMs, produced, fences = DEFAULT_FENCES, startInReasoning, onThinking, onOutput, headers, } = {}) {
     // Silence, not duration: the timer is rearmed on every chunk, so a model that is still
     // talking is never cut off however long it takes, and one that has stopped talking does not
     // hang the run until someone notices. A request that never answers at all is the same case
@@ -257,6 +257,7 @@ export async function streamTurn(client, body, { signal, idleMs, firstChunkMs, p
         const stream = await client.chat.completions.create(body, {
             signal: linked,
             ...(watched ? { timeout: NO_SDK_TIMEOUT } : {}),
+            ...(headers ? { headers } : {}),
         });
         // The field's reasoning here, the fenced kind in the splitter, which can still move text
         // already read as answer into reasoning when a closing tag turns up with no opening one.

package/llms.txt

@@ -94,6 +94,7 @@ What an endpoint turned out not to support, and answering it when it says so.
 - `NO_KEY` — The SDK insists on a non-empty key even where the server will not look at it.
 - `resetClients` — Forgets every cached client and listing, and puts `configureClients` back to the defaults.
 - `resolveApiKey` — The key to send, where an endpoint may inherit one from the settings it overrides.
+- `servedSlots` — How many slots a llama.cpp server runs a model in, for pinning a session to one of them.
 - `servedWindow` — The window a local server's own API says it is serving a model in, for the servers whose listing does not say.
 - `timeoutMs` — An endpoint's request timeout in the SDK's spelling: milliseconds, and `undefined` for no limit.
 
@@ -107,6 +108,17 @@ JSON Schema compatibility for llama.cpp-backed servers.
 - `sanitizeSchema` — One JSON Schema as a strict server will accept it, for a schema that is not a tool's parameters.
 - `sanitizeTools` — Tool definitions a strict server will accept, remembered per definition object.
 
+### slots
+
+Which slot of a multi-slot llama.cpp server each session is on, so concurrent runs stop evicting each other's prompt cache.
+
+- `CACHE_PROMPT_FIELD` — The request field that asks llama.cpp to reuse the slot's cached prefix.
+- `resetSlots` — Forgets every endpoint's slots, held ones included.
+- `SESSION_HEADER` — The header llm-router's sticky affinity reads first, which keeps a session on one provider.
+- `SLOT_FIELD` — The request field that names a llama.cpp slot.
+- `SlotAllocator` (type) — Hands one endpoint's slots to sessions.
+- `slotAllocator` — The slot allocator for one endpoint and model, sized to what its `/props` said.
+
 ### hook-events
 
 The points in a session a hook can be bound to, apart from everything that runs one.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@cubicecho/agent-core",
-  "version": "4.8.0",
+  "version": "4.9.0",
   "description": "The endpoint-agnostic half of an OpenAI-compatible agent loop: tool-schema compatibility, on-demand tool loading, one-shot side tasks, run events, and a pooled client.",
   "keywords": [
     "openai",