@cubicecho/agent-core 2.9.0 → 2.10.0

package/README.md

@@ -585,6 +585,25 @@ OpenRouter `:free` suffix, a typo) would otherwise fetch the listing on every ca
 same zero each time. Half a minute later it asks again, so a model pulled onto a box that has been
 up a week is still picked up without a restart.
 
+Both numbers are defaults. `configureClients` moves them for the process, the way `configureEvents`
+and `configureHooks` do — a multi-tenant host raising the pool so tenants stop evicting each other,
+a dev box shortening the miss window so a model it has just pulled shows up sooner:
+
+```ts
+import { configureClients } from "@cubicecho/agent-core";
+
+configureClients({ maxClients: 256, listingMissMs: 2_000 });
+```
+
+A field left out, or given anything that is not a number above zero, keeps what it has, and the
+call returns everything in force. Lowering `maxClients` below the pool's current size evicts down to
+it at once, least recently used first. `resetClients` (and `resetAll`) puts the defaults back.
+
+A host that keeps per-endpoint state of its own should key it on `endpointKey` — the JSON of the base
+URL and the key, an absent key read as `NO_KEY` — rather than rebuilding that string, so the two
+cannot drift. `endpointKey` holds the key in the clear; `endpointId`, its SHA-256 digest, is the one
+that is safe to write down.
+
 `resetAll` drops all four, and `reset.ts` names each seam separately for a test that wants one.
 
 The latches can outlive the process as well, because otherwise every restart spends one refused

package/dist/client.d.ts

@@ -12,6 +12,36 @@ export declare const NO_KEY = "agent-core";
  * @param config Read for `requestTimeoutSeconds` alone.
  */
 export declare const timeoutMs: (config: Pick<Endpoint, "requestTimeoutSeconds">) => number | undefined;
+/** What the client pool is held to across a process. Every field optional; see `configureClients`. */
+export interface ClientPoolOptions {
+    /** How many endpoints' clients are kept at once. The least recently asked for goes first. */
+    maxClients?: number;
+    /**
+     * How long, in milliseconds, a model an endpoint did not list — or a server that answered
+     * without a served window — is taken at its word before the endpoint is asked again.
+     */
+    listingMissMs?: number;
+}
+/**
+ * Changes what the client pool is held to, for a process whose endpoints are not shaped like the
+ * deployments these defaults were chosen for.
+ *
+ * Module-level for the same reason `configureEvents` is: the pool is one module-level thing, and
+ * its size is a deployment's setting, said once at startup. A multi-tenant host keying on a key
+ * per user raises `maxClients` so its tenants stop evicting each other's connection pools; a dev
+ * box shortens `listingMissMs` so a model it has just pulled is seen sooner.
+ *
+ * A `maxClients` below the pool's current size evicts down to it at once, least recently asked
+ * for first, as the next `getClient` would have. A shorter `listingMissMs` applies to misses
+ * already remembered, since each is a timestamp compared against it when read.
+ *
+ * @param options The bounds to change. A field left out — or given anything that is not a number
+ * above zero — keeps what it has, so a half-built config narrows nothing. `Infinity` is a number
+ * above zero: as `maxClients` it lifts the bound, and as `listingMissMs` a miss is never asked
+ * about again until `resetClients`.
+ * @returns Everything in force afterwards, including what this call did not change.
+ */
+export declare function configureClients(options?: ClientPoolOptions): Required<ClientPoolOptions>;
 /** How many idle windows the first chunk gets when `firstTokenSeconds` is not given. */
 export declare const FIRST_TOKEN_FACTOR = 5;
 /**
@@ -29,7 +59,7 @@ export declare const firstTokenMs: (config: Pick<Endpoint, "requestTimeoutSecond
  * The pool is per *deployment*, not per request: what belongs in this map is an endpoint an
  * operator configured, and everything cached in this package is bounded on that reading. A
  * consumer that mints an API key per user still gets a working client, but it is churning
- * connection pools rather than sharing them and holding every one of them — `MAX_CLIENTS` keeps
+ * connection pools rather than sharing them and holding every one of them — `maxClients` keeps
  * that from being unbounded, and it is the point at which a client of your own, built and held
  * per tenant, is the better answer than this.
  *
@@ -43,7 +73,7 @@ export interface ModelInfo {
 }
 /**
  * What counts as one endpoint, everywhere in this package that has to remember something about
- * one — this file's listings, `capabilities`, and the no-thinking hints in `side-task`.
+ * one — the model listings, `capabilities`, and the no-thinking hints in `side-task`.
  *
  * The URL and the key together, because the key is part of what is on the other end rather than
  * only how it is paid for: a router is free to send two keys to two different backends, and then
@@ -51,7 +81,9 @@ export interface ModelInfo {
  * endpoint with no key and one that passes `undefined` are one entry rather than two.
  *
  * Stringified rather than joined on a separator, for the reason `getClient` gives: no character
- * is impossible in a URL or a key, and two endpoints must never collide on one entry.
+ * is impossible in a URL or a key, and two endpoints must never collide on one entry. A host
+ * keeping its own per-endpoint state keys it on this rather than on a copy of it, so the two
+ * cannot drift apart. It holds the key in the clear; `endpointId` is the one to write down.
  *
  * @param config Read for `baseUrl` and `apiKey` alone, and the key is optional here where
  * `Endpoint` requires it — `capabilitiesFor` is handed a URL and maybe a key rather than a whole
@@ -67,7 +99,8 @@ export declare const endpointKey: (config: {
  *
  * What an endpoint refused is exported by `exportCapabilities` to be written into a settings row
  * or a file, and a key inside that blob is a credential copied somewhere nobody meant to keep one.
- * A digest identifies the same endpoint on the next boot without saying what the key was.
+ * A digest identifies the same endpoint on the next boot without saying what the key was, and it
+ * is how a host finds its own endpoint's entry in a `CapabilitySnapshot`.
  *
  * @param config Read for `baseUrl` and `apiKey` alone, as `endpointKey` reads it.
  */
@@ -118,5 +151,8 @@ export declare function listModels(config: Endpoint): Promise<ModelInfo[]>;
 export declare function contextLimitFor(config: Endpoint & {
     model: string;
 }, declared?: number): Promise<number>;
-/** Forgets every cached client and listing. For tests, and for a settings change under test. */
+/**
+ * Forgets every cached client and listing, and puts `configureClients` back to the defaults. For
+ * tests, and for a settings change under test.
+ */
 export declare function resetClients(): void;

package/dist/client.js

@@ -33,7 +33,7 @@ export const timeoutMs = (config) => {
  */
 const clients = new Map();
 /**
- * How many endpoints' clients are kept at once.
+ * How many endpoints' clients are kept at once, until `configureClients` moves it.
  *
  * This is a backstop rather than a design. The key includes the API key, and every argument in
  * this file for what bounds these caches is "a settings row's worth" — true of the deployments
@@ -47,6 +47,56 @@ const clients = new Map();
  * constraint.
  */
 const MAX_CLIENTS = 32;
+/** How long a model an endpoint did not list stays unlisted before it is asked about again. */
+const LISTING_MISS_MS = 30_000;
+/** The numbers this module was written with. */
+const CLIENT_DEFAULTS = {
+    maxClients: MAX_CLIENTS,
+    listingMissMs: LISTING_MISS_MS,
+};
+/** What is in force now. Read where it is used, so a change applies from the next call. */
+let poolLimits = { ...CLIENT_DEFAULTS };
+/**
+ * Drops the least recently asked-for clients until the pool fits. Nothing is closed on the way
+ * out: the SDK holds no handle a caller can release, and an evicted client is garbage once
+ * whatever request is still in flight on it has finished — dropping the reference is the whole
+ * of the eviction.
+ */
+const evict = () => {
+    for (const oldest of clients.keys()) {
+        if (clients.size <= poolLimits.maxClients)
+            break;
+        clients.delete(oldest);
+    }
+};
+/**
+ * Changes what the client pool is held to, for a process whose endpoints are not shaped like the
+ * deployments these defaults were chosen for.
+ *
+ * Module-level for the same reason `configureEvents` is: the pool is one module-level thing, and
+ * its size is a deployment's setting, said once at startup. A multi-tenant host keying on a key
+ * per user raises `maxClients` so its tenants stop evicting each other's connection pools; a dev
+ * box shortens `listingMissMs` so a model it has just pulled is seen sooner.
+ *
+ * A `maxClients` below the pool's current size evicts down to it at once, least recently asked
+ * for first, as the next `getClient` would have. A shorter `listingMissMs` applies to misses
+ * already remembered, since each is a timestamp compared against it when read.
+ *
+ * @param options The bounds to change. A field left out — or given anything that is not a number
+ * above zero — keeps what it has, so a half-built config narrows nothing. `Infinity` is a number
+ * above zero: as `maxClients` it lifts the bound, and as `listingMissMs` a miss is never asked
+ * about again until `resetClients`.
+ * @returns Everything in force afterwards, including what this call did not change.
+ */
+export function configureClients(options = {}) {
+    for (const [name, value] of Object.entries(options)) {
+        if (typeof value === "number" && value > 0) {
+            poolLimits[name] = value;
+        }
+    }
+    evict();
+    return { ...poolLimits };
+}
 /** How many idle windows the first chunk gets when `firstTokenSeconds` is not given. */
 export const FIRST_TOKEN_FACTOR = 5;
 /**
@@ -70,7 +120,7 @@ export const firstTokenMs = (config) => {
  * The pool is per *deployment*, not per request: what belongs in this map is an endpoint an
  * operator configured, and everything cached in this package is bounded on that reading. A
  * consumer that mints an API key per user still gets a working client, but it is churning
- * connection pools rather than sharing them and holding every one of them — `MAX_CLIENTS` keeps
+ * connection pools rather than sharing them and holding every one of them — `maxClients` keeps
  * that from being unbounded, and it is the point at which a client of your own, built and held
  * per tenant, is the better answer than this.
  *
@@ -92,14 +142,7 @@ export function getClient(config) {
     }
     const client = new OpenAI({ baseURL: config.baseUrl, apiKey, timeout, maxRetries: 0 });
     clients.set(key, client);
-    // Nothing is closed on the way out. The SDK holds no handle a caller can release, and an
-    // evicted client is garbage once whatever request is still in flight on it has finished —
-    // dropping the reference is the whole of the eviction.
-    for (const oldest of clients.keys()) {
-        if (clients.size <= MAX_CLIENTS)
-            break;
-        clients.delete(oldest);
-    }
+    evict();
     return client;
 }
 /**
@@ -156,17 +199,15 @@ const listings = new Map();
  * match the configured name, an OpenRouter `:free` suffix, a typo in a settings row — and there
  * every call fetches the listing, re-reads it, finds the same absence and answers the same zero.
  *
- * Remembering the miss for a moment answers both: within `LISTING_MISS_MS` nobody is asked, and
+ * Remembering the miss for a moment answers both: within `listingMissMs` nobody is asked, and
  * after it the endpoint is asked again, so an `ollama pull` on a box that has been up a week is
  * picked up within the minute instead of at the next restart. Bounded by the (endpoint, model)
  * pairs actually asked about, which is the bound the listings themselves have.
  */
 const misses = new Map();
-/** How long a model an endpoint did not list stays unlisted before it is asked about again. */
-const LISTING_MISS_MS = 30_000;
 /**
  * What counts as one endpoint, everywhere in this package that has to remember something about
- * one — this file's listings, `capabilities`, and the no-thinking hints in `side-task`.
+ * one — the model listings, `capabilities`, and the no-thinking hints in `side-task`.
  *
  * The URL and the key together, because the key is part of what is on the other end rather than
  * only how it is paid for: a router is free to send two keys to two different backends, and then
@@ -174,7 +215,9 @@ const LISTING_MISS_MS = 30_000;
  * endpoint with no key and one that passes `undefined` are one entry rather than two.
  *
  * Stringified rather than joined on a separator, for the reason `getClient` gives: no character
- * is impossible in a URL or a key, and two endpoints must never collide on one entry.
+ * is impossible in a URL or a key, and two endpoints must never collide on one entry. A host
+ * keeping its own per-endpoint state keys it on this rather than on a copy of it, so the two
+ * cannot drift apart. It holds the key in the clear; `endpointId` is the one to write down.
  *
  * @param config Read for `baseUrl` and `apiKey` alone, and the key is optional here where
  * `Endpoint` requires it — `capabilitiesFor` is handed a URL and maybe a key rather than a whole
@@ -187,7 +230,8 @@ export const endpointKey = (config) => JSON.stringify([config.baseUrl, config.ap
  *
  * What an endpoint refused is exported by `exportCapabilities` to be written into a settings row
  * or a file, and a key inside that blob is a credential copied somewhere nobody meant to keep one.
- * A digest identifies the same endpoint on the next boot without saying what the key was.
+ * A digest identifies the same endpoint on the next boot without saying what the key was, and it
+ * is how a host finds its own endpoint's entry in a `CapabilitySnapshot`.
  *
  * @param config Read for `baseUrl` and `apiKey` alone, as `endpointKey` reads it.
  */
@@ -196,7 +240,7 @@ export const endpointId = (config) => createHash("sha256").update(endpointKey(co
  * Served windows found by asking a server's own API, keyed on endpoint and model together.
  *
  * A model's entry stays until `resetClients`, like a listing; a server that answered without one
- * is asked again after `LISTING_MISS_MS`, like a listing that did not name the model.
+ * is asked again after `listingMissMs`, like a listing that did not name the model.
  */
 const served = new Map();
 /** Endpoints that answered both probes with a refusal, and are not asked again. */
@@ -246,7 +290,7 @@ export async function servedWindow(config) {
         return 0;
     const key = JSON.stringify([endpoint, config.model]);
     const known = served.get(key);
-    if (known && (known.window > 0 || Date.now() - known.at < LISTING_MISS_MS))
+    if (known && (known.window > 0 || Date.now() - known.at < poolLimits.listingMissMs))
         return known.window;
     let window = 0;
     try {
@@ -324,8 +368,8 @@ export async function contextLimitFor(config, declared = 0) {
         const asked = misses.get(missKey);
         // Asked again, but not on every call. A model that is never coming answers the same zero
         // however often the endpoint is asked, and a caller sizing a window per turn pays a round
-        // trip for each of them; `LISTING_MISS_MS` is how long that answer is allowed to stand.
-        if (asked !== undefined && Date.now() - asked < LISTING_MISS_MS)
+        // trip for each of them; `listingMissMs` is how long that answer is allowed to stand.
+        if (asked !== undefined && Date.now() - asked < poolLimits.listingMissMs)
             return 0;
         // A failure is not remembered: an endpoint that was down when the last run started is not
         // an endpoint with no models, and a window nobody could ask about is not a failed run.
@@ -346,8 +390,12 @@ export async function contextLimitFor(config, declared = 0) {
     }
     return listed()?.contextLength ?? 0;
 }
-/** Forgets every cached client and listing. For tests, and for a settings change under test. */
+/**
+ * Forgets every cached client and listing, and puts `configureClients` back to the defaults. For
+ * tests, and for a settings change under test.
+ */
 export function resetClients() {
+    poolLimits = { ...CLIENT_DEFAULTS };
     clients.clear();
     listings.clear();
     misses.clear();

package/dist/index.d.ts

@@ -12,7 +12,7 @@
 export { type AgentLoopHooks, type AgentLoopOptions, type AgentLoopResult, buildBody, preselect, preview, resolveApiKey, runAgentLoop, type ToolCallOutcome, type ToolCallRequest, } from "./agent-loop.ts";
 export { type Capabilities, capabilitiesFor, type ModelCapabilities, modelCapabilitiesFor, type NegotiateOptions, negotiate, resetCapabilities, } from "./capabilities.ts";
 export type { CatalogServer } from "./catalog.ts";
-export { contextLimitFor, FIRST_TOKEN_FACTOR, firstTokenMs, getClient, listModels, type ModelInfo, NO_KEY, resetClients, servedWindow, timeoutMs, } from "./client.ts";
+export { type ClientPoolOptions, configureClients, contextLimitFor, endpointId, endpointKey, FIRST_TOKEN_FACTOR, firstTokenMs, getClient, listModels, type ModelInfo, NO_KEY, resetClients, servedWindow, timeoutMs, } from "./client.ts";
 export { COMPACT_AT, type CompactionOptions, type CompactionPlan, compactTranscript, KEEP_RATIO, type PruneOptions, planCompaction, pruneToolResults, SUMMARY_LEAD, SUMMARY_PROMPT, summariser, summaryInput, } from "./compaction.ts";
 export type { AgentConfig, Endpoint, ModelParams, RetryPolicy, ToolPolicy, } from "./config.ts";
 export { errorMessage } from "./errors.ts";

package/dist/index.js

@@ -11,7 +11,7 @@
  */
 export { buildBody, preselect, preview, resolveApiKey, runAgentLoop, } from "./agent-loop.js";
 export { capabilitiesFor, modelCapabilitiesFor, negotiate, resetCapabilities, } from "./capabilities.js";
-export { contextLimitFor, FIRST_TOKEN_FACTOR, firstTokenMs, getClient, listModels, NO_KEY, resetClients, servedWindow, timeoutMs, } from "./client.js";
+export { configureClients, contextLimitFor, endpointId, endpointKey, FIRST_TOKEN_FACTOR, firstTokenMs, getClient, listModels, NO_KEY, resetClients, servedWindow, timeoutMs, } from "./client.js";
 export { COMPACT_AT, compactTranscript, KEEP_RATIO, planCompaction, pruneToolResults, SUMMARY_LEAD, SUMMARY_PROMPT, summariser, summaryInput, } from "./compaction.js";
 export { errorMessage } from "./errors.js";
 export { configureEvents, emit, endRun, fold, history, resetEvents, watch, } from "./events.js";

package/llms.txt

@@ -44,14 +44,18 @@ The contract between whatever holds the tools and the loop that offers them to a
 
 ### client
 
+- `ClientPoolOptions` (type) — What the client pool is held to across a process.
+- `configureClients` — Changes what the client pool is held to, for a process whose endpoints are not shaped like the deployments these defaults were chosen for.
 - `contextLimitFor` — How much a model will read, in tokens.
+- `endpointId` — `endpointKey` hashed, for the remembered facts that can leave the process.
+- `endpointKey` — What counts as one endpoint, everywhere in this package that has to remember something about one — the model listings, `capabilities`, and the no-thinking hints in `side-task`.
 - `FIRST_TOKEN_FACTOR` — How many idle windows the first chunk gets when `firstTokenSeconds` is not given.
 - `firstTokenMs` — The wait for a streamed turn's first chunk, in the SDK's spelling: `undefined` is no limit.
 - `getClient` — The client for an endpoint, built once and kept.
 - `listModels` — Asks an endpoint what it serves, and remembers the answer.
 - `ModelInfo` (type) — A model an endpoint offers, and what it says the model will read.
 - `NO_KEY` — The SDK insists on a non-empty key even where the server will not look at it.
-- `resetClients` — Forgets every cached client and listing.
+- `resetClients` — Forgets every cached client and listing, and puts `configureClients` back to the defaults.
 - `servedWindow` — The window a local server is actually serving a model in, which its listing does not say.
 - `timeoutMs` — Zero, less, or absent means no limit, which the SDK spells as `undefined`.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@cubicecho/agent-core",
-  "version": "2.9.0",
+  "version": "2.10.0",
   "description": "The endpoint-agnostic half of an OpenAI-compatible agent loop: tool-schema compatibility, on-demand tool loading, one-shot side tasks, run events, and a pooled client.",
   "keywords": [
     "openai",