@cubicecho/agent-core 2.2.4 → 2.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -25,6 +25,7 @@ only, Node >=22.
25
25
  | `stream` | Reads one streamed turn back into a message: token callbacks, tool-call reassembly, and the idle watchdog that turns a silent endpoint into `EndpointSilent`. |
26
26
  | `capabilities` | What an endpoint turned out not to support — and, under it, what one model on that endpoint did not — plus the loop that answers either when it says so. `capabilitiesFor`, `modelCapabilitiesFor`, `negotiate`. |
27
27
  | `side-task` | One-shot calls that support a run without being one — small prompt, short answer, no tools, never worth failing the run over. |
28
+ | `hooks` | The host's side of lifecycle hooks: `gather` before a request and `notify` after, the shared context budget, `withContext` to put what they add on the turn's question, and `turnMessages` to hand them a transcript. Running a hook is a runner the caller passes. |
28
29
  | `events` | The in-memory bus a watcher reads while a run happens: `emit`, `watch`, `history`, `fold`. A watcher's backlog is capped and reports its own gaps. |
29
30
  | `client` | A pooled `OpenAI` client per endpoint, plus the context-window listing and its cache. |
30
31
  | `retry` | What to do when a request is lost, refused or too big: `isTransient`, `backoffMs`, `ContextOverflow`, `EndpointSilent`, `requestTokens`. |
@@ -54,7 +55,7 @@ import {
54
55
  } from "@cubicecho/agent-core";
55
56
 
56
57
  const declared = sanitizeTools(tools);
57
- const supports = capabilitiesFor(config.baseUrl);
58
+ const supports = capabilitiesFor(config.baseUrl, config.apiKey);
58
59
 
59
60
  const turn = await negotiate(supports, (supports, produced) =>
60
61
  streamTurn(
@@ -70,6 +71,17 @@ const turn = await negotiate(supports, (supports, produced) =>
70
71
  );
71
72
  ```
72
73
 
74
+ The key is passed because it is part of what an endpoint *is* here, not only how it is paid
75
+ for: a router is free to send two keys to two different backends, and then what one of them
76
+ refused is not a fact about the other. Absent and empty read the same, so a local server with no
77
+ key is one entry however its caller spells it.
78
+
79
+ `Turn` is `content`, `toolCalls`, `usage` and `finishReason`. The last is worth reading: a turn
80
+ cut off at the token ceiling comes back looking exactly like a finished one, with truncated prose
81
+ or — the case that bites — a tool call whose `arguments` stop mid-JSON, so the caller meets a
82
+ parse failure with nothing to attribute it to. `finishReason` is `"length"` there, `""` where the
83
+ endpoint never said.
84
+
73
85
  ## What the model refuses, rather than the server
74
86
 
75
87
  `strictSchemas` and `usageInStream` are facts about a server. Three more arrive through the same
@@ -157,6 +169,12 @@ before it and the transcript does not change between retries. A `ContextOverflow
157
169
  neither a capability `negotiate` can answer nor something `isTransient` accepts, so it leaves
158
170
  both loops on the first attempt.
159
171
 
172
+ `contextLimit` decides how early the caller hears, not what it hears. Without one, the endpoint
173
+ refuses instead — one round trip later — and `runTurn` reads that refusal back through
174
+ `isOverflow` and raises the same `ContextOverflow`, carrying the endpoint's own wording and the
175
+ original error as `cause`. A rate limit borrows those words and means the opposite ("Request too
176
+ large for gpt-4o ... on tokens per min"); that is ruled out and waited through as the 429 it is.
177
+
160
178
  ## Watching a run
161
179
 
162
180
  `watch` replays what the run has already emitted, then yields what happens next until `done`.
@@ -166,13 +184,20 @@ de-duplicates on.
166
184
  ```ts
167
185
  import { watch } from "@cubicecho/agent-core";
168
186
 
169
- for await (const event of watch(runId)) {
187
+ // The signal is how a watcher leaves early — a disconnected client, a page navigated away from.
188
+ for await (const event of watch(runId, request.signal)) {
170
189
  render(event);
171
190
  }
172
191
  ```
173
192
 
193
+ Pass one for anything that can go away before the run ends. `watch` otherwise finishes only on
194
+ `done`, and returning the generator is not a substitute: waiting for the next event it is
195
+ suspended at an `await` rather than at a `yield`, so a `return()` queues behind a promise only
196
+ that event can settle. A watcher stuck there also holds its run's backlog past every deadline,
197
+ because the sweep skips any stream a listener is on.
198
+
174
199
  A watcher that stops keeping up is capped rather than left to grow. Its queue holds the most
175
- recent 1000 events — trimmed in batches, so it runs a little over that before cutting back —
200
+ recent 1000 events by default — trimmed in batches, so it runs a little over that before cutting back —
176
201
  the oldest go, and what is dropped is released where it is dropped, not held until a consumer
177
202
  that has already stalled next reads.
178
203
 
@@ -184,6 +209,90 @@ repeat, and a client doing what `seq` is documented for would throw away one of
184
209
  `history` reads what a run has emitted without subscribing, `fold` collapses a token stream into
185
210
  blocks for display, and `endRun` drops a finished run's buffer.
186
211
 
212
+ The four numbers behind that — the backlog cap, the slack it is trimmed in batches of, and the two
213
+ deadlines the sweep reaps on — are defaults rather than decisions. `configureEvents` moves them,
214
+ and returns the whole set as it now stands:
215
+
216
+ ```ts
217
+ import { configureEvents } from "@cubicecho/agent-core";
218
+
219
+ // A long-lived server with many concurrent runs, whose tool calls are minutes rather than hours.
220
+ configureEvents({ maxEvents: 200, retainUnendedMs: 5 * 60_000 });
221
+ ```
222
+
223
+ It is module-level because the bus is: there is one of each per process, and a run does not carry
224
+ its own. A field left out keeps what it had, and so does one given something that is not a
225
+ positive number — nothing here has a meaningful zero, and a `0` standing in for "no opinion" must
226
+ not turn the backlog off. `resetEvents` (and `resetAll`) puts the defaults back, so one test's cap
227
+ is not the next one's.
228
+
229
+ `expandNames`, `carryOver`, `preselection`, `preselectInput` and `preselectSystem` take their caps
230
+ the same way — as a last argument defaulting to `MAX_PER_LOAD` or `MAX_CARRIED`. A resolution
231
+ carries the cap it was held to, so `loadResult` tells the model the number it was actually
232
+ measured against rather than the module's own.
233
+
234
+ ## Hooks
235
+
236
+ A hook is something a host runs at a point in a session — `sessionStart` and `beforeTurn`, whose
237
+ output can reach the request, then `afterTurn`, `beforeCompact`, `sessionEnd` and `sessionDelete`,
238
+ which can only read what happened. What runs is not decided here. `gather` and `notify` take a
239
+ `HookRunner`, and `@cubicecho/agent-mcp-pool`'s `runHooks` is one as it stands: its types are the
240
+ shapes restated here, so its outcomes pass straight through without either package importing the
241
+ other.
242
+
243
+ ```ts
244
+ import {
245
+ emit,
246
+ gather,
247
+ type HookRunner,
248
+ notify,
249
+ turnIndex,
250
+ turnMessages,
251
+ withContext,
252
+ } from "@cubicecho/agent-core";
253
+
254
+ const run: HookRunner = (event, context, { signal }) =>
255
+ pool.runHooks(event, context, { signal, servers: agent.mcpServers });
256
+
257
+ const context = { session: { id }, host: "my-host", prompt, turn: { index: turnIndex(messages) } };
258
+ const gathered = await gather(
259
+ run,
260
+ messages.length === 0 ? ["sessionStart", "beforeTurn"] : ["beforeTurn"],
261
+ context,
262
+ { signal, onNote: (note) => emit(runId, { kind: "notice", name: note.hookId, text: note.error ?? note.text }) },
263
+ );
264
+ messages.push({ role: "user", content: prompt });
265
+ const request = withContext(messages, messages.length - 1, gathered.context);
266
+ // ... the turn ...
267
+ void notify(run, "afterTurn", { ...context, reply, turn: { ...context.turn, messages: turnMessages(id, messages, turnStart) } });
268
+ ```
269
+
270
+ The context goes on this turn's question and never into the system prompt — a prompt that changed
271
+ every turn would miss the prompt cache every turn — and `withContext` returns a new array, so a
272
+ host that stores what the user typed never stores the context as something they said. Every
273
+ injecting hook shares `HOOK_CONTEXT_TOKENS` (2000) by default, each held to its own `maxTokens`
274
+ inside that, so a generous hook cannot crowd out the conversation it was meant to inform.
275
+
276
+ The budget is the caller's to move, at either level. `configureHooks` sets it for the process, and
277
+ `maxTokens` on `gather` (or the second argument to `assembleContext`) sets it for one request and
278
+ wins over it — the one to reach for when the budget follows the model, since a 128k window can
279
+ afford more recall than an 8k one:
280
+
281
+ ```ts
282
+ import { configureHooks, gather } from "@cubicecho/agent-core";
283
+
284
+ configureHooks({ contextTokens: 4000 });
285
+ const gathered = await gather(run, ["beforeTurn"], context, { maxTokens: contextLimit / 20 });
286
+ ```
287
+
288
+ Either one given something that is not a number above zero keeps what was there, as
289
+ `configureEvents` does, so a `0` threaded through for "no opinion" does not switch recall off.
290
+ `resetHooks` (and `resetAll`) puts the default back.
291
+
292
+ Neither function rejects. A hook failing is an outcome, and a runner that throws outright is
293
+ noted once for its event and costs only that event's context. `notify` takes no signal: a reader
294
+ who leaves once the turn is answered has not asked for it not to be remembered.
295
+
187
296
  ## The config seam
188
297
 
189
298
  Nothing here imports a config type from a consumer, and no function asks for a whole
@@ -205,6 +314,32 @@ The seam is not finished. `timeoutMs` narrows to the one field it reads, but `ge
205
314
  asks for the whole of `Endpoint`. `requestTimeoutSeconds` became optional in v2, so a consumer
206
315
  with no timeout to give now leaves it out rather than inventing a `0`.
207
316
 
317
+ ## What is kept for the life of the process
318
+
319
+ Four caches outlive any one run: the `OpenAI` clients, the model listings, the latched
320
+ capabilities, and `side-task`'s no-thinking hints. All four are module-level and keyed on the same
321
+ notion of an endpoint — its base URL and its API key — and the clients' key carries the request
322
+ timeout as well, since that changes how a request is sent.
323
+
324
+ **They are keyed per deployment, not per request.** What belongs in them is an endpoint an
325
+ operator configured: a settings row, an agent definition, an environment variable. Everything
326
+ bounding them assumes that, and a consumer that mints an API key per *user* breaks the assumption
327
+ — each tenant gets its own client, its own connection pool and its own latches, held until
328
+ `resetAll`.
329
+
330
+ That is bounded rather than unbounded: the client pool is a 32-entry LRU, and an evicted client
331
+ costs its connection pool and nothing else, since the next request through that endpoint builds
332
+ another. But churning connection pools is not sharing them, and at that point a client of your
333
+ own, built and held per tenant, is the better answer than this.
334
+
335
+ The listings cache also remembers, for half a minute, that an endpoint did not name a model — the
336
+ case where a configured name never matches anything the server serves (a llama.cpp `-a` alias, an
337
+ OpenRouter `:free` suffix, a typo) would otherwise fetch the listing on every call and answer the
338
+ same zero each time. Half a minute later it asks again, so a model pulled onto a box that has been
339
+ up a week is still picked up without a restart.
340
+
341
+ `resetAll` drops all four, and `reset.ts` names each seam separately for a test that wants one.
342
+
208
343
  ## Where the merged behaviour came from
209
344
 
210
345
  - `schema-compat` — `kanban_server`/`task_server`'s version, which strips **every** sibling of a
@@ -70,10 +70,14 @@ export interface ModelCapabilities {
70
70
  * What this endpoint is known not to support. The same object every time, so what `negotiate`
71
71
  * latches off stays off.
72
72
  *
73
- * @param baseUrl Identifies the endpoint. The two flags on it are per-server; what is
74
- * per-model hangs off `models`, which `modelCapabilitiesFor` reads.
73
+ * @param baseUrl Where the endpoint is. The two flags on it are per-server; what is per-model
74
+ * hangs off `models`, which `modelCapabilitiesFor` reads.
75
+ * @param apiKey The rest of the endpoint's identity. Optional, because it changes nothing for
76
+ * the ordinary case of one key per base URL and a caller with one need not thread it through;
77
+ * absent reads as `NO_KEY`, exactly as it does in `getClient`, so an endpoint with no key and
78
+ * one that passes `undefined` share an entry rather than holding two.
75
79
  */
76
- export declare function capabilitiesFor(baseUrl: string): Capabilities;
80
+ export declare function capabilitiesFor(baseUrl: string, apiKey?: string): Capabilities;
77
81
  /**
78
82
  * What this model on this endpoint is known not to support. The same object every time, so what
79
83
  * `negotiate` latches off stays off.
@@ -1,29 +1,42 @@
1
+ import { endpointKey } from "./client.js";
1
2
  import { errorMessage } from "./errors.js";
2
3
  import { isGrammarError } from "./schema-compat.js";
3
4
  /**
4
5
  * What each endpoint cannot do, remembered for the life of the process.
5
6
  *
6
- * Keyed by base URL, because these are facts about the server on the other end and not about
7
- * this one. A llama.cpp box that cannot compile a grammar and a cloud API that can are both
8
- * reachable from one settings row over its lifetime — an operator retargets it from Ollama this
9
- * afternoon to OpenAI this evening — and the first one's refusal must not quietly strip
7
+ * Keyed by `endpointKey`, because these are facts about the server on the other end and not
8
+ * about this one. A llama.cpp box that cannot compile a grammar and a cloud API that can are
9
+ * both reachable from one settings row over its lifetime — an operator retargets it from Ollama
10
+ * this afternoon to OpenAI this evening — and the first one's refusal must not quietly strip
10
11
  * pattern/format from the second one's requests, or silently cost it its token counts, for the
11
12
  * rest of the process. Bounded by the number of endpoints ever configured, which is a settings
12
13
  * row's worth.
14
+ *
15
+ * The API key is part of that identity, the same as it is for the client pool and the model
16
+ * listings. A router — LiteLLM, OpenRouter, a gateway with several boxes behind it — is free to
17
+ * send two keys to two different backends, and then what one of them refused is not a fact about
18
+ * the other. Keyed on the URL alone the first caller through the gateway latched for everyone
19
+ * behind it, including the `models` map underneath, which is the level where two keys through
20
+ * one host are most likely to differ at all.
13
21
  */
14
22
  const capabilities = new Map();
15
23
  /**
16
24
  * What this endpoint is known not to support. The same object every time, so what `negotiate`
17
25
  * latches off stays off.
18
26
  *
19
- * @param baseUrl Identifies the endpoint. The two flags on it are per-server; what is
20
- * per-model hangs off `models`, which `modelCapabilitiesFor` reads.
27
+ * @param baseUrl Where the endpoint is. The two flags on it are per-server; what is per-model
28
+ * hangs off `models`, which `modelCapabilitiesFor` reads.
29
+ * @param apiKey The rest of the endpoint's identity. Optional, because it changes nothing for
30
+ * the ordinary case of one key per base URL and a caller with one need not thread it through;
31
+ * absent reads as `NO_KEY`, exactly as it does in `getClient`, so an endpoint with no key and
32
+ * one that passes `undefined` share an entry rather than holding two.
21
33
  */
22
- export function capabilitiesFor(baseUrl) {
23
- let known = capabilities.get(baseUrl);
34
+ export function capabilitiesFor(baseUrl, apiKey) {
35
+ const key = endpointKey({ baseUrl, apiKey });
36
+ let known = capabilities.get(key);
24
37
  if (!known) {
25
38
  known = { strictSchemas: true, usageInStream: true, models: new Map() };
26
- capabilities.set(baseUrl, known);
39
+ capabilities.set(key, known);
27
40
  }
28
41
  return known;
29
42
  }
package/dist/client.d.ts CHANGED
@@ -18,6 +18,13 @@ export declare const timeoutMs: (config: Pick<Endpoint, "requestTimeoutSeconds">
18
18
  * Pooled on the three fields that change how a request is sent, so agents sharing a server share
19
19
  * a connection and agents on different servers never share a client.
20
20
  *
21
+ * The pool is per *deployment*, not per request: what belongs in this map is an endpoint an
22
+ * operator configured, and everything cached in this package is bounded on that reading. A
23
+ * consumer that mints an API key per user still gets a working client, but it is churning
24
+ * connection pools rather than sharing them and holding every one of them — `MAX_CLIENTS` keeps
25
+ * that from being unbounded, and it is the point at which a client of your own, built and held
26
+ * per tenant, is the better answer than this.
27
+ *
21
28
  * @param config Where to send requests and how long to wait. An absent `apiKey` becomes `NO_KEY`.
22
29
  */
23
30
  export declare function getClient(config: Endpoint): OpenAI;
@@ -26,6 +33,27 @@ export interface ModelInfo {
26
33
  id: string;
27
34
  contextLength: number;
28
35
  }
36
+ /**
37
+ * What counts as one endpoint, everywhere in this package that has to remember something about
38
+ * one — this file's listings, `capabilities`, and the no-thinking hints in `side-task`.
39
+ *
40
+ * The URL and the key together, because the key is part of what is on the other end rather than
41
+ * only how it is paid for: a router is free to send two keys to two different backends, and then
42
+ * what one of them refused is not a fact about the other. Absent reads as `NO_KEY`, so an
43
+ * endpoint with no key and one that passes `undefined` are one entry rather than two.
44
+ *
45
+ * Stringified rather than joined on a separator, for the reason `getClient` gives: no character
46
+ * is impossible in a URL or a key, and two endpoints must never collide on one entry.
47
+ *
48
+ * @param config Read for `baseUrl` and `apiKey` alone, and the key is optional here where
49
+ * `Endpoint` requires it — `capabilitiesFor` is handed a URL and maybe a key rather than a whole
50
+ * config, and absent and empty already mean the same thing. The timeout is deliberately not in
51
+ * it; see `listings`.
52
+ */
53
+ export declare const endpointKey: (config: {
54
+ baseUrl: string;
55
+ apiKey?: string;
56
+ }) => string;
29
57
  /**
30
58
  * Asks an endpoint what it serves, and remembers the answer.
31
59
  *
package/dist/client.js CHANGED
@@ -26,14 +26,39 @@ export const timeoutMs = (config) => {
26
26
  * `maxRetries: 0` turns the SDK's own retrying off. Streaming is what this is for, and a stream
27
27
  * that has already emitted tokens must not be replayed from the top — the caller knows whether
28
28
  * anything has been produced yet and the SDK does not. See `retry.ts`.
29
+ *
30
+ * Bounded, because a cache is only a cache while its keys are. Insertion order is the whole of
31
+ * the eviction: a `Map` iterates oldest-first, and `getClient` re-inserts on a hit.
29
32
  */
30
33
  const clients = new Map();
34
+ /**
35
+ * How many endpoints' clients are kept at once.
36
+ *
37
+ * This is a backstop rather than a design. The key includes the API key, and every argument in
38
+ * this file for what bounds these caches is "a settings row's worth" — true of the deployments
39
+ * this was written for, and false the moment a consumer mints a key per *user*, where the map
40
+ * grows one client and one connection pool per tenant for the life of the process with
41
+ * `resetClients` as the only release.
42
+ *
43
+ * Thirty-two is far more endpoints than a settings table holds, so a deployment-shaped consumer
44
+ * never reaches it, and an evicted client costs its connection pool and nothing else — the next
45
+ * request through that endpoint builds another. See `getClient` for the constraint stated as a
46
+ * constraint.
47
+ */
48
+ const MAX_CLIENTS = 32;
31
49
  /**
32
50
  * The client for an endpoint, built once and kept.
33
51
  *
34
52
  * Pooled on the three fields that change how a request is sent, so agents sharing a server share
35
53
  * a connection and agents on different servers never share a client.
36
54
  *
55
+ * The pool is per *deployment*, not per request: what belongs in this map is an endpoint an
56
+ * operator configured, and everything cached in this package is bounded on that reading. A
57
+ * consumer that mints an API key per user still gets a working client, but it is churning
58
+ * connection pools rather than sharing them and holding every one of them — `MAX_CLIENTS` keeps
59
+ * that from being unbounded, and it is the point at which a client of your own, built and held
60
+ * per tenant, is the better answer than this.
61
+ *
37
62
  * @param config Where to send requests and how long to wait. An absent `apiKey` becomes `NO_KEY`.
38
63
  */
39
64
  export function getClient(config) {
@@ -43,10 +68,23 @@ export function getClient(config) {
43
68
  // key, and two different endpoints must never resolve to the same cached client.
44
69
  const key = JSON.stringify([config.baseUrl, apiKey, timeout]);
45
70
  const existing = clients.get(key);
46
- if (existing)
71
+ if (existing) {
72
+ // Re-inserted so it counts as the youngest. A `Map` keeps insertion order and hands the
73
+ // oldest key over first, which is the whole of the LRU below.
74
+ clients.delete(key);
75
+ clients.set(key, existing);
47
76
  return existing;
77
+ }
48
78
  const client = new OpenAI({ baseURL: config.baseUrl, apiKey, timeout, maxRetries: 0 });
49
79
  clients.set(key, client);
80
+ // Nothing is closed on the way out. The SDK holds no handle a caller can release, and an
81
+ // evicted client is garbage once whatever request is still in flight on it has finished —
82
+ // dropping the reference is the whole of the eviction.
83
+ for (const oldest of clients.keys()) {
84
+ if (clients.size <= MAX_CLIENTS)
85
+ break;
86
+ clients.delete(oldest);
87
+ }
50
88
  return client;
51
89
  }
52
90
  /**
@@ -76,7 +114,7 @@ function contextLengthOf(model) {
76
114
  /**
77
115
  * The last listing from each endpoint, so a run can size its window without a round trip.
78
116
  *
79
- * Keyed by base URL and key, because two endpoints are two different sets of models and one of
117
+ * Keyed by `endpointKey`, because two endpoints are two different sets of models and one of
80
118
  * them having answered says nothing about the other — and because a key can be the difference
81
119
  * between what a router will show one caller and another.
82
120
  *
@@ -88,7 +126,41 @@ function contextLengthOf(model) {
88
126
  * whatever was there rather than emptying it.
89
127
  */
90
128
  const listings = new Map();
91
- const endpointKey = (config) => JSON.stringify([config.baseUrl, config.apiKey || NO_KEY]);
129
+ /**
130
+ * When an endpoint was last asked about a model it did not name, keyed on the two together.
131
+ *
132
+ * `contextLimitFor` asks again whenever the listing does not hold the model, which is right for
133
+ * a model that arrives late and wrong for one that is never coming. The second is the ordinary
134
+ * case rather than the exotic one — a llama.cpp box served under a `-a` alias that does not
135
+ * match the configured name, an OpenRouter `:free` suffix, a typo in a settings row — and there
136
+ * every call fetches the listing, re-reads it, finds the same absence and answers the same zero.
137
+ *
138
+ * Remembering the miss for a moment answers both: within `LISTING_MISS_MS` nobody is asked, and
139
+ * after it the endpoint is asked again, so an `ollama pull` on a box that has been up a week is
140
+ * picked up within the minute instead of at the next restart. Bounded by the (endpoint, model)
141
+ * pairs actually asked about, which is the bound the listings themselves have.
142
+ */
143
+ const misses = new Map();
144
+ /** How long a model an endpoint did not list stays unlisted before it is asked about again. */
145
+ const LISTING_MISS_MS = 30_000;
146
+ /**
147
+ * What counts as one endpoint, everywhere in this package that has to remember something about
148
+ * one — this file's listings, `capabilities`, and the no-thinking hints in `side-task`.
149
+ *
150
+ * The URL and the key together, because the key is part of what is on the other end rather than
151
+ * only how it is paid for: a router is free to send two keys to two different backends, and then
152
+ * what one of them refused is not a fact about the other. Absent reads as `NO_KEY`, so an
153
+ * endpoint with no key and one that passes `undefined` are one entry rather than two.
154
+ *
155
+ * Stringified rather than joined on a separator, for the reason `getClient` gives: no character
156
+ * is impossible in a URL or a key, and two endpoints must never collide on one entry.
157
+ *
158
+ * @param config Read for `baseUrl` and `apiKey` alone, and the key is optional here where
159
+ * `Endpoint` requires it — `capabilitiesFor` is handed a URL and maybe a key rather than a whole
160
+ * config, and absent and empty already mean the same thing. The timeout is deliberately not in
161
+ * it; see `listings`.
162
+ */
163
+ export const endpointKey = (config) => JSON.stringify([config.baseUrl, config.apiKey || NO_KEY]);
92
164
  /**
93
165
  * Asks an endpoint what it serves, and remembers the answer.
94
166
  *
@@ -122,6 +194,7 @@ export async function contextLimitFor(config, declared = 0) {
122
194
  if (declared > 0)
123
195
  return declared;
124
196
  const key = endpointKey(config);
197
+ const listed = () => listings.get(key)?.find((model) => model.id === config.model);
125
198
  // The listing is asked for again when it does not name this model, rather than only when
126
199
  // there is no listing at all. Models arrive after a process starts — an `ollama pull` on a
127
200
  // box that has been up a week, a worker added to a router, a name the operator has only just
@@ -129,10 +202,14 @@ export async function contextLimitFor(config, declared = 0) {
129
202
  // of them until a restart. Zero means "nobody knows", so what the operator loses is the
130
203
  // context meter and, in a consumer that compacts on it, compaction: the session then runs at
131
204
  // the window instead of under it and fails against the endpoint's own refusal.
132
- //
133
- // The cost of asking again is one listing per call while the model really is absent, which
134
- // is exactly the case where the cached answer would have been wrong.
135
- if (!listings.get(key)?.some((model) => model.id === config.model)) {
205
+ if (!listed()) {
206
+ const missKey = JSON.stringify([key, config.model]);
207
+ const asked = misses.get(missKey);
208
+ // Asked again, but not on every call. A model that is never coming answers the same zero
209
+ // however often the endpoint is asked, and a caller sizing a window per turn pays a round
210
+ // trip for each of them; `LISTING_MISS_MS` is how long that answer is allowed to stand.
211
+ if (asked !== undefined && Date.now() - asked < LISTING_MISS_MS)
212
+ return 0;
136
213
  // A failure is not remembered: an endpoint that was down when the last run started is not
137
214
  // an endpoint with no models, and a window nobody could ask about is not a failed run.
138
215
  try {
@@ -141,12 +218,20 @@ export async function contextLimitFor(config, declared = 0) {
141
218
  catch {
142
219
  return 0;
143
220
  }
221
+ // Recorded where the endpoint was actually asked, and only there. Stamping it on the calls
222
+ // that skipped the request would push the deadline out ahead of any caller polling faster
223
+ // than the interval, which is a memory that never expires rather than one that expires in
224
+ // half a minute.
225
+ if (listed())
226
+ misses.delete(missKey);
227
+ else
228
+ misses.set(missKey, Date.now());
144
229
  }
145
- const listed = listings.get(key) ?? [];
146
- return listed.find((model) => model.id === config.model)?.contextLength ?? 0;
230
+ return listed()?.contextLength ?? 0;
147
231
  }
148
232
  /** Forgets every cached client and listing. For tests, and for a settings change under test. */
149
233
  export function resetClients() {
150
234
  clients.clear();
151
235
  listings.clear();
236
+ misses.clear();
152
237
  }
package/dist/events.d.ts CHANGED
@@ -9,6 +9,60 @@
9
9
  * In memory on purpose: it is debugging output, worth nothing once the run has finished and its
10
10
  * outcome is in the database. Nothing here survives a restart, and nothing here is the record.
11
11
  */
12
+ /** What the bus keeps and for how long. Every field optional; see `configureEvents`. */
13
+ export interface EventBusOptions {
14
+ /** How many events one run keeps for a watcher that joins late. A chatty run loses its oldest. */
15
+ maxEvents?: number;
16
+ /**
17
+ * How far past the cap the backlog is allowed to run before it is trimmed.
18
+ *
19
+ * Dropping the oldest event on every push means shifting a thousand-element array tens of
20
+ * thousands of times over a reasoning run — the one thing in here that would ever show up in a
21
+ * profile. Trimming in batches makes it a few dozen splices instead, at the cost of the backlog
22
+ * sometimes being a little longer than the cap, which nothing depends on. The one field here
23
+ * that is an implementation detail rather than a policy: raising it trades memory for fewer
24
+ * splices, and there is no reason to lower it.
25
+ */
26
+ trimSlack?: number;
27
+ /** How long a finished run stays readable, for a watcher that arrives just after the end. */
28
+ retainMs?: number;
29
+ /**
30
+ * The same for a run that has not said `done`, which is a far more dangerous thing to drop.
31
+ *
32
+ * A finished run has nothing more to say, so forgetting it a minute later costs a late watcher
33
+ * a backlog and nothing else. An unfinished one is still writing: `touched` only moves on
34
+ * `emit`, so a live run that spends a minute inside one slow tool call looked exactly like an
35
+ * abandoned one and was reaped out from under itself. What made that more than a lost backlog
36
+ * is that the next `emit` builds a fresh stream with `seq` back at zero — and `seq` is the
37
+ * field `RunEvent` documents for ordering and de-duplication, so a client that reconnects
38
+ * across the gap discards the new events as ones it has already seen.
39
+ *
40
+ * The sweep still has to reap them, because a run killed by a signal or simply forgotten
41
+ * reaches no `done` either. This is the backstop for a caller that never says so; `endRun` is
42
+ * for one that knows. Anything shorter than the longest a tool call may take is a live run
43
+ * reaped out from under itself.
44
+ */
45
+ retainUnendedMs?: number;
46
+ }
47
+ /**
48
+ * Changes what the bus keeps, for a process whose runs are not shaped like the ones these
49
+ * defaults were chosen for.
50
+ *
51
+ * The bus is one module-level thing rather than an object a caller holds, so this is too: it is
52
+ * a deployment's setting, said once at startup, and not something to move around under a run.
53
+ * What it costs is memory against how much of a run a late or slow watcher can still read —
54
+ * a server with hundreds of concurrent runs wants a smaller backlog, and one whose tool calls
55
+ * take an hour wants a longer `retainUnendedMs` than the thirty minutes assumed here.
56
+ *
57
+ * Changes apply from the next event and the next sweep. Nothing already buffered is trimmed to
58
+ * a cap that has just come down, because the trim happens on push; the backlog settles to the
59
+ * new number as the run goes on.
60
+ *
61
+ * @param options The bounds to change. A field left out — or given anything that is not a
62
+ * number above zero — keeps what it has, so a partial or a half-built config narrows nothing.
63
+ * @returns Everything in force afterwards, including what this call did not change.
64
+ */
65
+ export declare function configureEvents(options?: EventBusOptions): Required<EventBusOptions>;
12
66
  /** Which kind of thing happened, and what `text`, `name`, `ok` and `usage` carry for it. */
13
67
  export type RunEventKind =
14
68
  /** A step of a caller's own flow began. `name` is the step, `text` its kind. */
@@ -107,11 +161,22 @@ export declare function emit(runId: string, input: RunEventInput): RunEvent;
107
161
  * inside the retention window — reads the same story as one that was there from the start.
108
162
  *
109
163
  * @param runId The run to follow. One that has not started yet is waited on, not refused.
164
+ * @param signal Stops following. The only other way out is the run's own `done`, and a watcher
165
+ * with no way out is a leak rather than a lost backlog: the sweep below skips any stream a
166
+ * listener is on, so a run that dies without `done` pins its backlog for the life of the process.
167
+ * Returning the generator is not that way out — parked on the promise at the foot of this
168
+ * function it is suspended at an `await` rather than at a `yield`, and a `return()` there is
169
+ * queued behind a promise only the next event can settle. An abort resolves that promise itself.
110
170
  */
111
- export declare function watch(runId: string): AsyncGenerator<RunEvent>;
171
+ export declare function watch(runId: string, signal?: AbortSignal): AsyncGenerator<RunEvent>;
112
172
  /**
113
173
  * The backlog alone, for a caller that wants a snapshot rather than a subscription.
114
174
  *
175
+ * The array is a copy; the events in it are not. They are the same objects the bus holds and
176
+ * every watcher was handed, so writing to one rewrites the run for everybody — which is what
177
+ * `fold` copies to avoid, and this is the other half of the same warning. Read them, or copy
178
+ * what you mean to change.
179
+ *
115
180
  * @param runId The run to read. An unknown or already-swept run gives an empty array.
116
181
  */
117
182
  export declare const history: (runId: string) => RunEvent[];
@@ -121,6 +186,11 @@ export declare const history: (runId: string) => RunEvent[];
121
186
  * Named for what it forgets rather than bare `reset`, which sat in a consumer's imports beside
122
187
  * `resetAll`, `resetClients`, `resetCapabilities` and `resetHints` saying nothing about which
123
188
  * of the five it was — `reset.ts` had to alias it on the way in to stay readable.
189
+ *
190
+ * `configureEvents` is undone too, for the reason `reset.ts` gives about latches: a bus left
191
+ * holding a cap one test set is the same order-dependent suite, passing where that test ran
192
+ * first and failing where it did not. A consumer that configures at startup and resets at
193
+ * teardown configures again, which is the same line it already wrote once.
124
194
  */
125
195
  export declare const resetEvents: () => void;
126
196
  /**