@cubicecho/agent-core 2.2.4 → 2.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +138 -3
- package/dist/capabilities.d.ts +7 -3
- package/dist/capabilities.js +22 -9
- package/dist/client.d.ts +28 -0
- package/dist/client.js +94 -9
- package/dist/events.d.ts +71 -1
- package/dist/events.js +70 -35
- package/dist/hooks.d.ts +271 -0
- package/dist/hooks.js +256 -0
- package/dist/index.d.ts +5 -4
- package/dist/index.js +5 -4
- package/dist/reset.d.ts +6 -5
- package/dist/reset.js +8 -5
- package/dist/retry.d.ts +5 -0
- package/dist/retry.js +5 -0
- package/dist/run-turn.d.ts +5 -0
- package/dist/run-turn.js +14 -1
- package/dist/side-task.js +8 -8
- package/dist/stream.d.ts +19 -1
- package/dist/stream.js +11 -1
- package/dist/tool-loading.d.ts +30 -10
- package/dist/tool-loading.js +45 -18
- package/llms.txt +30 -2
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -25,6 +25,7 @@ only, Node >=22.
|
|
|
25
25
|
| `stream` | Reads one streamed turn back into a message: token callbacks, tool-call reassembly, and the idle watchdog that turns a silent endpoint into `EndpointSilent`. |
|
|
26
26
|
| `capabilities` | What an endpoint turned out not to support — and, under it, what one model on that endpoint did not — plus the loop that answers either when it says so. `capabilitiesFor`, `modelCapabilitiesFor`, `negotiate`. |
|
|
27
27
|
| `side-task` | One-shot calls that support a run without being one — small prompt, short answer, no tools, never worth failing the run over. |
|
|
28
|
+
| `hooks` | The host's side of lifecycle hooks: `gather` before a request and `notify` after, the shared context budget, `withContext` to put what they add on the turn's question, and `turnMessages` to hand them a transcript. Running a hook is a runner the caller passes. |
|
|
28
29
|
| `events` | The in-memory bus a watcher reads while a run happens: `emit`, `watch`, `history`, `fold`. A watcher's backlog is capped and reports its own gaps. |
|
|
29
30
|
| `client` | A pooled `OpenAI` client per endpoint, plus the context-window listing and its cache. |
|
|
30
31
|
| `retry` | What to do when a request is lost, refused or too big: `isTransient`, `backoffMs`, `ContextOverflow`, `EndpointSilent`, `requestTokens`. |
|
|
@@ -54,7 +55,7 @@ import {
|
|
|
54
55
|
} from "@cubicecho/agent-core";
|
|
55
56
|
|
|
56
57
|
const declared = sanitizeTools(tools);
|
|
57
|
-
const supports = capabilitiesFor(config.baseUrl);
|
|
58
|
+
const supports = capabilitiesFor(config.baseUrl, config.apiKey);
|
|
58
59
|
|
|
59
60
|
const turn = await negotiate(supports, (supports, produced) =>
|
|
60
61
|
streamTurn(
|
|
@@ -70,6 +71,17 @@ const turn = await negotiate(supports, (supports, produced) =>
|
|
|
70
71
|
);
|
|
71
72
|
```
|
|
72
73
|
|
|
74
|
+
The key is passed because it is part of what an endpoint *is* here, not only how it is paid
|
|
75
|
+
for: a router is free to send two keys to two different backends, and then what one of them
|
|
76
|
+
refused is not a fact about the other. Absent and empty read the same, so a local server with no
|
|
77
|
+
key is one entry however its caller spells it.
|
|
78
|
+
|
|
79
|
+
`Turn` is `content`, `toolCalls`, `usage` and `finishReason`. The last is worth reading: a turn
|
|
80
|
+
cut off at the token ceiling comes back looking exactly like a finished one, with truncated prose
|
|
81
|
+
or — the case that bites — a tool call whose `arguments` stop mid-JSON, so the caller meets a
|
|
82
|
+
parse failure with nothing to attribute it to. `finishReason` is `"length"` there, `""` where the
|
|
83
|
+
endpoint never said.
|
|
84
|
+
|
|
73
85
|
## What the model refuses, rather than the server
|
|
74
86
|
|
|
75
87
|
`strictSchemas` and `usageInStream` are facts about a server. Three more arrive through the same
|
|
@@ -157,6 +169,12 @@ before it and the transcript does not change between retries. A `ContextOverflow
|
|
|
157
169
|
neither a capability `negotiate` can answer nor something `isTransient` accepts, so it leaves
|
|
158
170
|
both loops on the first attempt.
|
|
159
171
|
|
|
172
|
+
`contextLimit` decides how early the caller hears, not what it hears. Without one, the endpoint
|
|
173
|
+
refuses instead — one round trip later — and `runTurn` reads that refusal back through
|
|
174
|
+
`isOverflow` and raises the same `ContextOverflow`, carrying the endpoint's own wording and the
|
|
175
|
+
original error as `cause`. A rate limit borrows those words and means the opposite ("Request too
|
|
176
|
+
large for gpt-4o ... on tokens per min"); that is ruled out and waited through as the 429 it is.
|
|
177
|
+
|
|
160
178
|
## Watching a run
|
|
161
179
|
|
|
162
180
|
`watch` replays what the run has already emitted, then yields what happens next until `done`.
|
|
@@ -166,13 +184,20 @@ de-duplicates on.
|
|
|
166
184
|
```ts
|
|
167
185
|
import { watch } from "@cubicecho/agent-core";
|
|
168
186
|
|
|
169
|
-
|
|
187
|
+
// The signal is how a watcher leaves early — a disconnected client, a page navigated away from.
|
|
188
|
+
for await (const event of watch(runId, request.signal)) {
|
|
170
189
|
render(event);
|
|
171
190
|
}
|
|
172
191
|
```
|
|
173
192
|
|
|
193
|
+
Pass one for anything that can go away before the run ends. `watch` otherwise finishes only on
|
|
194
|
+
`done`, and returning the generator is not a substitute: waiting for the next event it is
|
|
195
|
+
suspended at an `await` rather than at a `yield`, so a `return()` queues behind a promise only
|
|
196
|
+
that event can settle. A watcher stuck there also holds its run's backlog past every deadline,
|
|
197
|
+
because the sweep skips any stream a listener is on.
|
|
198
|
+
|
|
174
199
|
A watcher that stops keeping up is capped rather than left to grow. Its queue holds the most
|
|
175
|
-
recent 1000 events — trimmed in batches, so it runs a little over that before cutting back —
|
|
200
|
+
recent 1000 events by default — trimmed in batches, so it runs a little over that before cutting back —
|
|
176
201
|
the oldest go, and what is dropped is released where it is dropped, not held until a consumer
|
|
177
202
|
that has already stalled next reads.
|
|
178
203
|
|
|
@@ -184,6 +209,90 @@ repeat, and a client doing what `seq` is documented for would throw away one of
|
|
|
184
209
|
`history` reads what a run has emitted without subscribing, `fold` collapses a token stream into
|
|
185
210
|
blocks for display, and `endRun` drops a finished run's buffer.
|
|
186
211
|
|
|
212
|
+
The four numbers behind that — the backlog cap, the slack it is trimmed in batches of, and the two
|
|
213
|
+
deadlines the sweep reaps on — are defaults rather than decisions. `configureEvents` moves them,
|
|
214
|
+
and returns the whole set as it now stands:
|
|
215
|
+
|
|
216
|
+
```ts
|
|
217
|
+
import { configureEvents } from "@cubicecho/agent-core";
|
|
218
|
+
|
|
219
|
+
// A long-lived server with many concurrent runs, whose tool calls are minutes rather than hours.
|
|
220
|
+
configureEvents({ maxEvents: 200, retainUnendedMs: 5 * 60_000 });
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
It is module-level because the bus is: there is one of each per process, and a run does not carry
|
|
224
|
+
its own. A field left out keeps what it had, and so does one given something that is not a
|
|
225
|
+
positive number — nothing here has a meaningful zero, and a `0` standing in for "no opinion" must
|
|
226
|
+
not turn the backlog off. `resetEvents` (and `resetAll`) puts the defaults back, so one test's cap
|
|
227
|
+
is not the next one's.
|
|
228
|
+
|
|
229
|
+
`expandNames`, `carryOver`, `preselection`, `preselectInput` and `preselectSystem` take their caps
|
|
230
|
+
the same way — as a last argument defaulting to `MAX_PER_LOAD` or `MAX_CARRIED`. A resolution
|
|
231
|
+
carries the cap it was held to, so `loadResult` tells the model the number it was actually
|
|
232
|
+
measured against rather than the module's own.
|
|
233
|
+
|
|
234
|
+
## Hooks
|
|
235
|
+
|
|
236
|
+
A hook is something a host runs at a point in a session — `sessionStart` and `beforeTurn`, whose
|
|
237
|
+
output can reach the request, then `afterTurn`, `beforeCompact`, `sessionEnd` and `sessionDelete`,
|
|
238
|
+
which can only read what happened. What runs is not decided here. `gather` and `notify` take a
|
|
239
|
+
`HookRunner`, and `@cubicecho/agent-mcp-pool`'s `runHooks` is one as it stands: its types are the
|
|
240
|
+
shapes restated here, so its outcomes pass straight through without either package importing the
|
|
241
|
+
other.
|
|
242
|
+
|
|
243
|
+
```ts
|
|
244
|
+
import {
|
|
245
|
+
emit,
|
|
246
|
+
gather,
|
|
247
|
+
type HookRunner,
|
|
248
|
+
notify,
|
|
249
|
+
turnIndex,
|
|
250
|
+
turnMessages,
|
|
251
|
+
withContext,
|
|
252
|
+
} from "@cubicecho/agent-core";
|
|
253
|
+
|
|
254
|
+
const run: HookRunner = (event, context, { signal }) =>
|
|
255
|
+
pool.runHooks(event, context, { signal, servers: agent.mcpServers });
|
|
256
|
+
|
|
257
|
+
const context = { session: { id }, host: "my-host", prompt, turn: { index: turnIndex(messages) } };
|
|
258
|
+
const gathered = await gather(
|
|
259
|
+
run,
|
|
260
|
+
messages.length === 0 ? ["sessionStart", "beforeTurn"] : ["beforeTurn"],
|
|
261
|
+
context,
|
|
262
|
+
{ signal, onNote: (note) => emit(runId, { kind: "notice", name: note.hookId, text: note.error ?? note.text }) },
|
|
263
|
+
);
|
|
264
|
+
messages.push({ role: "user", content: prompt });
|
|
265
|
+
const request = withContext(messages, messages.length - 1, gathered.context);
|
|
266
|
+
// ... the turn ...
|
|
267
|
+
void notify(run, "afterTurn", { ...context, reply, turn: { ...context.turn, messages: turnMessages(id, messages, turnStart) } });
|
|
268
|
+
```
|
|
269
|
+
|
|
270
|
+
The context goes on this turn's question and never into the system prompt — a prompt that changed
|
|
271
|
+
every turn would miss the prompt cache every turn — and `withContext` returns a new array, so a
|
|
272
|
+
host that stores what the user typed never stores the context as something they said. Every
|
|
273
|
+
injecting hook shares `HOOK_CONTEXT_TOKENS` (2000) by default, each held to its own `maxTokens`
|
|
274
|
+
inside that, so a generous hook cannot crowd out the conversation it was meant to inform.
|
|
275
|
+
|
|
276
|
+
The budget is the caller's to move, at either level. `configureHooks` sets it for the process, and
|
|
277
|
+
`maxTokens` on `gather` (or the second argument to `assembleContext`) sets it for one request and
|
|
278
|
+
wins over it — the one to reach for when the budget follows the model, since a 128k window can
|
|
279
|
+
afford more recall than an 8k one:
|
|
280
|
+
|
|
281
|
+
```ts
|
|
282
|
+
import { configureHooks, gather } from "@cubicecho/agent-core";
|
|
283
|
+
|
|
284
|
+
configureHooks({ contextTokens: 4000 });
|
|
285
|
+
const gathered = await gather(run, ["beforeTurn"], context, { maxTokens: contextLimit / 20 });
|
|
286
|
+
```
|
|
287
|
+
|
|
288
|
+
Either one given something that is not a number above zero keeps what was there, as
|
|
289
|
+
`configureEvents` does, so a `0` threaded through for "no opinion" does not switch recall off.
|
|
290
|
+
`resetHooks` (and `resetAll`) puts the default back.
|
|
291
|
+
|
|
292
|
+
Neither function rejects. A hook failing is an outcome, and a runner that throws outright is
|
|
293
|
+
noted once for its event and costs only that event's context. `notify` takes no signal: a reader
|
|
294
|
+
who leaves once the turn is answered has not asked for it not to be remembered.
|
|
295
|
+
|
|
187
296
|
## The config seam
|
|
188
297
|
|
|
189
298
|
Nothing here imports a config type from a consumer, and no function asks for a whole
|
|
@@ -205,6 +314,32 @@ The seam is not finished. `timeoutMs` narrows to the one field it reads, but `ge
|
|
|
205
314
|
asks for the whole of `Endpoint`. `requestTimeoutSeconds` became optional in v2, so a consumer
|
|
206
315
|
with no timeout to give now leaves it out rather than inventing a `0`.
|
|
207
316
|
|
|
317
|
+
## What is kept for the life of the process
|
|
318
|
+
|
|
319
|
+
Four caches outlive any one run: the `OpenAI` clients, the model listings, the latched
|
|
320
|
+
capabilities, and `side-task`'s no-thinking hints. All four are module-level and keyed on the same
|
|
321
|
+
notion of an endpoint — its base URL and its API key — and the clients' key carries the request
|
|
322
|
+
timeout as well, since that changes how a request is sent.
|
|
323
|
+
|
|
324
|
+
**They are keyed per deployment, not per request.** What belongs in them is an endpoint an
|
|
325
|
+
operator configured: a settings row, an agent definition, an environment variable. Everything
|
|
326
|
+
bounding them assumes that, and a consumer that mints an API key per *user* breaks the assumption
|
|
327
|
+
— each tenant gets its own client, its own connection pool and its own latches, held until
|
|
328
|
+
`resetAll`.
|
|
329
|
+
|
|
330
|
+
That is bounded rather than unbounded: the client pool is a 32-entry LRU, and an evicted client
|
|
331
|
+
costs its connection pool and nothing else, since the next request through that endpoint builds
|
|
332
|
+
another. But churning connection pools is not sharing them, and at that point a client of your
|
|
333
|
+
own, built and held per tenant, is the better answer than this.
|
|
334
|
+
|
|
335
|
+
The listings cache also remembers, for half a minute, that an endpoint did not name a model — the
|
|
336
|
+
case where a configured name never matches anything the server serves (a llama.cpp `-a` alias, an
|
|
337
|
+
OpenRouter `:free` suffix, a typo) would otherwise fetch the listing on every call and answer the
|
|
338
|
+
same zero each time. Half a minute later it asks again, so a model pulled onto a box that has been
|
|
339
|
+
up a week is still picked up without a restart.
|
|
340
|
+
|
|
341
|
+
`resetAll` drops all four, and `reset.ts` names each seam separately for a test that wants one.
|
|
342
|
+
|
|
208
343
|
## Where the merged behaviour came from
|
|
209
344
|
|
|
210
345
|
- `schema-compat` — `kanban_server`/`task_server`'s version, which strips **every** sibling of a
|
package/dist/capabilities.d.ts
CHANGED
|
@@ -70,10 +70,14 @@ export interface ModelCapabilities {
|
|
|
70
70
|
* What this endpoint is known not to support. The same object every time, so what `negotiate`
|
|
71
71
|
* latches off stays off.
|
|
72
72
|
*
|
|
73
|
-
* @param baseUrl
|
|
74
|
-
*
|
|
73
|
+
* @param baseUrl Where the endpoint is. The two flags on it are per-server; what is per-model
|
|
74
|
+
* hangs off `models`, which `modelCapabilitiesFor` reads.
|
|
75
|
+
* @param apiKey The rest of the endpoint's identity. Optional, because it changes nothing for
|
|
76
|
+
* the ordinary case of one key per base URL and a caller with one need not thread it through;
|
|
77
|
+
* absent reads as `NO_KEY`, exactly as it does in `getClient`, so an endpoint with no key and
|
|
78
|
+
* one that passes `undefined` share an entry rather than holding two.
|
|
75
79
|
*/
|
|
76
|
-
export declare function capabilitiesFor(baseUrl: string): Capabilities;
|
|
80
|
+
export declare function capabilitiesFor(baseUrl: string, apiKey?: string): Capabilities;
|
|
77
81
|
/**
|
|
78
82
|
* What this model on this endpoint is known not to support. The same object every time, so what
|
|
79
83
|
* `negotiate` latches off stays off.
|
package/dist/capabilities.js
CHANGED
|
@@ -1,29 +1,42 @@
|
|
|
1
|
+
import { endpointKey } from "./client.js";
|
|
1
2
|
import { errorMessage } from "./errors.js";
|
|
2
3
|
import { isGrammarError } from "./schema-compat.js";
|
|
3
4
|
/**
|
|
4
5
|
* What each endpoint cannot do, remembered for the life of the process.
|
|
5
6
|
*
|
|
6
|
-
* Keyed by
|
|
7
|
-
* this one. A llama.cpp box that cannot compile a grammar and a cloud API that can are
|
|
8
|
-
* reachable from one settings row over its lifetime — an operator retargets it from Ollama
|
|
9
|
-
* afternoon to OpenAI this evening — and the first one's refusal must not quietly strip
|
|
7
|
+
* Keyed by `endpointKey`, because these are facts about the server on the other end and not
|
|
8
|
+
* about this one. A llama.cpp box that cannot compile a grammar and a cloud API that can are
|
|
9
|
+
* both reachable from one settings row over its lifetime — an operator retargets it from Ollama
|
|
10
|
+
* this afternoon to OpenAI this evening — and the first one's refusal must not quietly strip
|
|
10
11
|
* pattern/format from the second one's requests, or silently cost it its token counts, for the
|
|
11
12
|
* rest of the process. Bounded by the number of endpoints ever configured, which is a settings
|
|
12
13
|
* row's worth.
|
|
14
|
+
*
|
|
15
|
+
* The API key is part of that identity, the same as it is for the client pool and the model
|
|
16
|
+
* listings. A router — LiteLLM, OpenRouter, a gateway with several boxes behind it — is free to
|
|
17
|
+
* send two keys to two different backends, and then what one of them refused is not a fact about
|
|
18
|
+
* the other. Keyed on the URL alone the first caller through the gateway latched for everyone
|
|
19
|
+
* behind it, including the `models` map underneath, which is the level where two keys through
|
|
20
|
+
* one host are most likely to differ at all.
|
|
13
21
|
*/
|
|
14
22
|
const capabilities = new Map();
|
|
15
23
|
/**
|
|
16
24
|
* What this endpoint is known not to support. The same object every time, so what `negotiate`
|
|
17
25
|
* latches off stays off.
|
|
18
26
|
*
|
|
19
|
-
* @param baseUrl
|
|
20
|
-
*
|
|
27
|
+
* @param baseUrl Where the endpoint is. The two flags on it are per-server; what is per-model
|
|
28
|
+
* hangs off `models`, which `modelCapabilitiesFor` reads.
|
|
29
|
+
* @param apiKey The rest of the endpoint's identity. Optional, because it changes nothing for
|
|
30
|
+
* the ordinary case of one key per base URL and a caller with one need not thread it through;
|
|
31
|
+
* absent reads as `NO_KEY`, exactly as it does in `getClient`, so an endpoint with no key and
|
|
32
|
+
* one that passes `undefined` share an entry rather than holding two.
|
|
21
33
|
*/
|
|
22
|
-
export function capabilitiesFor(baseUrl) {
|
|
23
|
-
|
|
34
|
+
export function capabilitiesFor(baseUrl, apiKey) {
|
|
35
|
+
const key = endpointKey({ baseUrl, apiKey });
|
|
36
|
+
let known = capabilities.get(key);
|
|
24
37
|
if (!known) {
|
|
25
38
|
known = { strictSchemas: true, usageInStream: true, models: new Map() };
|
|
26
|
-
capabilities.set(
|
|
39
|
+
capabilities.set(key, known);
|
|
27
40
|
}
|
|
28
41
|
return known;
|
|
29
42
|
}
|
package/dist/client.d.ts
CHANGED
|
@@ -18,6 +18,13 @@ export declare const timeoutMs: (config: Pick<Endpoint, "requestTimeoutSeconds">
|
|
|
18
18
|
* Pooled on the three fields that change how a request is sent, so agents sharing a server share
|
|
19
19
|
* a connection and agents on different servers never share a client.
|
|
20
20
|
*
|
|
21
|
+
* The pool is per *deployment*, not per request: what belongs in this map is an endpoint an
|
|
22
|
+
* operator configured, and everything cached in this package is bounded on that reading. A
|
|
23
|
+
* consumer that mints an API key per user still gets a working client, but it is churning
|
|
24
|
+
* connection pools rather than sharing them and holding every one of them — `MAX_CLIENTS` keeps
|
|
25
|
+
* that from being unbounded, and it is the point at which a client of your own, built and held
|
|
26
|
+
* per tenant, is the better answer than this.
|
|
27
|
+
*
|
|
21
28
|
* @param config Where to send requests and how long to wait. An absent `apiKey` becomes `NO_KEY`.
|
|
22
29
|
*/
|
|
23
30
|
export declare function getClient(config: Endpoint): OpenAI;
|
|
@@ -26,6 +33,27 @@ export interface ModelInfo {
|
|
|
26
33
|
id: string;
|
|
27
34
|
contextLength: number;
|
|
28
35
|
}
|
|
36
|
+
/**
|
|
37
|
+
* What counts as one endpoint, everywhere in this package that has to remember something about
|
|
38
|
+
* one — this file's listings, `capabilities`, and the no-thinking hints in `side-task`.
|
|
39
|
+
*
|
|
40
|
+
* The URL and the key together, because the key is part of what is on the other end rather than
|
|
41
|
+
* only how it is paid for: a router is free to send two keys to two different backends, and then
|
|
42
|
+
* what one of them refused is not a fact about the other. Absent reads as `NO_KEY`, so an
|
|
43
|
+
* endpoint with no key and one that passes `undefined` are one entry rather than two.
|
|
44
|
+
*
|
|
45
|
+
* Stringified rather than joined on a separator, for the reason `getClient` gives: no character
|
|
46
|
+
* is impossible in a URL or a key, and two endpoints must never collide on one entry.
|
|
47
|
+
*
|
|
48
|
+
* @param config Read for `baseUrl` and `apiKey` alone, and the key is optional here where
|
|
49
|
+
* `Endpoint` requires it — `capabilitiesFor` is handed a URL and maybe a key rather than a whole
|
|
50
|
+
* config, and absent and empty already mean the same thing. The timeout is deliberately not in
|
|
51
|
+
* it; see `listings`.
|
|
52
|
+
*/
|
|
53
|
+
export declare const endpointKey: (config: {
|
|
54
|
+
baseUrl: string;
|
|
55
|
+
apiKey?: string;
|
|
56
|
+
}) => string;
|
|
29
57
|
/**
|
|
30
58
|
* Asks an endpoint what it serves, and remembers the answer.
|
|
31
59
|
*
|
package/dist/client.js
CHANGED
|
@@ -26,14 +26,39 @@ export const timeoutMs = (config) => {
|
|
|
26
26
|
* `maxRetries: 0` turns the SDK's own retrying off. Streaming is what this is for, and a stream
|
|
27
27
|
* that has already emitted tokens must not be replayed from the top — the caller knows whether
|
|
28
28
|
* anything has been produced yet and the SDK does not. See `retry.ts`.
|
|
29
|
+
*
|
|
30
|
+
* Bounded, because a cache is only a cache while its keys are. Insertion order is the whole of
|
|
31
|
+
* the eviction: a `Map` iterates oldest-first, and `getClient` re-inserts on a hit.
|
|
29
32
|
*/
|
|
30
33
|
const clients = new Map();
|
|
34
|
+
/**
|
|
35
|
+
* How many endpoints' clients are kept at once.
|
|
36
|
+
*
|
|
37
|
+
* This is a backstop rather than a design. The key includes the API key, and every argument in
|
|
38
|
+
* this file for what bounds these caches is "a settings row's worth" — true of the deployments
|
|
39
|
+
* this was written for, and false the moment a consumer mints a key per *user*, where the map
|
|
40
|
+
* grows one client and one connection pool per tenant for the life of the process with
|
|
41
|
+
* `resetClients` as the only release.
|
|
42
|
+
*
|
|
43
|
+
* Thirty-two is far more endpoints than a settings table holds, so a deployment-shaped consumer
|
|
44
|
+
* never reaches it, and an evicted client costs its connection pool and nothing else — the next
|
|
45
|
+
* request through that endpoint builds another. See `getClient` for the constraint stated as a
|
|
46
|
+
* constraint.
|
|
47
|
+
*/
|
|
48
|
+
const MAX_CLIENTS = 32;
|
|
31
49
|
/**
|
|
32
50
|
* The client for an endpoint, built once and kept.
|
|
33
51
|
*
|
|
34
52
|
* Pooled on the three fields that change how a request is sent, so agents sharing a server share
|
|
35
53
|
* a connection and agents on different servers never share a client.
|
|
36
54
|
*
|
|
55
|
+
* The pool is per *deployment*, not per request: what belongs in this map is an endpoint an
|
|
56
|
+
* operator configured, and everything cached in this package is bounded on that reading. A
|
|
57
|
+
* consumer that mints an API key per user still gets a working client, but it is churning
|
|
58
|
+
* connection pools rather than sharing them and holding every one of them — `MAX_CLIENTS` keeps
|
|
59
|
+
* that from being unbounded, and it is the point at which a client of your own, built and held
|
|
60
|
+
* per tenant, is the better answer than this.
|
|
61
|
+
*
|
|
37
62
|
* @param config Where to send requests and how long to wait. An absent `apiKey` becomes `NO_KEY`.
|
|
38
63
|
*/
|
|
39
64
|
export function getClient(config) {
|
|
@@ -43,10 +68,23 @@ export function getClient(config) {
|
|
|
43
68
|
// key, and two different endpoints must never resolve to the same cached client.
|
|
44
69
|
const key = JSON.stringify([config.baseUrl, apiKey, timeout]);
|
|
45
70
|
const existing = clients.get(key);
|
|
46
|
-
if (existing)
|
|
71
|
+
if (existing) {
|
|
72
|
+
// Re-inserted so it counts as the youngest. A `Map` keeps insertion order and hands the
|
|
73
|
+
// oldest key over first, which is the whole of the LRU below.
|
|
74
|
+
clients.delete(key);
|
|
75
|
+
clients.set(key, existing);
|
|
47
76
|
return existing;
|
|
77
|
+
}
|
|
48
78
|
const client = new OpenAI({ baseURL: config.baseUrl, apiKey, timeout, maxRetries: 0 });
|
|
49
79
|
clients.set(key, client);
|
|
80
|
+
// Nothing is closed on the way out. The SDK holds no handle a caller can release, and an
|
|
81
|
+
// evicted client is garbage once whatever request is still in flight on it has finished —
|
|
82
|
+
// dropping the reference is the whole of the eviction.
|
|
83
|
+
for (const oldest of clients.keys()) {
|
|
84
|
+
if (clients.size <= MAX_CLIENTS)
|
|
85
|
+
break;
|
|
86
|
+
clients.delete(oldest);
|
|
87
|
+
}
|
|
50
88
|
return client;
|
|
51
89
|
}
|
|
52
90
|
/**
|
|
@@ -76,7 +114,7 @@ function contextLengthOf(model) {
|
|
|
76
114
|
/**
|
|
77
115
|
* The last listing from each endpoint, so a run can size its window without a round trip.
|
|
78
116
|
*
|
|
79
|
-
* Keyed by
|
|
117
|
+
* Keyed by `endpointKey`, because two endpoints are two different sets of models and one of
|
|
80
118
|
* them having answered says nothing about the other — and because a key can be the difference
|
|
81
119
|
* between what a router will show one caller and another.
|
|
82
120
|
*
|
|
@@ -88,7 +126,41 @@ function contextLengthOf(model) {
|
|
|
88
126
|
* whatever was there rather than emptying it.
|
|
89
127
|
*/
|
|
90
128
|
const listings = new Map();
|
|
91
|
-
|
|
129
|
+
/**
|
|
130
|
+
* When an endpoint was last asked about a model it did not name, keyed on the two together.
|
|
131
|
+
*
|
|
132
|
+
* `contextLimitFor` asks again whenever the listing does not hold the model, which is right for
|
|
133
|
+
* a model that arrives late and wrong for one that is never coming. The second is the ordinary
|
|
134
|
+
* case rather than the exotic one — a llama.cpp box served under a `-a` alias that does not
|
|
135
|
+
* match the configured name, an OpenRouter `:free` suffix, a typo in a settings row — and there
|
|
136
|
+
* every call fetches the listing, re-reads it, finds the same absence and answers the same zero.
|
|
137
|
+
*
|
|
138
|
+
* Remembering the miss for a moment answers both: within `LISTING_MISS_MS` nobody is asked, and
|
|
139
|
+
* after it the endpoint is asked again, so an `ollama pull` on a box that has been up a week is
|
|
140
|
+
* picked up within the minute instead of at the next restart. Bounded by the (endpoint, model)
|
|
141
|
+
* pairs actually asked about, which is the bound the listings themselves have.
|
|
142
|
+
*/
|
|
143
|
+
const misses = new Map();
|
|
144
|
+
/** How long a model an endpoint did not list stays unlisted before it is asked about again. */
|
|
145
|
+
const LISTING_MISS_MS = 30_000;
|
|
146
|
+
/**
|
|
147
|
+
* What counts as one endpoint, everywhere in this package that has to remember something about
|
|
148
|
+
* one — this file's listings, `capabilities`, and the no-thinking hints in `side-task`.
|
|
149
|
+
*
|
|
150
|
+
* The URL and the key together, because the key is part of what is on the other end rather than
|
|
151
|
+
* only how it is paid for: a router is free to send two keys to two different backends, and then
|
|
152
|
+
* what one of them refused is not a fact about the other. Absent reads as `NO_KEY`, so an
|
|
153
|
+
* endpoint with no key and one that passes `undefined` are one entry rather than two.
|
|
154
|
+
*
|
|
155
|
+
* Stringified rather than joined on a separator, for the reason `getClient` gives: no character
|
|
156
|
+
* is impossible in a URL or a key, and two endpoints must never collide on one entry.
|
|
157
|
+
*
|
|
158
|
+
* @param config Read for `baseUrl` and `apiKey` alone, and the key is optional here where
|
|
159
|
+
* `Endpoint` requires it — `capabilitiesFor` is handed a URL and maybe a key rather than a whole
|
|
160
|
+
* config, and absent and empty already mean the same thing. The timeout is deliberately not in
|
|
161
|
+
* it; see `listings`.
|
|
162
|
+
*/
|
|
163
|
+
export const endpointKey = (config) => JSON.stringify([config.baseUrl, config.apiKey || NO_KEY]);
|
|
92
164
|
/**
|
|
93
165
|
* Asks an endpoint what it serves, and remembers the answer.
|
|
94
166
|
*
|
|
@@ -122,6 +194,7 @@ export async function contextLimitFor(config, declared = 0) {
|
|
|
122
194
|
if (declared > 0)
|
|
123
195
|
return declared;
|
|
124
196
|
const key = endpointKey(config);
|
|
197
|
+
const listed = () => listings.get(key)?.find((model) => model.id === config.model);
|
|
125
198
|
// The listing is asked for again when it does not name this model, rather than only when
|
|
126
199
|
// there is no listing at all. Models arrive after a process starts — an `ollama pull` on a
|
|
127
200
|
// box that has been up a week, a worker added to a router, a name the operator has only just
|
|
@@ -129,10 +202,14 @@ export async function contextLimitFor(config, declared = 0) {
|
|
|
129
202
|
// of them until a restart. Zero means "nobody knows", so what the operator loses is the
|
|
130
203
|
// context meter and, in a consumer that compacts on it, compaction: the session then runs at
|
|
131
204
|
// the window instead of under it and fails against the endpoint's own refusal.
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
205
|
+
if (!listed()) {
|
|
206
|
+
const missKey = JSON.stringify([key, config.model]);
|
|
207
|
+
const asked = misses.get(missKey);
|
|
208
|
+
// Asked again, but not on every call. A model that is never coming answers the same zero
|
|
209
|
+
// however often the endpoint is asked, and a caller sizing a window per turn pays a round
|
|
210
|
+
// trip for each of them; `LISTING_MISS_MS` is how long that answer is allowed to stand.
|
|
211
|
+
if (asked !== undefined && Date.now() - asked < LISTING_MISS_MS)
|
|
212
|
+
return 0;
|
|
136
213
|
// A failure is not remembered: an endpoint that was down when the last run started is not
|
|
137
214
|
// an endpoint with no models, and a window nobody could ask about is not a failed run.
|
|
138
215
|
try {
|
|
@@ -141,12 +218,20 @@ export async function contextLimitFor(config, declared = 0) {
|
|
|
141
218
|
catch {
|
|
142
219
|
return 0;
|
|
143
220
|
}
|
|
221
|
+
// Recorded where the endpoint was actually asked, and only there. Stamping it on the calls
|
|
222
|
+
// that skipped the request would push the deadline out ahead of any caller polling faster
|
|
223
|
+
// than the interval, which is a memory that never expires rather than one that expires in
|
|
224
|
+
// half a minute.
|
|
225
|
+
if (listed())
|
|
226
|
+
misses.delete(missKey);
|
|
227
|
+
else
|
|
228
|
+
misses.set(missKey, Date.now());
|
|
144
229
|
}
|
|
145
|
-
|
|
146
|
-
return listed.find((model) => model.id === config.model)?.contextLength ?? 0;
|
|
230
|
+
return listed()?.contextLength ?? 0;
|
|
147
231
|
}
|
|
148
232
|
/** Forgets every cached client and listing. For tests, and for a settings change under test. */
|
|
149
233
|
export function resetClients() {
|
|
150
234
|
clients.clear();
|
|
151
235
|
listings.clear();
|
|
236
|
+
misses.clear();
|
|
152
237
|
}
|
package/dist/events.d.ts
CHANGED
|
@@ -9,6 +9,60 @@
|
|
|
9
9
|
* In memory on purpose: it is debugging output, worth nothing once the run has finished and its
|
|
10
10
|
* outcome is in the database. Nothing here survives a restart, and nothing here is the record.
|
|
11
11
|
*/
|
|
12
|
+
/** What the bus keeps and for how long. Every field optional; see `configureEvents`. */
|
|
13
|
+
export interface EventBusOptions {
|
|
14
|
+
/** How many events one run keeps for a watcher that joins late. A chatty run loses its oldest. */
|
|
15
|
+
maxEvents?: number;
|
|
16
|
+
/**
|
|
17
|
+
* How far past the cap the backlog is allowed to run before it is trimmed.
|
|
18
|
+
*
|
|
19
|
+
* Dropping the oldest event on every push means shifting a thousand-element array tens of
|
|
20
|
+
* thousands of times over a reasoning run — the one thing in here that would ever show up in a
|
|
21
|
+
* profile. Trimming in batches makes it a few dozen splices instead, at the cost of the backlog
|
|
22
|
+
* sometimes being a little longer than the cap, which nothing depends on. The one field here
|
|
23
|
+
* that is an implementation detail rather than a policy: raising it trades memory for fewer
|
|
24
|
+
* splices, and there is no reason to lower it.
|
|
25
|
+
*/
|
|
26
|
+
trimSlack?: number;
|
|
27
|
+
/** How long a finished run stays readable, for a watcher that arrives just after the end. */
|
|
28
|
+
retainMs?: number;
|
|
29
|
+
/**
|
|
30
|
+
* The same for a run that has not said `done`, which is a far more dangerous thing to drop.
|
|
31
|
+
*
|
|
32
|
+
* A finished run has nothing more to say, so forgetting it a minute later costs a late watcher
|
|
33
|
+
* a backlog and nothing else. An unfinished one is still writing: `touched` only moves on
|
|
34
|
+
* `emit`, so a live run that spends a minute inside one slow tool call looked exactly like an
|
|
35
|
+
* abandoned one and was reaped out from under itself. What made that more than a lost backlog
|
|
36
|
+
* is that the next `emit` builds a fresh stream with `seq` back at zero — and `seq` is the
|
|
37
|
+
* field `RunEvent` documents for ordering and de-duplication, so a client that reconnects
|
|
38
|
+
* across the gap discards the new events as ones it has already seen.
|
|
39
|
+
*
|
|
40
|
+
* The sweep still has to reap them, because a run killed by a signal or simply forgotten
|
|
41
|
+
* reaches no `done` either. This is the backstop for a caller that never says so; `endRun` is
|
|
42
|
+
* for one that knows. Anything shorter than the longest a tool call may take is a live run
|
|
43
|
+
* reaped out from under itself.
|
|
44
|
+
*/
|
|
45
|
+
retainUnendedMs?: number;
|
|
46
|
+
}
|
|
47
|
+
/**
|
|
48
|
+
* Changes what the bus keeps, for a process whose runs are not shaped like the ones these
|
|
49
|
+
* defaults were chosen for.
|
|
50
|
+
*
|
|
51
|
+
* The bus is one module-level thing rather than an object a caller holds, so this is too: it is
|
|
52
|
+
* a deployment's setting, said once at startup, and not something to move around under a run.
|
|
53
|
+
* What it costs is memory against how much of a run a late or slow watcher can still read —
|
|
54
|
+
* a server with hundreds of concurrent runs wants a smaller backlog, and one whose tool calls
|
|
55
|
+
* take an hour wants a longer `retainUnendedMs` than the thirty minutes assumed here.
|
|
56
|
+
*
|
|
57
|
+
* Changes apply from the next event and the next sweep. Nothing already buffered is trimmed to
|
|
58
|
+
* a cap that has just come down, because the trim happens on push; the backlog settles to the
|
|
59
|
+
* new number as the run goes on.
|
|
60
|
+
*
|
|
61
|
+
* @param options The bounds to change. A field left out — or given anything that is not a
|
|
62
|
+
* number above zero — keeps what it has, so a partial or a half-built config narrows nothing.
|
|
63
|
+
* @returns Everything in force afterwards, including what this call did not change.
|
|
64
|
+
*/
|
|
65
|
+
export declare function configureEvents(options?: EventBusOptions): Required<EventBusOptions>;
|
|
12
66
|
/** Which kind of thing happened, and what `text`, `name`, `ok` and `usage` carry for it. */
|
|
13
67
|
export type RunEventKind =
|
|
14
68
|
/** A step of a caller's own flow began. `name` is the step, `text` its kind. */
|
|
@@ -107,11 +161,22 @@ export declare function emit(runId: string, input: RunEventInput): RunEvent;
|
|
|
107
161
|
* inside the retention window — reads the same story as one that was there from the start.
|
|
108
162
|
*
|
|
109
163
|
* @param runId The run to follow. One that has not started yet is waited on, not refused.
|
|
164
|
+
* @param signal Stops following. The only other way out is the run's own `done`, and a watcher
|
|
165
|
+
* with no way out is a leak rather than a lost backlog: the sweep below skips any stream a
|
|
166
|
+
* listener is on, so a run that dies without `done` pins its backlog for the life of the process.
|
|
167
|
+
* Returning the generator is not that way out — parked on the promise at the foot of this
|
|
168
|
+
* function it is suspended at an `await` rather than at a `yield`, and a `return()` there is
|
|
169
|
+
* queued behind a promise only the next event can settle. An abort resolves that promise itself.
|
|
110
170
|
*/
|
|
111
|
-
export declare function watch(runId: string): AsyncGenerator<RunEvent>;
|
|
171
|
+
export declare function watch(runId: string, signal?: AbortSignal): AsyncGenerator<RunEvent>;
|
|
112
172
|
/**
|
|
113
173
|
* The backlog alone, for a caller that wants a snapshot rather than a subscription.
|
|
114
174
|
*
|
|
175
|
+
* The array is a copy; the events in it are not. They are the same objects the bus holds and
|
|
176
|
+
* every watcher was handed, so writing to one rewrites the run for everybody — which is what
|
|
177
|
+
* `fold` copies to avoid, and this is the other half of the same warning. Read them, or copy
|
|
178
|
+
* what you mean to change.
|
|
179
|
+
*
|
|
115
180
|
* @param runId The run to read. An unknown or already-swept run gives an empty array.
|
|
116
181
|
*/
|
|
117
182
|
export declare const history: (runId: string) => RunEvent[];
|
|
@@ -121,6 +186,11 @@ export declare const history: (runId: string) => RunEvent[];
|
|
|
121
186
|
* Named for what it forgets rather than bare `reset`, which sat in a consumer's imports beside
|
|
122
187
|
* `resetAll`, `resetClients`, `resetCapabilities` and `resetHints` saying nothing about which
|
|
123
188
|
* of the five it was — `reset.ts` had to alias it on the way in to stay readable.
|
|
189
|
+
*
|
|
190
|
+
* `configureEvents` is undone too, for the reason `reset.ts` gives about latches: a bus left
|
|
191
|
+
* holding a cap one test set is the same order-dependent suite, passing where that test ran
|
|
192
|
+
* first and failing where it did not. A consumer that configures at startup and resets at
|
|
193
|
+
* teardown configures again, which is the same line it already wrote once.
|
|
124
194
|
*/
|
|
125
195
|
export declare const resetEvents: () => void;
|
|
126
196
|
/**
|