@cubicecho/agent-core 4.10.0 → 4.12.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +76 -5
- package/dist/core/config.d.ts +34 -1
- package/dist/core/config.js +45 -0
- package/dist/endpoint/client.d.ts +74 -6
- package/dist/endpoint/client.js +181 -15
- package/dist/endpoint/slots.js +3 -2
- package/dist/index.d.ts +3 -3
- package/dist/index.js +2 -2
- package/dist/run/agent-loop.d.ts +37 -8
- package/dist/run/agent-loop.js +30 -10
- package/dist/run/events.d.ts +10 -2
- package/dist/run/events.js +20 -6
- package/dist/run/run-calls.d.ts +10 -2
- package/dist/run/run-calls.js +39 -8
- package/dist/wire/tool-calls.d.ts +21 -0
- package/dist/wire/wire.d.ts +1 -0
- package/dist/wire/wire.js +1 -0
- package/llms.txt +7 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -30,11 +30,11 @@ only, Node >=22.
|
|
|
30
30
|
| `side-task` | One-shot calls that support a run without being one — small prompt, short answer, no tools, never worth failing the run over. `askJson` holds the answer to a schema where the server can, and `taskCall` turns the settings an agent spec gives a task into the call that honours them. |
|
|
31
31
|
| `hooks` | The host's side of lifecycle hooks: `gather` before a request and `notify` after, the shared context budget, `withContext` to put what they add on the turn's question, `untrusted` to fence text nobody vouched for, and `turnMessages` to hand them a transcript. Running a hook is a runner the caller passes. |
|
|
32
32
|
| `events` | The in-memory bus a watcher reads while a run happens: `emit`, `watch`, `history`, `fold`, and `runMetrics` for what a run cost. A watcher's backlog is capped and reports its own gaps. |
|
|
33
|
-
| `client` | A pooled `OpenAI` client per endpoint, plus the context window: the served one where a local server says, the listed one otherwise, and their caches. And `resolveApiKey`, for a caller deciding which key an endpoint gets. |
|
|
33
|
+
| `client` | A pooled `OpenAI` client per endpoint, plus the context window: the served one where a local server says, the listed one otherwise, and their caches. And `resolveApiKey`, for a caller deciding which key an endpoint gets, and `probeEndpoint`, for listing one a form has not saved yet. |
|
|
34
34
|
| `retry` | What to do when a request is lost, refused or too big: `isTransient`, `isModelLoading`, `backoffMs`, `ContextOverflow`, `EndpointSilent`. |
|
|
35
35
|
| `calibration` | How many characters a token is worth on one model, learned from the prompt counts its endpoint reports: `charsPerTokenFor`, `calibrate`. |
|
|
36
36
|
| `continuation` | `continueTurn`: carries on an answer the token ceiling cut off, by prefilling it as a trailing assistant message. |
|
|
37
|
-
| `config` | The structural interfaces every function here asks for. |
|
|
37
|
+
| `config` | The structural interfaces every function here asks for, and `normalizeBaseUrl`, the one rule for when two base URLs are one endpoint. |
|
|
38
38
|
| `run-turn` | `runTurn`: one turn with the retry loop around the negotiation around the stream. The whole loop, for a caller that wants it rather than its parts. Sizes the request against an opt-in `contextLimit`. |
|
|
39
39
|
| `agent-loop` | `runAgentLoop`: the loop above a turn — `runTurn` per step, the tools between, `load_tools` and preselection handled, until the model stops asking. |
|
|
40
40
|
| `request-body` | `buildBody`: the one place a streamed request's body is decided from a config and what the endpoint and the model have refused. |
|
|
@@ -306,7 +306,7 @@ const turn = await runTurn(client, supports, build, {
|
|
|
306
306
|
|
|
307
307
|
`contextLimitFor` answers the operator's number when there is one. Otherwise it reads the
|
|
308
308
|
`/v1/models` listing first. A window at the top of the model's entry (`context_length`,
|
|
309
|
-
`max_context_window`, `max_model_len`, `context_window` or `n_ctx`: vLLM's `max_model_len`,
|
|
309
|
+
`max_context_length`, `max_context_window`, `max_model_len`, `context_window` or `n_ctx`: vLLM's `max_model_len`,
|
|
310
310
|
OpenRouter's `context_length`) is taken as the one being served, and nothing else is asked.
|
|
311
311
|
|
|
312
312
|
Where the listing does not settle it, the server's own API is asked (`servedWindow`): llama.cpp's
|
|
@@ -316,7 +316,10 @@ listed with only `meta.n_ctx_train`, which is llama.cpp. The trained window diff
|
|
|
316
316
|
one in the case the guard exists for, a 256k model started at `-c 16384`, so it is used only when
|
|
317
317
|
neither native route gives a number. A server with neither route is latched and not asked again.
|
|
318
318
|
`/api/v0/models` is also dropped on its own: for the life of the process once it answers 404, 405
|
|
319
|
-
or 501, and for `listingMissMs` after any other error.
|
|
319
|
+
or 501, and for `listingMissMs` after any other error. A 200 that is not JSON counts as a missing
|
|
320
|
+
route: it is what a proxy with a web app of its own (llm-router) answers on every path it does not
|
|
321
|
+
route. Neither route is sent `NO_KEY` as a bearer; an endpoint with no key is asked with no
|
|
322
|
+
`Authorization` header at all.
|
|
320
323
|
|
|
321
324
|
A host that sizes the window before each run passes the run's signal as the third argument, so a
|
|
322
325
|
stopped run does not wait on the lookup:
|
|
@@ -335,6 +338,33 @@ nothing would otherwise hold the lookup for that long, or for ever where there i
|
|
|
335
338
|
bound `contextLimitFor` answers zero, as it does for a server it could not reach, and
|
|
336
339
|
`listModels` rejects with a `TimeoutError`.
|
|
337
340
|
|
|
341
|
+
### Listing an endpoint that is not saved yet
|
|
342
|
+
|
|
343
|
+
`listModels` is for an endpoint that is configured: it pools a client per URL and remembers what
|
|
344
|
+
came back. A settings form asking at each keystroke would leave a client behind for every
|
|
345
|
+
half-typed URL, so it asks `probeEndpoint` instead, which makes one plain request and remembers
|
|
346
|
+
nothing:
|
|
347
|
+
|
|
348
|
+
```ts
|
|
349
|
+
import { EndpointProbeError, ProbeFailure, probeEndpoint } from "@cubicecho/agent-core";
|
|
350
|
+
|
|
351
|
+
try {
|
|
352
|
+
const models = await probeEndpoint({ baseUrl: typed, apiKey: typedKey }, { signal });
|
|
353
|
+
} catch (error) {
|
|
354
|
+
if (error instanceof EndpointProbeError) {
|
|
355
|
+
// error.kind: invalid-url, unreachable, timeout, unauthorized, redirected, not-openai, failed.
|
|
356
|
+
// error.message is a sentence naming the endpoint by its origin, fit to show under the field.
|
|
357
|
+
return { problem: error.message };
|
|
358
|
+
}
|
|
359
|
+
throw error; // the signal's own reason
|
|
360
|
+
}
|
|
361
|
+
```
|
|
362
|
+
|
|
363
|
+
It refuses a URL `normalizeBaseUrl` refuses before asking anybody, follows no redirect (so a typed
|
|
364
|
+
key is never carried on to a host nobody typed), reads both OpenAI's `{ data: [...] }` and
|
|
365
|
+
Ollama's native `{ models: [...] }` (by `name` where an entry has no `id`), and hands back each id
|
|
366
|
+
once, sorted, at most `limit` of them (500 unless told). It is bounded as a lookup is.
|
|
367
|
+
|
|
338
368
|
A top-level listing key is trusted without a second opinion, so on a server that puts a trained
|
|
339
369
|
window there, pass `contextLength`. Ollama reports no window on any route this reads, and truncates
|
|
340
370
|
an over-long prompt rather than refusing it, so on Ollama pass `contextLength` or there is no guard
|
|
@@ -532,6 +562,41 @@ its calls made on the error too, read off with `failedRun`.
|
|
|
532
562
|
A link meant for the model alone — one whose MCP `audience` leaves out `"user"` — is the
|
|
533
563
|
dispatcher's to drop, since that is MCP's rule and this package knows nothing of MCP.
|
|
534
564
|
|
|
565
|
+
### A call that ends the run
|
|
566
|
+
|
|
567
|
+
A tool whose call is the end of the job — handing off, asking a person — says so with `end` on
|
|
568
|
+
its `ToolAnswer`, `true` or `{ reason }`. The loop answers every call of that step, sends no
|
|
569
|
+
further request, and resolves with `ended`; aborting the run's signal from inside `dispatch` was
|
|
570
|
+
the way out before, and that is a failure the host then has to dig the run back out of.
|
|
571
|
+
|
|
572
|
+
```ts
|
|
573
|
+
const result = await runAgentLoop({
|
|
574
|
+
// ...
|
|
575
|
+
dispatch: async ({ name, args }, signal) =>
|
|
576
|
+
name === "ask_person"
|
|
577
|
+
? { text: "Asked. This run ends here.", end: { reason: String(args.question) } }
|
|
578
|
+
: pool.call(name, args, { signal }),
|
|
579
|
+
});
|
|
580
|
+
if (result.ended) {
|
|
581
|
+
// { callId, name: "ask_person", reason }
|
|
582
|
+
const { callId, name, reason = "" } = result.ended;
|
|
583
|
+
emit(runId, { kind: RunEventKind.Done, ok: true, id: callId, name, text: reason });
|
|
584
|
+
}
|
|
585
|
+
```
|
|
586
|
+
|
|
587
|
+
The step's other calls still run, whichever side of the end the model asked for them and however
|
|
588
|
+
`parallel` batches them: they were asked for in the same turn, and a model that says "write the
|
|
589
|
+
file, then hand off" wants the file written in either order. Where two calls end it, `ended` is
|
|
590
|
+
the first in the order asked. Everything else is a finished run's — `usage`, `metrics`,
|
|
591
|
+
`messages`, `artifacts`, the `afterTurn` hooks — with `metrics.outcome` `ended` and a notice
|
|
592
|
+
saying which tool ended it. `turn` is the turn that made the call, so its `content` is whatever
|
|
593
|
+
the model wrote beside its calls, often nothing; the run's answer is what the ending tool took.
|
|
594
|
+
The tool, its definition and its wording stay the host's: this is only the exit.
|
|
595
|
+
|
|
596
|
+
A host's `done` that carries the call — `id`, `name` the tool, `text` the reason — is read by
|
|
597
|
+
`runMetrics` as `ended` too, so the outcome is the same whether it is read off the result or the
|
|
598
|
+
events.
|
|
599
|
+
|
|
535
600
|
Arguments go through `parseToolArguments`, which is lenient where the model's meaning is plain:
|
|
536
601
|
JSON held in a string is opened, and the almost-JSON local models write — single quotes, Python's
|
|
537
602
|
`True` and `None`, bare keys, a trailing comma — is repaired, without touching what is inside a
|
|
@@ -874,6 +939,12 @@ it encodes is the conservative one: an endpoint's own key wins; one that names a
|
|
|
874
939
|
own, different from the settings it inherits from, gets `NO_KEY` rather than the operator's key or
|
|
875
940
|
`$OPENAI_API_KEY`; anything else inherits.
|
|
876
941
|
|
|
942
|
+
"Different" is decided by `normalizeBaseUrl`, the same rule `endpointKey` and every cache here
|
|
943
|
+
compares by: http or https, no credentials, query or fragment, scheme and host in lower case, no
|
|
944
|
+
default port and no trailing slash. A host storing a base URL should store what it returns, and
|
|
945
|
+
refuse a URL it answers `null` for. A URL it refuses still equals itself, trimmed of whitespace
|
|
946
|
+
and trailing slashes, so a row saved before the rule still finds its own key.
|
|
947
|
+
|
|
877
948
|
## Fields this interface cannot spell
|
|
878
949
|
|
|
879
950
|
`ModelParams` has `temperature`, `maxTokens` and `reasoningEffort`. Everything else a model card or
|
|
@@ -1438,7 +1509,7 @@ call returns everything in force. Lowering `maxClients` below the pool's current
|
|
|
1438
1509
|
it at once, least recently used first. `resetClients` (and `resetAll`) puts the defaults back.
|
|
1439
1510
|
|
|
1440
1511
|
A host that keeps per-endpoint state of its own should key it on `endpointKey` — the JSON of the base
|
|
1441
|
-
URL and the key, an absent key read as `NO_KEY` — rather than rebuilding that string, so the two
|
|
1512
|
+
URL as `normalizeBaseUrl` writes it and the key, an absent key read as `NO_KEY` — rather than rebuilding that string, so the two
|
|
1442
1513
|
cannot drift. `endpointKey` holds the key in the clear; `endpointId`, its SHA-256 digest, is the one
|
|
1443
1514
|
that is safe to write down.
|
|
1444
1515
|
|
package/dist/core/config.d.ts
CHANGED
|
@@ -57,11 +57,44 @@ export interface Endpoint {
|
|
|
57
57
|
* not in it, since a different patience is not a different server.
|
|
58
58
|
*/
|
|
59
59
|
export interface EndpointIdentity {
|
|
60
|
-
/**
|
|
60
|
+
/**
|
|
61
|
+
* The base URL as given. Compared as `baseUrlKey` has it, so a trailing slash, a host's case and
|
|
62
|
+
* a default port are the same endpoint, and a path's case and a `/v1` are not.
|
|
63
|
+
*/
|
|
61
64
|
baseUrl: string;
|
|
62
65
|
/** Absent and empty mean the same: no key, which `endpointKey` reads as `NO_KEY`. */
|
|
63
66
|
apiKey?: string;
|
|
64
67
|
}
|
|
68
|
+
/**
|
|
69
|
+
* A base URL worth storing: http or https, no credentials, query or fragment, no trailing slash.
|
|
70
|
+
*
|
|
71
|
+
* @param raw - The URL as typed. Surrounding whitespace is trimmed.
|
|
72
|
+
* @returns The URL as the WHATWG parser writes it, less its trailing slashes — scheme and host in
|
|
73
|
+
* lower case, a default port dropped — or `null` for anything that is not such a URL.
|
|
74
|
+
*
|
|
75
|
+
* @remarks
|
|
76
|
+
* One rule for every host, because "a stored key only goes to its own endpoint" is decided by
|
|
77
|
+
* comparing two of these, and four hosts writing four rules had begun to disagree about it. A key
|
|
78
|
+
* in the URL is a credential nobody meant to store; a query or a fragment is one more thing two
|
|
79
|
+
* copies of the same endpoint could differ in, and the SDK appends its paths after it.
|
|
80
|
+
*
|
|
81
|
+
* A host that stores a base URL stores this, and a form that refuses one refuses on `null`, so what
|
|
82
|
+
* is saved is already in the form every comparison here reads.
|
|
83
|
+
*/
|
|
84
|
+
export declare function normalizeBaseUrl(raw: string): string | null;
|
|
85
|
+
/**
|
|
86
|
+
* A base URL in the form two of them are compared in, valid or not.
|
|
87
|
+
*
|
|
88
|
+
* @param raw - The URL as configured.
|
|
89
|
+
* @returns `normalizeBaseUrl`'s answer where it has one, and otherwise the URL trimmed of
|
|
90
|
+
* whitespace and trailing slashes, which is the rule every comparison here used before.
|
|
91
|
+
*
|
|
92
|
+
* @remarks
|
|
93
|
+
* A URL the normaliser refuses is still somebody's endpoint — an old settings row, a key written
|
|
94
|
+
* into the URL — and it still has to equal itself, so it falls back to the older, looser rule
|
|
95
|
+
* rather than to "unequal to everything".
|
|
96
|
+
*/
|
|
97
|
+
export declare const baseUrlKey: (raw: string) => string;
|
|
65
98
|
/**
|
|
66
99
|
* The request fields `extraBody` may not set, because the loop's request is built around them.
|
|
67
100
|
*
|
package/dist/core/config.js
CHANGED
|
@@ -11,6 +11,51 @@
|
|
|
11
11
|
* it structurally. Nothing imports a config *type* from a consumer, and no consumer has to grow
|
|
12
12
|
* a field it has no use for in order to call `getClient`.
|
|
13
13
|
*/
|
|
14
|
+
/**
|
|
15
|
+
* A base URL worth storing: http or https, no credentials, query or fragment, no trailing slash.
|
|
16
|
+
*
|
|
17
|
+
* @param raw - The URL as typed. Surrounding whitespace is trimmed.
|
|
18
|
+
* @returns The URL as the WHATWG parser writes it, less its trailing slashes — scheme and host in
|
|
19
|
+
* lower case, a default port dropped — or `null` for anything that is not such a URL.
|
|
20
|
+
*
|
|
21
|
+
* @remarks
|
|
22
|
+
* One rule for every host, because "a stored key only goes to its own endpoint" is decided by
|
|
23
|
+
* comparing two of these, and four hosts writing four rules had begun to disagree about it. A key
|
|
24
|
+
* in the URL is a credential nobody meant to store; a query or a fragment is one more thing two
|
|
25
|
+
* copies of the same endpoint could differ in, and the SDK appends its paths after it.
|
|
26
|
+
*
|
|
27
|
+
* A host that stores a base URL stores this, and a form that refuses one refuses on `null`, so what
|
|
28
|
+
* is saved is already in the form every comparison here reads.
|
|
29
|
+
*/
|
|
30
|
+
export function normalizeBaseUrl(raw) {
|
|
31
|
+
let url;
|
|
32
|
+
try {
|
|
33
|
+
url = new URL(raw.trim());
|
|
34
|
+
}
|
|
35
|
+
catch {
|
|
36
|
+
return null;
|
|
37
|
+
}
|
|
38
|
+
if (url.protocol !== 'http:' && url.protocol !== 'https:') {
|
|
39
|
+
return null;
|
|
40
|
+
}
|
|
41
|
+
if (url.username || url.password || url.search || url.hash) {
|
|
42
|
+
return null;
|
|
43
|
+
}
|
|
44
|
+
return url.toString().replace(/\/+$/, '');
|
|
45
|
+
}
|
|
46
|
+
/**
|
|
47
|
+
* A base URL in the form two of them are compared in, valid or not.
|
|
48
|
+
*
|
|
49
|
+
* @param raw - The URL as configured.
|
|
50
|
+
* @returns `normalizeBaseUrl`'s answer where it has one, and otherwise the URL trimmed of
|
|
51
|
+
* whitespace and trailing slashes, which is the rule every comparison here used before.
|
|
52
|
+
*
|
|
53
|
+
* @remarks
|
|
54
|
+
* A URL the normaliser refuses is still somebody's endpoint — an old settings row, a key written
|
|
55
|
+
* into the URL — and it still has to equal itself, so it falls back to the older, looser rule
|
|
56
|
+
* rather than to "unequal to everything".
|
|
57
|
+
*/
|
|
58
|
+
export const baseUrlKey = (raw) => normalizeBaseUrl(raw) ?? raw.trim().replace(/\/+$/, '');
|
|
14
59
|
/**
|
|
15
60
|
* The request fields `extraBody` may not set, because the loop's request is built around them.
|
|
16
61
|
*
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
import OpenAI from 'openai';
|
|
2
|
-
import type
|
|
2
|
+
import { type Endpoint, type EndpointIdentity, type RetryPolicy } from '../core/config.ts';
|
|
3
3
|
/**
|
|
4
4
|
* The SDK insists on a non-empty key even where the server will not look at it. This is what it
|
|
5
5
|
* gets. It also reads as "this endpoint has no key" at a call site, which an empty string does
|
|
@@ -112,13 +112,15 @@ export interface ModelInfo {
|
|
|
112
112
|
* `Endpoint` requires it — `capabilitiesFor` is handed a URL and maybe a key rather than a whole
|
|
113
113
|
* config, and absent and empty already mean the same thing. The timeout is deliberately not in
|
|
114
114
|
* it; see `listings`.
|
|
115
|
-
* @returns The URL and the key as one JSON string, the key in the clear.
|
|
115
|
+
* @returns The URL as `baseUrlKey` writes it and the key, as one JSON string, the key in the clear.
|
|
116
116
|
*
|
|
117
117
|
* @remarks
|
|
118
118
|
* The URL and the key together, because the key is part of what is on the other end rather than
|
|
119
119
|
* only how it is paid for: a router is free to send two keys to two different backends, and then
|
|
120
120
|
* what one of them refused is not a fact about the other. Absent reads as `NO_KEY`, so an
|
|
121
|
-
* endpoint with no key and one that passes `undefined` are one entry rather than two.
|
|
121
|
+
* endpoint with no key and one that passes `undefined` are one entry rather than two. The URL is
|
|
122
|
+
* normalised for the same reason: `…/v1/` and `…/v1` are one server, and two entries for it would
|
|
123
|
+
* learn every refusal twice.
|
|
122
124
|
*
|
|
123
125
|
* Stringified rather than joined on a separator, for the reason `getClient` gives: no character
|
|
124
126
|
* is impossible in a URL or a key, and two endpoints must never collide on one entry. A host
|
|
@@ -207,6 +209,72 @@ export declare function servedSlots(config: Endpoint & {
|
|
|
207
209
|
* request's own error otherwise.
|
|
208
210
|
*/
|
|
209
211
|
export declare function listModels(config: Endpoint, { signal }?: LookupOptions): Promise<ModelInfo[]>;
|
|
212
|
+
/** Why `probeEndpoint` could not list an endpoint's models, in words a form can branch on. */
|
|
213
|
+
export declare const ProbeFailure: {
|
|
214
|
+
/** Not an http or https URL, or one with credentials, a query or a fragment in it. */
|
|
215
|
+
readonly InvalidUrl: 'invalid-url';
|
|
216
|
+
/** Nothing answered: no such host, a refused connection, a TLS failure. */
|
|
217
|
+
readonly Unreachable: 'unreachable';
|
|
218
|
+
/** Something answered, but not inside the lookup's bound. */
|
|
219
|
+
readonly Timeout: 'timeout';
|
|
220
|
+
/** A 401 or a 403: the key is wrong, or one is needed. */
|
|
221
|
+
readonly Unauthorized: 'unauthorized';
|
|
222
|
+
/** A redirect, which is not followed, so the key is never carried to a host nobody typed. */
|
|
223
|
+
readonly Redirected: 'redirected';
|
|
224
|
+
/** An answer that is not a model listing: no such route, a web page, JSON of another shape. */
|
|
225
|
+
readonly NotOpenAi: 'not-openai';
|
|
226
|
+
/** Any other error status, which is on the error as `status`. */
|
|
227
|
+
readonly Failed: 'failed';
|
|
228
|
+
};
|
|
229
|
+
export type ProbeFailure = (typeof ProbeFailure)[keyof typeof ProbeFailure];
|
|
230
|
+
/**
|
|
231
|
+
* `probeEndpoint` could not list an endpoint's models; `kind` says why.
|
|
232
|
+
*
|
|
233
|
+
* @remarks
|
|
234
|
+
* The message is a sentence for a person, naming the endpoint by its origin only, so it can go
|
|
235
|
+
* under a form field as it is. `kind` is for a form that words it itself.
|
|
236
|
+
*/
|
|
237
|
+
export declare class EndpointProbeError extends Error {
|
|
238
|
+
readonly name: string;
|
|
239
|
+
readonly kind: ProbeFailure;
|
|
240
|
+
/** The HTTP status, where the endpoint answered with one. */
|
|
241
|
+
readonly status?: number;
|
|
242
|
+
/**
|
|
243
|
+
* @param kind - Why the listing failed.
|
|
244
|
+
* @param message - The same, as a sentence.
|
|
245
|
+
* @param [status] - The status the endpoint answered with, where it answered.
|
|
246
|
+
*/
|
|
247
|
+
constructor(kind: ProbeFailure, message: string, status?: number);
|
|
248
|
+
}
|
|
249
|
+
/** What `probeEndpoint` takes besides the endpoint. */
|
|
250
|
+
export interface ProbeOptions extends LookupOptions {
|
|
251
|
+
/** How many models come back at most, the first by id. Defaults to `PROBE_MODEL_LIMIT`. */
|
|
252
|
+
limit?: number;
|
|
253
|
+
}
|
|
254
|
+
/**
|
|
255
|
+
* Asks an endpoint nobody has saved yet which models it serves, for a form a person is typing it
|
|
256
|
+
* into.
|
|
257
|
+
*
|
|
258
|
+
* @param endpoint - The endpoint as typed. Its key goes as a bearer token where it has one other
|
|
259
|
+
* than `NO_KEY`; its `requestTimeoutSeconds` bounds the request where that is shorter than
|
|
260
|
+
* `lookupTimeoutMs`.
|
|
261
|
+
* @param [options] - Cancellation, and how many models to hand back. See `ProbeOptions`.
|
|
262
|
+
* @returns The models in order of id, one entry per id, at most `options.limit` of them. Rejects
|
|
263
|
+
* with the reason of `options.signal` where that stopped it, and with an `EndpointProbeError`
|
|
264
|
+
* for every other failure.
|
|
265
|
+
*
|
|
266
|
+
* @remarks
|
|
267
|
+
* `listModels` is for an endpoint that is configured: it pools a client per URL and remembers the
|
|
268
|
+
* listing, and a form asking at every keystroke would leave a client and a listing behind for
|
|
269
|
+
* each half-typed URL. This makes one plain request and remembers nothing, so asking again asks
|
|
270
|
+
* again.
|
|
271
|
+
*
|
|
272
|
+
* Redirects are not followed, so a typed key is never carried on to a host nobody typed. Both
|
|
273
|
+
* listing shapes are read: OpenAI's `{ data: [...] }` by `id`, and Ollama's native
|
|
274
|
+
* `{ models: [...] }` by `name` where an entry has no `id`. A window is read from the keys
|
|
275
|
+
* `contextLimitFor` reads, and an id listed twice keeps the window one of its entries stated.
|
|
276
|
+
*/
|
|
277
|
+
export declare function probeEndpoint(endpoint: EndpointIdentity & Pick<Endpoint, 'requestTimeoutSeconds'>, { signal, limit }?: ProbeOptions): Promise<ModelInfo[]>;
|
|
210
278
|
/**
|
|
211
279
|
* How much a model will read, in tokens. Zero means nobody knows.
|
|
212
280
|
*
|
|
@@ -245,12 +313,12 @@ export declare function contextLimitFor(config: Endpoint & {
|
|
|
245
313
|
*/
|
|
246
314
|
export declare function resetClients(): void;
|
|
247
315
|
/**
|
|
248
|
-
*
|
|
316
|
+
* Whether two base URLs are one endpoint, by `baseUrlKey`.
|
|
249
317
|
*
|
|
250
318
|
* @param a - The URL as one row has it.
|
|
251
319
|
* @param b - The URL as the other has it.
|
|
252
|
-
* @returns `true` where the two differ in nothing but surrounding whitespace
|
|
253
|
-
*
|
|
320
|
+
* @returns `true` where the two differ in nothing but surrounding whitespace, trailing slashes, the
|
|
321
|
+
* case of the scheme and host, and a default port. A path's case counts, and so does a `/v1`.
|
|
254
322
|
*/
|
|
255
323
|
export declare const sameUrl: (a: string, b: string) => boolean;
|
|
256
324
|
/**
|
package/dist/endpoint/client.js
CHANGED
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import OpenAI from 'openai';
|
|
2
|
+
import { baseUrlKey, normalizeBaseUrl, } from '../core/config.js';
|
|
2
3
|
import { digestOf } from '../core/digest.js';
|
|
3
4
|
import { byCodeUnit, isPositive } from '../core/guards.js';
|
|
4
5
|
import { MS_PER_SECOND } from '../core/platform.js';
|
|
@@ -192,7 +193,14 @@ export function getClient(config) {
|
|
|
192
193
|
* reporting the same number twice. llama.cpp puts its number under `meta` instead, and LM Studio
|
|
193
194
|
* and Ollama put none on this route at all; see `servedWindow`.
|
|
194
195
|
*/
|
|
195
|
-
const CONTEXT_KEYS = [
|
|
196
|
+
const CONTEXT_KEYS = [
|
|
197
|
+
'context_length',
|
|
198
|
+
'max_context_length',
|
|
199
|
+
'max_context_window',
|
|
200
|
+
'max_model_len',
|
|
201
|
+
'context_window',
|
|
202
|
+
'n_ctx',
|
|
203
|
+
];
|
|
196
204
|
/**
|
|
197
205
|
* A number above zero as itself and anything else as zero, which here means "not stated".
|
|
198
206
|
*
|
|
@@ -276,20 +284,22 @@ const misses = scoped(() => new Map());
|
|
|
276
284
|
* `Endpoint` requires it — `capabilitiesFor` is handed a URL and maybe a key rather than a whole
|
|
277
285
|
* config, and absent and empty already mean the same thing. The timeout is deliberately not in
|
|
278
286
|
* it; see `listings`.
|
|
279
|
-
* @returns The URL and the key as one JSON string, the key in the clear.
|
|
287
|
+
* @returns The URL as `baseUrlKey` writes it and the key, as one JSON string, the key in the clear.
|
|
280
288
|
*
|
|
281
289
|
* @remarks
|
|
282
290
|
* The URL and the key together, because the key is part of what is on the other end rather than
|
|
283
291
|
* only how it is paid for: a router is free to send two keys to two different backends, and then
|
|
284
292
|
* what one of them refused is not a fact about the other. Absent reads as `NO_KEY`, so an
|
|
285
|
-
* endpoint with no key and one that passes `undefined` are one entry rather than two.
|
|
293
|
+
* endpoint with no key and one that passes `undefined` are one entry rather than two. The URL is
|
|
294
|
+
* normalised for the same reason: `…/v1/` and `…/v1` are one server, and two entries for it would
|
|
295
|
+
* learn every refusal twice.
|
|
286
296
|
*
|
|
287
297
|
* Stringified rather than joined on a separator, for the reason `getClient` gives: no character
|
|
288
298
|
* is impossible in a URL or a key, and two endpoints must never collide on one entry. A host
|
|
289
299
|
* keeping its own per-endpoint state keys it on this rather than on a copy of it, so the two
|
|
290
300
|
* cannot drift apart. It holds the key in the clear; `endpointId` is the one to write down.
|
|
291
301
|
*/
|
|
292
|
-
export const endpointKey = (config) => JSON.stringify([config.baseUrl, config.apiKey || NO_KEY]);
|
|
302
|
+
export const endpointKey = (config) => JSON.stringify([baseUrlKey(config.baseUrl), config.apiKey || NO_KEY]);
|
|
293
303
|
/**
|
|
294
304
|
* One model on one endpoint, as every per-model cache keys it: stringified, so neither half runs
|
|
295
305
|
* into the other.
|
|
@@ -402,23 +412,39 @@ const NOT_THERE = new Set([
|
|
|
402
412
|
HttpStatus.MethodNotAllowed,
|
|
403
413
|
HttpStatus.NotImplemented,
|
|
404
414
|
]);
|
|
415
|
+
/**
|
|
416
|
+
* The headers that carry an endpoint's key, and none where it has none.
|
|
417
|
+
*
|
|
418
|
+
* @param apiKey - The key as configured. Empty, absent and `NO_KEY` all mean there is none.
|
|
419
|
+
* @returns A bearer `authorization` header, or no headers at all.
|
|
420
|
+
*
|
|
421
|
+
* @remarks
|
|
422
|
+
* `NO_KEY` is the SDK's placeholder, not a credential, and a request made outside the SDK has no
|
|
423
|
+
* reason to send it: a server that checks keys refuses `Bearer agent-core` as it refuses a wrong
|
|
424
|
+
* one, where it might have answered an unauthenticated route.
|
|
425
|
+
*/
|
|
426
|
+
const authHeaders = (apiKey) => apiKey && apiKey !== NO_KEY ? { authorization: `Bearer ${apiKey}` } : {};
|
|
405
427
|
/**
|
|
406
428
|
* Asks one native endpoint.
|
|
407
429
|
*
|
|
408
|
-
* @param config - The endpoint. Its key goes as a bearer token where it has one
|
|
430
|
+
* @param config - The endpoint. Its key goes as a bearer token where it has one other than `NO_KEY`.
|
|
409
431
|
* @param path - The route and its query, from the server root rather than from the base URL.
|
|
410
432
|
* @param within - The lookup's signal. Fired, the request throws as an unreachable server does.
|
|
411
433
|
* @returns How the server answered, with `body` only for a 2xx that parsed as JSON.
|
|
412
434
|
*
|
|
413
435
|
* @remarks
|
|
414
|
-
* `failed` is any answer that was not a 2xx and `missing` the ones among them
|
|
415
|
-
* without the route; `body` is absent for
|
|
416
|
-
*
|
|
436
|
+
* `failed` is any answer that was not a 2xx, or was not JSON, and `missing` the ones among them
|
|
437
|
+
* that mean a server without the route; `body` is absent for all of them. A server that could not
|
|
438
|
+
* be reached throws.
|
|
439
|
+
*
|
|
440
|
+
* A 2xx that is not JSON is a missing route rather than a route with nothing to say. It is what a
|
|
441
|
+
* proxy with a web app of its own answers on every path it does not route — llm-router serves its
|
|
442
|
+
* `index.html` for `/props` with a 200 — and read as "no window" it was asked again every
|
|
443
|
+
* `listingMissMs` instead of being latched as a server without the route.
|
|
417
444
|
*/
|
|
418
445
|
async function probe(config, path, within) {
|
|
419
|
-
const apiKey = config.apiKey || undefined;
|
|
420
446
|
const response = await fetch(`${rootOf(config.baseUrl)}${path}`, {
|
|
421
|
-
headers: apiKey
|
|
447
|
+
headers: authHeaders(config.apiKey),
|
|
422
448
|
signal: within,
|
|
423
449
|
});
|
|
424
450
|
if (!response.ok) {
|
|
@@ -428,7 +454,9 @@ async function probe(config, path, within) {
|
|
|
428
454
|
return { failed: false, missing: false, body: await response.json() };
|
|
429
455
|
}
|
|
430
456
|
catch {
|
|
431
|
-
|
|
457
|
+
// A fired signal surfaces here too, mid-body, and is a server that stopped answering.
|
|
458
|
+
within.throwIfAborted();
|
|
459
|
+
return { failed: true, missing: true };
|
|
432
460
|
}
|
|
433
461
|
}
|
|
434
462
|
/**
|
|
@@ -594,6 +622,144 @@ async function readListing(config, within) {
|
|
|
594
622
|
trainedOnly().set(key, trained);
|
|
595
623
|
return models;
|
|
596
624
|
}
|
|
625
|
+
/** Why `probeEndpoint` could not list an endpoint's models, in words a form can branch on. */
|
|
626
|
+
export const ProbeFailure = {
|
|
627
|
+
/** Not an http or https URL, or one with credentials, a query or a fragment in it. */
|
|
628
|
+
InvalidUrl: 'invalid-url',
|
|
629
|
+
/** Nothing answered: no such host, a refused connection, a TLS failure. */
|
|
630
|
+
Unreachable: 'unreachable',
|
|
631
|
+
/** Something answered, but not inside the lookup's bound. */
|
|
632
|
+
Timeout: 'timeout',
|
|
633
|
+
/** A 401 or a 403: the key is wrong, or one is needed. */
|
|
634
|
+
Unauthorized: 'unauthorized',
|
|
635
|
+
/** A redirect, which is not followed, so the key is never carried to a host nobody typed. */
|
|
636
|
+
Redirected: 'redirected',
|
|
637
|
+
/** An answer that is not a model listing: no such route, a web page, JSON of another shape. */
|
|
638
|
+
NotOpenAi: 'not-openai',
|
|
639
|
+
/** Any other error status, which is on the error as `status`. */
|
|
640
|
+
Failed: 'failed',
|
|
641
|
+
};
|
|
642
|
+
/**
|
|
643
|
+
* `probeEndpoint` could not list an endpoint's models; `kind` says why.
|
|
644
|
+
*
|
|
645
|
+
* @remarks
|
|
646
|
+
* The message is a sentence for a person, naming the endpoint by its origin only, so it can go
|
|
647
|
+
* under a form field as it is. `kind` is for a form that words it itself.
|
|
648
|
+
*/
|
|
649
|
+
export class EndpointProbeError extends Error {
|
|
650
|
+
name = 'EndpointProbeError';
|
|
651
|
+
kind;
|
|
652
|
+
/** The HTTP status, where the endpoint answered with one. */
|
|
653
|
+
status;
|
|
654
|
+
/**
|
|
655
|
+
* @param kind - Why the listing failed.
|
|
656
|
+
* @param message - The same, as a sentence.
|
|
657
|
+
* @param [status] - The status the endpoint answered with, where it answered.
|
|
658
|
+
*/
|
|
659
|
+
constructor(kind, message, status) {
|
|
660
|
+
super(message);
|
|
661
|
+
this.kind = kind;
|
|
662
|
+
if (status !== undefined) {
|
|
663
|
+
this.status = status;
|
|
664
|
+
}
|
|
665
|
+
}
|
|
666
|
+
}
|
|
667
|
+
/** How many models `probeEndpoint` hands back unless told otherwise: enough for any picker. */
|
|
668
|
+
const PROBE_MODEL_LIMIT = 500;
|
|
669
|
+
/** The statuses that mean the key, not the server, is what is wrong. */
|
|
670
|
+
const UNAUTHORIZED = new Set([HttpStatus.Unauthorized, HttpStatus.Forbidden]);
|
|
671
|
+
/** The range a redirect is answered in. */
|
|
672
|
+
const REDIRECT_MIN = 300;
|
|
673
|
+
const REDIRECT_MAX = 399;
|
|
674
|
+
/**
|
|
675
|
+
* Asks an endpoint nobody has saved yet which models it serves, for a form a person is typing it
|
|
676
|
+
* into.
|
|
677
|
+
*
|
|
678
|
+
* @param endpoint - The endpoint as typed. Its key goes as a bearer token where it has one other
|
|
679
|
+
* than `NO_KEY`; its `requestTimeoutSeconds` bounds the request where that is shorter than
|
|
680
|
+
* `lookupTimeoutMs`.
|
|
681
|
+
* @param [options] - Cancellation, and how many models to hand back. See `ProbeOptions`.
|
|
682
|
+
* @returns The models in order of id, one entry per id, at most `options.limit` of them. Rejects
|
|
683
|
+
* with the reason of `options.signal` where that stopped it, and with an `EndpointProbeError`
|
|
684
|
+
* for every other failure.
|
|
685
|
+
*
|
|
686
|
+
* @remarks
|
|
687
|
+
* `listModels` is for an endpoint that is configured: it pools a client per URL and remembers the
|
|
688
|
+
* listing, and a form asking at every keystroke would leave a client and a listing behind for
|
|
689
|
+
* each half-typed URL. This makes one plain request and remembers nothing, so asking again asks
|
|
690
|
+
* again.
|
|
691
|
+
*
|
|
692
|
+
* Redirects are not followed, so a typed key is never carried on to a host nobody typed. Both
|
|
693
|
+
* listing shapes are read: OpenAI's `{ data: [...] }` by `id`, and Ollama's native
|
|
694
|
+
* `{ models: [...] }` by `name` where an entry has no `id`. A window is read from the keys
|
|
695
|
+
* `contextLimitFor` reads, and an id listed twice keeps the window one of its entries stated.
|
|
696
|
+
*/
|
|
697
|
+
export async function probeEndpoint(endpoint, { signal, limit = PROBE_MODEL_LIMIT } = {}) {
|
|
698
|
+
const baseUrl = normalizeBaseUrl(endpoint.baseUrl);
|
|
699
|
+
if (baseUrl === null) {
|
|
700
|
+
throw new EndpointProbeError(ProbeFailure.InvalidUrl, 'A base URL is http:// or https://, with no credentials, query or fragment in it.');
|
|
701
|
+
}
|
|
702
|
+
const url = new URL(`${baseUrl}/models`);
|
|
703
|
+
const within = lookupSignal(endpoint, signal);
|
|
704
|
+
/** The failure for a request that threw: the caller's stop, the bound, or no answer at all. */
|
|
705
|
+
const thrown = () => {
|
|
706
|
+
signal?.throwIfAborted();
|
|
707
|
+
return within.aborted
|
|
708
|
+
? new EndpointProbeError(ProbeFailure.Timeout, `${url.origin} did not answer in time.`)
|
|
709
|
+
: new EndpointProbeError(ProbeFailure.Unreachable, `Could not reach ${url.origin}.`);
|
|
710
|
+
};
|
|
711
|
+
let response;
|
|
712
|
+
try {
|
|
713
|
+
response = await fetch(url, { headers: authHeaders(endpoint.apiKey), signal: within, redirect: 'manual' });
|
|
714
|
+
}
|
|
715
|
+
catch {
|
|
716
|
+
throw thrown();
|
|
717
|
+
}
|
|
718
|
+
const { status } = response;
|
|
719
|
+
// A browser hands a refused redirect back as an opaque response with status zero.
|
|
720
|
+
if (response.type === 'opaqueredirect' || (status >= REDIRECT_MIN && status <= REDIRECT_MAX)) {
|
|
721
|
+
const location = response.headers.get('location');
|
|
722
|
+
throw new EndpointProbeError(ProbeFailure.Redirected, `${url.origin} redirected the request${location ? ` to ${location}` : ''}; use the URL it redirects to.`, status || undefined);
|
|
723
|
+
}
|
|
724
|
+
if (UNAUTHORIZED.has(status)) {
|
|
725
|
+
throw new EndpointProbeError(ProbeFailure.Unauthorized, `${url.origin} refused the key (${status}).`, status);
|
|
726
|
+
}
|
|
727
|
+
if (NOT_THERE.has(status)) {
|
|
728
|
+
throw new EndpointProbeError(ProbeFailure.NotOpenAi, `${url.origin} has no model listing there (${status}); is the base URL missing a /v1?`, status);
|
|
729
|
+
}
|
|
730
|
+
if (response.ok === false) {
|
|
731
|
+
throw new EndpointProbeError(ProbeFailure.Failed, `${url.origin} answered ${status} when asked for its models.`, status);
|
|
732
|
+
}
|
|
733
|
+
let body;
|
|
734
|
+
try {
|
|
735
|
+
body = await response.json();
|
|
736
|
+
}
|
|
737
|
+
catch {
|
|
738
|
+
if (within.aborted) {
|
|
739
|
+
throw thrown();
|
|
740
|
+
}
|
|
741
|
+
body = undefined;
|
|
742
|
+
}
|
|
743
|
+
const { data, models } = (body ?? {});
|
|
744
|
+
const rows = Array.isArray(data) ? data : Array.isArray(models) ? models : undefined;
|
|
745
|
+
if (rows === undefined) {
|
|
746
|
+
throw new EndpointProbeError(ProbeFailure.NotOpenAi, `${url.origin} did not answer with a list of models.`);
|
|
747
|
+
}
|
|
748
|
+
const found = new Map();
|
|
749
|
+
for (const row of rows) {
|
|
750
|
+
if (row === null || typeof row !== 'object') {
|
|
751
|
+
continue;
|
|
752
|
+
}
|
|
753
|
+
const entry = row;
|
|
754
|
+
const id = typeof entry.id === 'string' ? entry.id : typeof entry.name === 'string' ? entry.name : '';
|
|
755
|
+
if (id === '') {
|
|
756
|
+
continue;
|
|
757
|
+
}
|
|
758
|
+
const { contextLength } = contextLengthOf(row);
|
|
759
|
+
found.set(id, { id, contextLength: contextLength || (found.get(id)?.contextLength ?? 0) });
|
|
760
|
+
}
|
|
761
|
+
return [...found.values()].sort((a, b) => byCodeUnit(a.id, b.id)).slice(0, limit);
|
|
762
|
+
}
|
|
597
763
|
/**
|
|
598
764
|
* How much a model will read, in tokens. Zero means nobody knows.
|
|
599
765
|
*
|
|
@@ -692,14 +858,14 @@ export function resetClients() {
|
|
|
692
858
|
refusals().clear();
|
|
693
859
|
}
|
|
694
860
|
/**
|
|
695
|
-
*
|
|
861
|
+
* Whether two base URLs are one endpoint, by `baseUrlKey`.
|
|
696
862
|
*
|
|
697
863
|
* @param a - The URL as one row has it.
|
|
698
864
|
* @param b - The URL as the other has it.
|
|
699
|
-
* @returns `true` where the two differ in nothing but surrounding whitespace
|
|
700
|
-
*
|
|
865
|
+
* @returns `true` where the two differ in nothing but surrounding whitespace, trailing slashes, the
|
|
866
|
+
* case of the scheme and host, and a default port. A path's case counts, and so does a `/v1`.
|
|
701
867
|
*/
|
|
702
|
-
export const sameUrl = (a, b) => a
|
|
868
|
+
export const sameUrl = (a, b) => baseUrlKey(a) === baseUrlKey(b);
|
|
703
869
|
/**
|
|
704
870
|
* The key to send, where an endpoint may inherit one from the settings it overrides.
|
|
705
871
|
*
|
package/dist/endpoint/slots.js
CHANGED
|
@@ -1,3 +1,4 @@
|
|
|
1
|
+
import { baseUrlKey } from '../core/config.js';
|
|
1
2
|
import { scoped } from '../core/scope.js';
|
|
2
3
|
import { modelKey } from './client.js';
|
|
3
4
|
/**
|
|
@@ -22,14 +23,14 @@ const pools = scoped(() => ({ slots: new Map(), clock: 0 }));
|
|
|
22
23
|
*
|
|
23
24
|
* @param endpoint - The server, read for `baseUrl` alone.
|
|
24
25
|
* @param model - The model, which a llama.cpp router serves from a process with slots of its own.
|
|
25
|
-
* @returns The key: the base URL
|
|
26
|
+
* @returns The key: the base URL as `baseUrlKey` writes it, and the model.
|
|
26
27
|
*
|
|
27
28
|
* @remarks
|
|
28
29
|
* The API key is left out where every other per-endpoint cache keys on it. Two keys reaching
|
|
29
30
|
* different servers through one URL cost only a slot left unused; two keys reaching the same
|
|
30
31
|
* server under two allocators hand one slot to two runs, which is the collision this exists to stop.
|
|
31
32
|
*/
|
|
32
|
-
const poolKey = (endpoint, model) => modelKey(endpoint.baseUrl
|
|
33
|
+
const poolKey = (endpoint, model) => modelKey(baseUrlKey(endpoint.baseUrl), model);
|
|
33
34
|
/**
|
|
34
35
|
* The slot allocator for one endpoint and model, sized to what its `/props` said.
|
|
35
36
|
*
|
package/dist/index.d.ts
CHANGED
|
@@ -11,10 +11,10 @@
|
|
|
11
11
|
*/
|
|
12
12
|
export { applyCompaction, COMPACT_AT, type CompactionOptions, type CompactionPlan, type CompactionRecord, type CompactionRunOptions, compactTranscript, KEEP_RATIO, type PruneOptions, planCompaction, pruneToolResults, requestIndex, runCompaction, SUMMARY_LEAD, SUMMARY_PROMPT, summariser, } from './context/compaction.ts';
|
|
13
13
|
export { estimateFrom, type LedgerEntry, type LedgerEstimateOptions, type LedgerRequest, rebaseLedger, recordRequest, type TokenLedger, tokensBetween, } from './context/ledger.ts';
|
|
14
|
-
export { type AgentConfig, type Endpoint, type ModelParams, type RetryPolicy, ToolDiscovery, type ToolPolicy, } from './core/config.ts';
|
|
14
|
+
export { type AgentConfig, type Endpoint, type ModelParams, normalizeBaseUrl, type RetryPolicy, ToolDiscovery, type ToolPolicy, } from './core/config.ts';
|
|
15
15
|
export { calibrate, charsPerTokenFor, resetCalibration } from './endpoint/calibration.ts';
|
|
16
16
|
export { type Capabilities, capabilitiesFor, EFFORT_LADDER, EFFORT_NONE, EFFORT_OFF, effortFor, expireCapabilities, type ModelCapabilities, modelCapabilitiesFor, type NegotiateOptions, negotiate, resetCapabilities, } from './endpoint/capabilities.ts';
|
|
17
|
-
export { type ClientPoolOptions, configureClients, contextLimitFor, endpointId, endpointKey, firstTokenMs, getClient, type LookupOptions, listModels, type ModelInfo, NO_KEY, resetClients, resolveApiKey, servedSlots, servedWindow, timeoutMs, } from './endpoint/client.ts';
|
|
17
|
+
export { type ClientPoolOptions, configureClients, contextLimitFor, EndpointProbeError, endpointId, endpointKey, firstTokenMs, getClient, type LookupOptions, listModels, type ModelInfo, NO_KEY, ProbeFailure, type ProbeOptions, probeEndpoint, resetClients, resolveApiKey, servedSlots, servedWindow, timeoutMs, } from './endpoint/client.ts';
|
|
18
18
|
export { isGrammarError, relaxSchema, relaxTools, sanitizeSchema, sanitizeTools, } from './endpoint/schema-compat.ts';
|
|
19
19
|
export { CACHE_PROMPT_FIELD, resetSlots, SESSION_HEADER, SLOT_FIELD, type SlotAllocator, slotAllocator, } from './endpoint/slots.ts';
|
|
20
20
|
export { HOOK_EVENTS, HookEvent, INJECT_EVENTS } from './hooks/hook-events.ts';
|
|
@@ -39,5 +39,5 @@ export { backoffMs, ContextOverflow, EndpointSilent, isModelLoading, isOverflow,
|
|
|
39
39
|
export { CacheBreakReason, type Produced, type StreamTurnOptions, streamTurn, type Turn, type TurnUsage, } from './wire/stream.ts';
|
|
40
40
|
export { ALL_FENCES, DEFAULT_FENCES, type Fence, FenceSplitter, type FenceSplitterOptions, SPLIT_OUTPUT, SPLIT_REASONING, type Split, stripThinking, THINK_FENCE, } from './wire/thinking.ts';
|
|
41
41
|
export { type ContextBreakdown, type ContextBreakdownOptions, compact, contextChars, contextTokens, estimateTokens, messageTokens, requestChars, requestTokens, type TokenEstimateOptions, } from './wire/tokens.ts';
|
|
42
|
-
export { ARGUMENTS_MALFORMED, ARGUMENTS_TRUNCATED, parseToolArguments, type ResourceLink, type RunArtifact, recoverToolCalls, type ServerLogEntry, ServerLogLevel, type ToolAnswer, ToolArgumentsError, type ToolCall, type ToolCallOutcome, type ToolCallReport, type ToolCallRequest, type ToolCallResult, } from './wire/tool-calls.ts';
|
|
42
|
+
export { ARGUMENTS_MALFORMED, ARGUMENTS_TRUNCATED, type EndedBy, parseToolArguments, type ResourceLink, type RunArtifact, recoverToolCalls, type ServerLogEntry, ServerLogLevel, type ToolAnswer, ToolArgumentsError, type ToolCall, type ToolCallOutcome, type ToolCallReport, type ToolCallRequest, type ToolCallResult, } from './wire/tool-calls.ts';
|
|
43
43
|
export { FinishReason, FUNCTION_TOOL, JSON_SCHEMA_FORMAT, PartType, Role, SchemaType } from './wire/wire.ts';
|
package/dist/index.js
CHANGED
|
@@ -11,10 +11,10 @@
|
|
|
11
11
|
*/
|
|
12
12
|
export { applyCompaction, COMPACT_AT, compactTranscript, KEEP_RATIO, planCompaction, pruneToolResults, requestIndex, runCompaction, SUMMARY_LEAD, SUMMARY_PROMPT, summariser, } from './context/compaction.js';
|
|
13
13
|
export { estimateFrom, rebaseLedger, recordRequest, tokensBetween, } from './context/ledger.js';
|
|
14
|
-
export { ToolDiscovery, } from './core/config.js';
|
|
14
|
+
export { normalizeBaseUrl, ToolDiscovery, } from './core/config.js';
|
|
15
15
|
export { calibrate, charsPerTokenFor, resetCalibration } from './endpoint/calibration.js';
|
|
16
16
|
export { capabilitiesFor, EFFORT_LADDER, EFFORT_NONE, EFFORT_OFF, effortFor, expireCapabilities, modelCapabilitiesFor, negotiate, resetCapabilities, } from './endpoint/capabilities.js';
|
|
17
|
-
export { configureClients, contextLimitFor, endpointId, endpointKey, firstTokenMs, getClient, listModels, NO_KEY, resetClients, resolveApiKey, servedSlots, servedWindow, timeoutMs, } from './endpoint/client.js';
|
|
17
|
+
export { configureClients, contextLimitFor, EndpointProbeError, endpointId, endpointKey, firstTokenMs, getClient, listModels, NO_KEY, ProbeFailure, probeEndpoint, resetClients, resolveApiKey, servedSlots, servedWindow, timeoutMs, } from './endpoint/client.js';
|
|
18
18
|
export { isGrammarError, relaxSchema, relaxTools, sanitizeSchema, sanitizeTools, } from './endpoint/schema-compat.js';
|
|
19
19
|
export { CACHE_PROMPT_FIELD, resetSlots, SESSION_HEADER, SLOT_FIELD, slotAllocator, } from './endpoint/slots.js';
|
|
20
20
|
export { HOOK_EVENTS, HookEvent, INJECT_EVENTS } from './hooks/hook-events.js';
|
package/dist/run/agent-loop.d.ts
CHANGED
|
@@ -10,7 +10,7 @@ import { type KeywordPreselectOptions } from '../tools/preselect.ts';
|
|
|
10
10
|
import { type ToolOrder } from '../tools/tool-loading.ts';
|
|
11
11
|
import { type SideTask } from '../turn/side-task.ts';
|
|
12
12
|
import { type Turn, type TurnUsage } from '../wire/stream.ts';
|
|
13
|
-
import { type RunArtifact, type ToolCallOutcome, type ToolCallRequest, type ToolCallResult } from '../wire/tool-calls.ts';
|
|
13
|
+
import { type EndedBy, type RunArtifact, type ToolCallOutcome, type ToolCallRequest, type ToolCallResult } from '../wire/tool-calls.ts';
|
|
14
14
|
import { type RunEventInput, type RunMetrics } from './events.ts';
|
|
15
15
|
import { type ToolDispatch } from './run-calls.ts';
|
|
16
16
|
/**
|
|
@@ -243,7 +243,8 @@ export interface AgentLoopOptions {
|
|
|
243
243
|
* @remarks
|
|
244
244
|
* A `ToolAnswer` in place of the string adds what the call produced: its `text` is the whole of
|
|
245
245
|
* what goes into the transcript, and its `links` become the result's `artifacts`, an `artifact`
|
|
246
|
-
* event the first time each uri is seen, and `links` on what `onToolResult` is handed.
|
|
246
|
+
* event the first time each uri is seen, and `links` on what `onToolResult` is handed. Its `end`
|
|
247
|
+
* ends the run once the step's calls are answered; see `AgentLoopResult.ended`.
|
|
247
248
|
*/
|
|
248
249
|
dispatch: ToolDispatch;
|
|
249
250
|
/**
|
|
@@ -395,8 +396,29 @@ export interface AgentLoopOptions {
|
|
|
395
396
|
}
|
|
396
397
|
/** What a finished loop hands back. */
|
|
397
398
|
export interface AgentLoopResult {
|
|
398
|
-
/**
|
|
399
|
+
/**
|
|
400
|
+
* The last turn, as `onTurn` was handed it: the one that asked for no tools, or — where a call
|
|
401
|
+
* `ended` the run — the one that made that call.
|
|
402
|
+
*
|
|
403
|
+
* @remarks
|
|
404
|
+
* Ended by a call, its `content` is what the model wrote beside its calls, which is often
|
|
405
|
+
* nothing, and its `toolCalls` are not empty. The run's answer is then whatever the ending tool
|
|
406
|
+
* took — the question it asked, the work it handed off — and not this text.
|
|
407
|
+
*/
|
|
399
408
|
turn: Turn;
|
|
409
|
+
/**
|
|
410
|
+
* The call whose answer said `end`, where one did: the first in the order the model asked, once
|
|
411
|
+
* every call of its step was answered. Absent where the run ended on a turn that asked for no
|
|
412
|
+
* tools.
|
|
413
|
+
*
|
|
414
|
+
* @remarks
|
|
415
|
+
* Every other field is as a finished run has it: `messages` close on the step's tool results
|
|
416
|
+
* with no assistant turn after them, the hooks are told `afterTurn` with `turn.content`, and
|
|
417
|
+
* `metrics.outcome` is `ended`. No `beforeStep` or `compact` runs after the end, since no request
|
|
418
|
+
* follows it. A host saying `done` passes the call on it — `id` the call, `name` the tool, `text`
|
|
419
|
+
* the reason — which is how `runMetrics` reads the same outcome off the events.
|
|
420
|
+
*/
|
|
421
|
+
ended?: EndedBy;
|
|
400
422
|
/** The transcript, with every assistant turn and tool result the run added. No system prompt. */
|
|
401
423
|
messages: OpenAI.ChatCompletionMessageParam[];
|
|
402
424
|
/** Summed over every turn of the run. */
|
|
@@ -463,14 +485,16 @@ export interface AgentLoopResult {
|
|
|
463
485
|
}
|
|
464
486
|
/**
|
|
465
487
|
* Runs a question to its answer: one `runTurn` per step, the tools it asks for between them,
|
|
466
|
-
* until a turn asks for none. Throws when `maxToolIterations` is
|
|
467
|
-
* whatever `runTurn` throws — `ContextOverflow` among them, however it
|
|
488
|
+
* until a turn asks for none or a call's answer ends the run. Throws when `maxToolIterations` is
|
|
489
|
+
* spent, when stopped, and on whatever `runTurn` throws — `ContextOverflow` among them, however it
|
|
490
|
+
* was found out.
|
|
468
491
|
*
|
|
469
492
|
* @param options - The config, transcript, tools and dispatcher, plus the optional hooks, events
|
|
470
493
|
* and cancellation. See `AgentLoopOptions`.
|
|
471
|
-
* @returns The turn that asked for no tools
|
|
472
|
-
*
|
|
473
|
-
* and the ledger. See
|
|
494
|
+
* @returns The turn that asked for no tools — or made the call that ended the run, with `ended`
|
|
495
|
+
* naming it — and the run around it: the transcript with what the run added, usage, every call,
|
|
496
|
+
* what is loaded and used, the hooks' notes and context, metrics and the ledger. See
|
|
497
|
+
* `AgentLoopResult`.
|
|
474
498
|
*
|
|
475
499
|
* @remarks
|
|
476
500
|
* What it throws carries the run as it stood — `messages`, `usage`, `toolCalls`, `loaded`, `used`
|
|
@@ -482,6 +506,11 @@ export interface AgentLoopResult {
|
|
|
482
506
|
* well-formed: the results that came back are kept, and each call without one is answered with a
|
|
483
507
|
* line saying it was stopped or never run, since a call with no result is refused on replay.
|
|
484
508
|
*
|
|
509
|
+
* A tool that finishes the job — a hand-off, a question for a person — ends the run with a
|
|
510
|
+
* `ToolAnswer` whose `end` is set, rather than by aborting it: the step's calls are all run and
|
|
511
|
+
* answered, no further request is sent, and the loop resolves with `ended`. An end in the last step
|
|
512
|
+
* `maxToolIterations` allows resolves too, since nothing was left undone.
|
|
513
|
+
*
|
|
485
514
|
* On-demand loading is handled here, `load_tools` and all: the catalogue rides on the system
|
|
486
515
|
* prompt unchanged from step to step, a load adds to the tool array — which every request sends
|
|
487
516
|
* in the stable order `toolOrder` asks for — and
|
package/dist/run/agent-loop.js
CHANGED
|
@@ -63,14 +63,16 @@ function cacheDiagnosis(previous, next, usage) {
|
|
|
63
63
|
}
|
|
64
64
|
/**
|
|
65
65
|
* Runs a question to its answer: one `runTurn` per step, the tools it asks for between them,
|
|
66
|
-
* until a turn asks for none. Throws when `maxToolIterations` is
|
|
67
|
-
* whatever `runTurn` throws — `ContextOverflow` among them, however it
|
|
66
|
+
* until a turn asks for none or a call's answer ends the run. Throws when `maxToolIterations` is
|
|
67
|
+
* spent, when stopped, and on whatever `runTurn` throws — `ContextOverflow` among them, however it
|
|
68
|
+
* was found out.
|
|
68
69
|
*
|
|
69
70
|
* @param options - The config, transcript, tools and dispatcher, plus the optional hooks, events
|
|
70
71
|
* and cancellation. See `AgentLoopOptions`.
|
|
71
|
-
* @returns The turn that asked for no tools
|
|
72
|
-
*
|
|
73
|
-
* and the ledger. See
|
|
72
|
+
* @returns The turn that asked for no tools — or made the call that ended the run, with `ended`
|
|
73
|
+
* naming it — and the run around it: the transcript with what the run added, usage, every call,
|
|
74
|
+
* what is loaded and used, the hooks' notes and context, metrics and the ledger. See
|
|
75
|
+
* `AgentLoopResult`.
|
|
74
76
|
*
|
|
75
77
|
* @remarks
|
|
76
78
|
* What it throws carries the run as it stood — `messages`, `usage`, `toolCalls`, `loaded`, `used`
|
|
@@ -82,6 +84,11 @@ function cacheDiagnosis(previous, next, usage) {
|
|
|
82
84
|
* well-formed: the results that came back are kept, and each call without one is answered with a
|
|
83
85
|
* line saying it was stopped or never run, since a call with no result is refused on replay.
|
|
84
86
|
*
|
|
87
|
+
* A tool that finishes the job — a hand-off, a question for a person — ends the run with a
|
|
88
|
+
* `ToolAnswer` whose `end` is set, rather than by aborting it: the step's calls are all run and
|
|
89
|
+
* answered, no further request is sent, and the loop resolves with `ended`. An end in the last step
|
|
90
|
+
* `maxToolIterations` allows resolves too, since nothing was left undone.
|
|
91
|
+
*
|
|
85
92
|
* On-demand loading is handled here, `load_tools` and all: the catalogue rides on the system
|
|
86
93
|
* prompt unchanged from step to step, a load adds to the tool array — which every request sends
|
|
87
94
|
* in the stable order `toolOrder` asks for — and
|
|
@@ -471,8 +478,8 @@ function turnCalls(turn, recoverable, notice) {
|
|
|
471
478
|
* @param standing - Written to: once the steps have set themselves up, its `read` is theirs, so the
|
|
472
479
|
* caller can read the run as it stands whether they return or throw.
|
|
473
480
|
* @param [pin] - What the run's session adds to each request. Absent adds nothing.
|
|
474
|
-
* @returns The result of the turn that asked for no tools
|
|
475
|
-
* is spent.
|
|
481
|
+
* @returns The result of the turn that asked for no tools or whose calls ended the run, or
|
|
482
|
+
* `undefined` when `maxToolIterations` is spent.
|
|
476
483
|
*/
|
|
477
484
|
async function runSteps(options, standing, pin) {
|
|
478
485
|
const { config, system = '', tools = [], catalog = [], dispatch, hooks, signal } = options;
|
|
@@ -706,13 +713,27 @@ async function runSteps(options, standing, pin) {
|
|
|
706
713
|
const assistant = assistantMessage(content, parsed);
|
|
707
714
|
messages.push(assistant);
|
|
708
715
|
await announce(assistant, step, shown);
|
|
709
|
-
|
|
716
|
+
const isAnswer = calls.length === 0;
|
|
717
|
+
const ended = isAnswer ? undefined : await runCalls(calling, parsed, messages, step);
|
|
718
|
+
if (ended) {
|
|
719
|
+
const by = ended.reason ? `: ${ended.reason}` : '';
|
|
720
|
+
notice(`${ended.name} ended the run${by}`);
|
|
721
|
+
}
|
|
722
|
+
if (isAnswer || ended) {
|
|
710
723
|
// Not awaited: the answer is ready, and remembering it is not something to hold it for.
|
|
711
724
|
// Handed back instead, for a host that wants its notes.
|
|
712
725
|
const afterTurn = hooks ? tellAfterTurn(hooks, messages, questionAt(), turn.content) : Promise.resolve([]);
|
|
713
726
|
const metrics = runMetrics(recorded, { contextLength: config.contextLength });
|
|
727
|
+
let outcome = RunOutcome.Answered;
|
|
728
|
+
if (ended) {
|
|
729
|
+
outcome = RunOutcome.Ended;
|
|
730
|
+
}
|
|
731
|
+
else if (turn.finishReason === FinishReason.Length) {
|
|
732
|
+
outcome = RunOutcome.Truncated;
|
|
733
|
+
}
|
|
714
734
|
return {
|
|
715
735
|
turn: shown,
|
|
736
|
+
...(ended ? { ended } : {}),
|
|
716
737
|
messages,
|
|
717
738
|
usage,
|
|
718
739
|
toolCalls,
|
|
@@ -728,12 +749,11 @@ async function runSteps(options, standing, pin) {
|
|
|
728
749
|
...metrics,
|
|
729
750
|
...(onDemand ? loads : {}),
|
|
730
751
|
wallMs: Date.now() - started,
|
|
731
|
-
outcome
|
|
752
|
+
outcome,
|
|
732
753
|
},
|
|
733
754
|
ledger,
|
|
734
755
|
};
|
|
735
756
|
}
|
|
736
|
-
await runCalls(calling, parsed, messages, step);
|
|
737
757
|
}
|
|
738
758
|
return undefined;
|
|
739
759
|
}
|
package/dist/run/events.d.ts
CHANGED
|
@@ -109,7 +109,10 @@ export declare const RunEventKind: {
|
|
|
109
109
|
* measured either way.
|
|
110
110
|
*/
|
|
111
111
|
readonly Usage: 'usage';
|
|
112
|
-
/**
|
|
112
|
+
/**
|
|
113
|
+
* The run ended. Always last, and always sent. A run a tool call ended carries that call: `id`
|
|
114
|
+
* the call's, `name` the tool, and `text` the reason, as `AgentLoopResult.ended` has them.
|
|
115
|
+
*/
|
|
113
116
|
readonly Done: 'done';
|
|
114
117
|
};
|
|
115
118
|
/** Any one of the kinds in `RunEventKind`. */
|
|
@@ -171,7 +174,7 @@ export interface RunEvent {
|
|
|
171
174
|
kind: RunEventKind;
|
|
172
175
|
/** The delta, the arguments, the result, or the reason — whatever the kind carries. */
|
|
173
176
|
text: string;
|
|
174
|
-
/** Tool name on the tool kinds, otherwise empty. */
|
|
177
|
+
/** Tool name on the tool kinds, and on a `done` the tool that ended the run; otherwise empty. */
|
|
175
178
|
name: string;
|
|
176
179
|
/**
|
|
177
180
|
* The call's id on the tool kinds, which pairs a `tool-result` with its `tool-call` when one
|
|
@@ -390,6 +393,11 @@ export declare const RunOutcome: {
|
|
|
390
393
|
readonly Answered: 'answered';
|
|
391
394
|
/** The run did not fail, and its last turn is an answer the reply ceiling cut off. */
|
|
392
395
|
readonly Truncated: 'truncated';
|
|
396
|
+
/**
|
|
397
|
+
* The run did not fail, and a tool call ended it: `AgentLoopResult.ended` says which, and so
|
|
398
|
+
* does a `done` that names the tool.
|
|
399
|
+
*/
|
|
400
|
+
readonly Ended: 'ended';
|
|
393
401
|
/**
|
|
394
402
|
* `done` said the run was not ok. Whether that was an error, a stop or the tool budget is in
|
|
395
403
|
* the host's own `done` text and not here.
|
package/dist/run/events.js
CHANGED
|
@@ -76,7 +76,10 @@ export const RunEventKind = {
|
|
|
76
76
|
* measured either way.
|
|
77
77
|
*/
|
|
78
78
|
Usage: 'usage',
|
|
79
|
-
/**
|
|
79
|
+
/**
|
|
80
|
+
* The run ended. Always last, and always sent. A run a tool call ended carries that call: `id`
|
|
81
|
+
* the call's, `name` the tool, and `text` the reason, as `AgentLoopResult.ended` has them.
|
|
82
|
+
*/
|
|
80
83
|
Done: 'done',
|
|
81
84
|
};
|
|
82
85
|
/**
|
|
@@ -471,6 +474,11 @@ export const RunOutcome = {
|
|
|
471
474
|
Answered: 'answered',
|
|
472
475
|
/** The run did not fail, and its last turn is an answer the reply ceiling cut off. */
|
|
473
476
|
Truncated: 'truncated',
|
|
477
|
+
/**
|
|
478
|
+
* The run did not fail, and a tool call ended it: `AgentLoopResult.ended` says which, and so
|
|
479
|
+
* does a `done` that names the tool.
|
|
480
|
+
*/
|
|
481
|
+
Ended: 'ended',
|
|
474
482
|
/**
|
|
475
483
|
* `done` said the run was not ok. Whether that was an error, a stop or the tool budget is in
|
|
476
484
|
* the host's own `done` text and not here.
|
|
@@ -480,15 +488,21 @@ export const RunOutcome = {
|
|
|
480
488
|
/**
|
|
481
489
|
* How a run that reported `done` ended.
|
|
482
490
|
*
|
|
483
|
-
* @param
|
|
491
|
+
* @param done - The `done` event. Only an `ok` of `false` is a failure; one that names a tool was
|
|
492
|
+
* ended by it; anything else is read off the turn.
|
|
484
493
|
* @param last - The last turn a `usage` event reported, if any did.
|
|
485
|
-
* @returns `failed` for a `done` that was not ok, `
|
|
486
|
-
* reply ceiling, and `answered` otherwise.
|
|
494
|
+
* @returns `failed` for a `done` that was not ok, `ended` for one that names the tool that ended
|
|
495
|
+
* the run, `truncated` where the last turn ran into the reply ceiling, and `answered` otherwise.
|
|
487
496
|
*/
|
|
488
|
-
function outcomeOf(ok, last) {
|
|
497
|
+
function outcomeOf({ ok, name }, last) {
|
|
489
498
|
if (ok === false) {
|
|
490
499
|
return RunOutcome.Failed;
|
|
491
500
|
}
|
|
501
|
+
// Read as a truthy string, so an event put together by hand without one is not ended.
|
|
502
|
+
const isEnded = Boolean(name);
|
|
503
|
+
if (isEnded) {
|
|
504
|
+
return RunOutcome.Ended;
|
|
505
|
+
}
|
|
492
506
|
if (last?.finishReason === FinishReason.Length) {
|
|
493
507
|
return RunOutcome.Truncated;
|
|
494
508
|
}
|
|
@@ -614,7 +628,7 @@ export function runMetrics(events, { contextLength } = {}) {
|
|
|
614
628
|
}
|
|
615
629
|
}
|
|
616
630
|
else if (event.kind === RunEventKind.Done) {
|
|
617
|
-
metrics.outcome = outcomeOf(event
|
|
631
|
+
metrics.outcome = outcomeOf(event, last);
|
|
618
632
|
}
|
|
619
633
|
}
|
|
620
634
|
if (events.length > 1) {
|
package/dist/run/run-calls.d.ts
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
import type OpenAI from 'openai';
|
|
2
2
|
import type { CatalogServer } from '../tools/catalog.ts';
|
|
3
3
|
import type { Turn } from '../wire/stream.ts';
|
|
4
|
-
import { type RunArtifact, type ToolAnswer, type ToolCall, type ToolCallOutcome, type ToolCallReport, type ToolCallRequest, type ToolCallResult } from '../wire/tool-calls.ts';
|
|
4
|
+
import { type EndedBy, type RunArtifact, type ToolAnswer, type ToolCall, type ToolCallOutcome, type ToolCallReport, type ToolCallRequest, type ToolCallResult } from '../wire/tool-calls.ts';
|
|
5
5
|
import type { RunEventInput } from './events.ts';
|
|
6
6
|
/**
|
|
7
7
|
* The tools between two turns: a step's calls read, run and written into the transcript.
|
|
@@ -114,13 +114,21 @@ export interface Calling {
|
|
|
114
114
|
* @param parsed - The step's calls, with their arguments read.
|
|
115
115
|
* @param messages - The transcript the results are written into.
|
|
116
116
|
* @param step - Which step this is, for the host told of each result.
|
|
117
|
+
* @returns The first call, in the order asked, whose answer ended the run, or `undefined` where
|
|
118
|
+
* none did.
|
|
117
119
|
*
|
|
118
120
|
* @remarks
|
|
119
121
|
* A failure part-way leaves the transcript well-formed before it is rethrown: a call with no
|
|
120
122
|
* result is a transcript no endpoint takes back, so every call of the step is answered — with what
|
|
121
123
|
* it returned where it had, and otherwise with a line saying it was stopped, or never run.
|
|
124
|
+
*
|
|
125
|
+
* An answer that ends the run does not end the step. The calls after it still run, in whatever
|
|
126
|
+
* batch `parallel` puts them, because the model asked for them in the same breath and how a host
|
|
127
|
+
* schedules calls is not what they mean: a reply that says "write the file, then hand off" in one
|
|
128
|
+
* turn wants the file written whichever order it named the two in, and a call batched beside the
|
|
129
|
+
* end was running already. A tool that must have nothing after it says so in its description.
|
|
122
130
|
*/
|
|
123
|
-
export declare function runCalls(run: Calling, parsed: ReadCall[], messages: OpenAI.ChatCompletionMessageParam[], step: number): Promise<
|
|
131
|
+
export declare function runCalls(run: Calling, parsed: ReadCall[], messages: OpenAI.ChatCompletionMessageParam[], step: number): Promise<EndedBy | undefined>;
|
|
124
132
|
/**
|
|
125
133
|
* Writes a proxied preselection into the transcript as a `load_tools` exchange: the tool array is
|
|
126
134
|
* fixed, so a shortlist has nowhere else to go, and it is answered as though the model had loaded
|
package/dist/run/run-calls.js
CHANGED
|
@@ -128,9 +128,19 @@ function reportFor(run, id, name) {
|
|
|
128
128
|
* A dispatcher's answer in the one shape, whichever it gave.
|
|
129
129
|
*
|
|
130
130
|
* @param answer - What `dispatch` resolved to.
|
|
131
|
-
* @returns The text and the
|
|
131
|
+
* @returns The text, the links and whether it ends the run, a string answer having no links and
|
|
132
|
+
* ending nothing.
|
|
132
133
|
*/
|
|
133
|
-
|
|
134
|
+
function answerOf(answer) {
|
|
135
|
+
if (typeof answer === 'string') {
|
|
136
|
+
return { text: answer, links: [], end: false };
|
|
137
|
+
}
|
|
138
|
+
const { text, links = [], end = false } = answer;
|
|
139
|
+
if (typeof end === 'object') {
|
|
140
|
+
return { text, links, end: { reason: end.reason } };
|
|
141
|
+
}
|
|
142
|
+
return { text, links, end: end ? {} : false };
|
|
143
|
+
}
|
|
134
144
|
/**
|
|
135
145
|
* What a call is shown, counted and dispatched as: a `call_tool` as the tool it names.
|
|
136
146
|
*
|
|
@@ -322,9 +332,9 @@ function batchEnd(run, ready, from, total, answered) {
|
|
|
322
332
|
* @param prepared - The call, with its request built.
|
|
323
333
|
* @param answered - The step's calls already made, by tool and arguments, for `dedupeToolCalls`.
|
|
324
334
|
* @param step - Which step this is, for the artifacts the call's links become.
|
|
325
|
-
* @returns The call's id, the name it was shown under, whether it worked, the whole
|
|
326
|
-
* model reads, and the links its answer carried, where it carried any
|
|
327
|
-
* stopped or a host callback threw.
|
|
335
|
+
* @returns The result — the call's id, the name it was shown under, whether it worked, the whole
|
|
336
|
+
* of what the model reads, and the links its answer carried, where it carried any — and whether
|
|
337
|
+
* the answer ended the run. Rejects where the run was stopped or a host callback threw.
|
|
328
338
|
*/
|
|
329
339
|
async function runCall(run, prepared, answered, step) {
|
|
330
340
|
const { catalog, onDemand, proxied, signal } = run;
|
|
@@ -336,6 +346,7 @@ async function runCall(run, prepared, answered, step) {
|
|
|
336
346
|
const { report, close } = reportFor(run, call.id, name);
|
|
337
347
|
let content;
|
|
338
348
|
let links = [];
|
|
349
|
+
let end = false;
|
|
339
350
|
let ok = true;
|
|
340
351
|
// Answered from an earlier identical call of the step, whose links are already counted.
|
|
341
352
|
let shared = false;
|
|
@@ -364,7 +375,7 @@ async function runCall(run, prepared, answered, step) {
|
|
|
364
375
|
const answer = dedupable
|
|
365
376
|
? await once(answered, key, () => run.dispatch(request, signal, report))
|
|
366
377
|
: await run.dispatch(request, signal, report);
|
|
367
|
-
({ text: content, links } = answerOf(answer));
|
|
378
|
+
({ text: content, links, end } = answerOf(answer));
|
|
368
379
|
}
|
|
369
380
|
}
|
|
370
381
|
catch (error) {
|
|
@@ -384,7 +395,7 @@ async function runCall(run, prepared, answered, step) {
|
|
|
384
395
|
}
|
|
385
396
|
const result = { id: call.id, name, ok, content, ...(links.length ? { links } : {}) };
|
|
386
397
|
run.onToolResult?.(result);
|
|
387
|
-
return result;
|
|
398
|
+
return { result, end };
|
|
388
399
|
}
|
|
389
400
|
/**
|
|
390
401
|
* Runs a step's calls and writes each result into the transcript, in the order asked.
|
|
@@ -393,11 +404,19 @@ async function runCall(run, prepared, answered, step) {
|
|
|
393
404
|
* @param parsed - The step's calls, with their arguments read.
|
|
394
405
|
* @param messages - The transcript the results are written into.
|
|
395
406
|
* @param step - Which step this is, for the host told of each result.
|
|
407
|
+
* @returns The first call, in the order asked, whose answer ended the run, or `undefined` where
|
|
408
|
+
* none did.
|
|
396
409
|
*
|
|
397
410
|
* @remarks
|
|
398
411
|
* A failure part-way leaves the transcript well-formed before it is rethrown: a call with no
|
|
399
412
|
* result is a transcript no endpoint takes back, so every call of the step is answered — with what
|
|
400
413
|
* it returned where it had, and otherwise with a line saying it was stopped, or never run.
|
|
414
|
+
*
|
|
415
|
+
* An answer that ends the run does not end the step. The calls after it still run, in whatever
|
|
416
|
+
* batch `parallel` puts them, because the model asked for them in the same breath and how a host
|
|
417
|
+
* schedules calls is not what they mean: a reply that says "write the file, then hand off" in one
|
|
418
|
+
* turn wants the file written whichever order it named the two in, and a call batched beside the
|
|
419
|
+
* end was running already. A tool that must have nothing after it says so in its description.
|
|
401
420
|
*/
|
|
402
421
|
export async function runCalls(run, parsed, messages, step) {
|
|
403
422
|
const { signal, toolCalls } = run;
|
|
@@ -406,6 +425,7 @@ export async function runCalls(run, parsed, messages, step) {
|
|
|
406
425
|
const answered = new Map();
|
|
407
426
|
// By position, so a call that came back while another was being stopped is still found.
|
|
408
427
|
const outcomes = parsed.map(() => undefined);
|
|
428
|
+
const ends = parsed.map(() => false);
|
|
409
429
|
// How many of the step's calls were handed off, and how many have their result in the
|
|
410
430
|
// transcript. Both count from the front, since results are written in the order asked.
|
|
411
431
|
let begun = 0;
|
|
@@ -436,7 +456,9 @@ export async function runCalls(run, parsed, messages, step) {
|
|
|
436
456
|
const to = batchEnd(run, ready, start, parsed.length, answered);
|
|
437
457
|
begun = to;
|
|
438
458
|
await Promise.all(parsed.slice(start, to).map(async (_, offset) => {
|
|
439
|
-
|
|
459
|
+
const { result, end } = await runCall(run, ready(start + offset), answered, step);
|
|
460
|
+
outcomes[start + offset] = result;
|
|
461
|
+
ends[start + offset] = end;
|
|
440
462
|
}));
|
|
441
463
|
for (const { id, name, ok, content } of outcomes.slice(from, to).flatMap((outcome) => outcome ?? [])) {
|
|
442
464
|
toolCalls.push({ id, name, ok });
|
|
@@ -444,6 +466,15 @@ export async function runCalls(run, parsed, messages, step) {
|
|
|
444
466
|
}
|
|
445
467
|
from = to;
|
|
446
468
|
}
|
|
469
|
+
// The first by position rather than by finishing: a batch's calls end in no order worth
|
|
470
|
+
// reporting, and a deduplicated repeat shares its answer's `end` with the call before it.
|
|
471
|
+
const at = ends.findIndex((end) => end !== false);
|
|
472
|
+
const end = ends[at];
|
|
473
|
+
const outcome = outcomes[at];
|
|
474
|
+
if (end === undefined || end === false || outcome === undefined) {
|
|
475
|
+
return undefined;
|
|
476
|
+
}
|
|
477
|
+
return { callId: outcome.id, name: outcome.name, ...(end.reason === undefined ? {} : { reason: end.reason }) };
|
|
447
478
|
}
|
|
448
479
|
catch (error) {
|
|
449
480
|
// Only a call that was made counts in `toolCalls`.
|
|
@@ -188,6 +188,27 @@ export interface ToolAnswer {
|
|
|
188
188
|
text: string;
|
|
189
189
|
/** What the call produced, in the order the tool gave them. Never sent to the model. */
|
|
190
190
|
links?: ResourceLink[];
|
|
191
|
+
/**
|
|
192
|
+
* Ends the run once this step's calls are answered: no further request is sent, and the loop
|
|
193
|
+
* resolves with `ended` rather than throwing. An object says why, for the host and its `done`.
|
|
194
|
+
*
|
|
195
|
+
* @remarks
|
|
196
|
+
* The step's other calls still run and are answered, so the transcript stays one an endpoint
|
|
197
|
+
* will take back. For a tool whose call is the end of the job — handing off, asking a person —
|
|
198
|
+
* where the alternative is an abort, which the loop reads as a failure.
|
|
199
|
+
*/
|
|
200
|
+
end?: boolean | {
|
|
201
|
+
reason: string;
|
|
202
|
+
};
|
|
203
|
+
}
|
|
204
|
+
/** The call that ended a run through its answer's `end`, as `AgentLoopResult.ended` says it. */
|
|
205
|
+
export interface EndedBy {
|
|
206
|
+
/** The call's id. */
|
|
207
|
+
callId: string;
|
|
208
|
+
/** The tool, under the name it was announced by: for a `call_tool`, the tool it names. */
|
|
209
|
+
name: string;
|
|
210
|
+
/** Why, where the answer said: absent for `end: true`. */
|
|
211
|
+
reason?: string;
|
|
191
212
|
}
|
|
192
213
|
/**
|
|
193
214
|
* The levels a server's log message is sent at, quietest first: MCP's, which are syslog's.
|
package/dist/wire/wire.d.ts
CHANGED
package/dist/wire/wire.js
CHANGED
package/llms.txt
CHANGED
|
@@ -50,6 +50,7 @@ What this package needs to know about a caller's configuration.
|
|
|
50
50
|
- `AgentConfig` (type) — A whole agent configuration — every part, plus the two fields that belong to no group.
|
|
51
51
|
- `Endpoint` (type) — Where to send a request and how long to wait.
|
|
52
52
|
- `ModelParams` (type) — What to ask the model for.
|
|
53
|
+
- `normalizeBaseUrl` — A base URL worth storing: http or https, no credentials, query or fragment, no trailing slash.
|
|
53
54
|
- `RetryPolicy` (type) — How many times a lost or refused request is worth sending again.
|
|
54
55
|
- `ToolDiscovery` — Any one of the modes in `ToolDiscovery`.
|
|
55
56
|
- `ToolPolicy` (type) — How tools reach the model, and how long it may keep calling them.
|
|
@@ -84,6 +85,7 @@ What an endpoint turned out not to support, and answering it when it says so.
|
|
|
84
85
|
- `ClientPoolOptions` (type) — What the client pool is held to across a process.
|
|
85
86
|
- `configureClients` — Changes what the client pool is held to, for a process whose endpoints are not shaped like the deployments these defaults were chosen for.
|
|
86
87
|
- `contextLimitFor` — How much a model will read, in tokens.
|
|
88
|
+
- `EndpointProbeError` — `probeEndpoint` could not list an endpoint's models; `kind` says why.
|
|
87
89
|
- `endpointId` — `endpointKey` hashed, for the remembered facts that can leave the process.
|
|
88
90
|
- `endpointKey` — What counts as one endpoint, everywhere in this package that has to remember something about one — the model listings, `capabilities`, and the no-thinking hints in `side-task`.
|
|
89
91
|
- `firstTokenMs` — The wait for a streamed turn's first chunk, in the SDK's spelling: `undefined` is no limit.
|
|
@@ -92,6 +94,9 @@ What an endpoint turned out not to support, and answering it when it says so.
|
|
|
92
94
|
- `listModels` — Asks an endpoint what it serves, and remembers the answer.
|
|
93
95
|
- `ModelInfo` (type) — A model an endpoint offers, and what it says the model will read.
|
|
94
96
|
- `NO_KEY` — The SDK insists on a non-empty key even where the server will not look at it.
|
|
97
|
+
- `ProbeFailure` — Why `probeEndpoint` could not list an endpoint's models, in words a form can branch on.
|
|
98
|
+
- `ProbeOptions` (type) — What `probeEndpoint` takes besides the endpoint.
|
|
99
|
+
- `probeEndpoint` — Asks an endpoint nobody has saved yet which models it serves, for a form a person is typing it into.
|
|
95
100
|
- `resetClients` — Forgets every cached client and listing, and puts `configureClients` back to the defaults.
|
|
96
101
|
- `resolveApiKey` — The key to send, where an endpoint may inherit one from the settings it overrides.
|
|
97
102
|
- `servedSlots` — How many slots a llama.cpp server runs a model in, for pinning a session to one of them.
|
|
@@ -163,7 +168,7 @@ The loop above a turn: send, run the tools the model asked for, send again, unti
|
|
|
163
168
|
- `PRESELECT_APPEND` — The `preselectRouting` that loads the shortlist after what was carried, and sends the first step like any other.
|
|
164
169
|
- `PRESELECT_EXCLUSIVE` — The `preselectRouting` that sends the first step the shortlist alone.
|
|
165
170
|
- `PRESELECTED_NOTICE` — What opens the notice naming the loop's own picks, ahead of the names.
|
|
166
|
-
- `runAgentLoop` — Runs a question to its answer: one `runTurn` per step, the tools it asks for between them, until a turn asks for none.
|
|
171
|
+
- `runAgentLoop` — Runs a question to its answer: one `runTurn` per step, the tools it asks for between them, until a turn asks for none or a call's answer ends the run.
|
|
167
172
|
- `StepWindow` (type) — How full the window is as a step is about to be sent, as `beforeStep` is told it.
|
|
168
173
|
|
|
169
174
|
### events
|
|
@@ -402,6 +407,7 @@ Reading what a model meant by a tool call when it did not write one cleanly.
|
|
|
402
407
|
|
|
403
408
|
- `ARGUMENTS_MALFORMED` — The `ToolArgumentsError` kind for a call the model wrote wrongly.
|
|
404
409
|
- `ARGUMENTS_TRUNCATED` — The `ToolArgumentsError` kind for a call cut off at the reply ceiling.
|
|
410
|
+
- `EndedBy` (type) — The call that ended a run through its answer's `end`, as `AgentLoopResult.ended` says it.
|
|
405
411
|
- `parseToolArguments` — A tool call's arguments as the object the tool is handed.
|
|
406
412
|
- `ResourceLink` (type) — A handle to something a tool call made or found — a document it wrote, a file it saved.
|
|
407
413
|
- `RunArtifact` (type) — One thing a run's tool calls produced: the first link to a uri, and where it came from.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@cubicecho/agent-core",
|
|
3
|
-
"version": "4.
|
|
3
|
+
"version": "4.12.0",
|
|
4
4
|
"description": "The endpoint-agnostic half of an OpenAI-compatible agent loop: tool-schema compatibility, on-demand tool loading, one-shot side tasks, run events, and a pooled client.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"openai",
|