@cubicecho/agent-core 4.18.0 → 4.20.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +21 -8
- package/dist/index.d.ts +3 -3
- package/dist/run/agent-loop.d.ts +33 -6
- package/dist/run/agent-loop.js +37 -28
- package/dist/run/events.d.ts +55 -7
- package/dist/run/events.js +12 -4
- package/dist/run/run-calls.d.ts +15 -7
- package/dist/run/run-calls.js +72 -16
- package/dist/wire/errors.d.ts +8 -5
- package/dist/wire/stream.d.ts +54 -0
- package/dist/wire/stream.js +25 -0
- package/dist/wire/tool-calls.d.ts +16 -1
- package/llms.txt +4 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -610,8 +610,10 @@ string. What still is not an object throws a `ToolArgumentsError` whose `kind` i
|
|
|
610
610
|
the turn stopped at the ceiling, with a message telling the model so, and `malformed` otherwise.
|
|
611
611
|
The repaired JSON is what the transcript keeps, and an unreadable call is replayed as `{}`, because
|
|
612
612
|
a server that parses replayed arguments refuses the originals on every later request. `dispatch`
|
|
613
|
-
is still handed the model's own text as `raw`,
|
|
614
|
-
|
|
613
|
+
is still handed the model's own text as `raw`, with `repaired` true where reading it took a repair
|
|
614
|
+
— whitespace and blank arguments do not count — and the dedupe compares repaired arguments, so
|
|
615
|
+
`{'a': 1}` and `{"a": 1}` are one call. A call recovered from text, below, is always `repaired`,
|
|
616
|
+
and its `raw` is the arguments recovery read out of the reply, as JSON.
|
|
615
617
|
|
|
616
618
|
A server whose tool-call parser was written for another template streams the model's call as
|
|
617
619
|
plain text, and the run ends on a reply that is nothing but a call nobody made. Unless
|
|
@@ -826,7 +828,12 @@ failure short of a stop, the cut-off answer is kept with a notice. The cap is on
|
|
|
826
828
|
continue, one to find out.
|
|
827
829
|
|
|
828
830
|
Every turn ends in a `usage` event, whether or not the endpoint reported tokens. Its totals are the
|
|
829
|
-
run's so far
|
|
831
|
+
run's so far — `promptTokens`, `completionTokens`, `totalTokens`, `cachedTokens` and
|
|
832
|
+
`cacheReported`, the `UsageTotals` spelling that `result.usage` and a failed run's `usage` share
|
|
833
|
+
(their older `prompt`, `completion`, `total` and `cached` are still there, deprecated). A
|
|
834
|
+
`cachedTokens` of zero is a miss only where `cacheReported` is true; false is an endpoint that never
|
|
835
|
+
said. Its `turn` is that turn's own `TurnUsage` and `finishReason`, with the `contextLength` the loop
|
|
836
|
+
was told. The loop adds what only
|
|
830
837
|
it can know: `toolsDeclared` and `toolSchemaTokens` for the tool block it sent, and, from the second
|
|
831
838
|
request on, `cacheExpected` (the previous prompt plus its reply) and — where a cache count was
|
|
832
839
|
reported — `cacheBroken`, a hit short of 90% of the previous prompt. A broken cache is given a
|
|
@@ -855,10 +862,12 @@ a lesser body and a continuation with the reply so far on the end, and neither i
|
|
|
855
862
|
|
|
856
863
|
`runMetrics(events)` adds a run up from those events — tokens, cache hit ratio and breaks by
|
|
857
864
|
reason, prefill, decode and tool time, the slowest turn, mean time to first token, draft
|
|
858
|
-
acceptance, the largest prompt against
|
|
859
|
-
|
|
860
|
-
|
|
861
|
-
|
|
865
|
+
acceptance, the largest prompt against its window, turns cut off, tool errors by name, what
|
|
866
|
+
`load_tools` resolved to (`toolsLoaded`, `redundantLoads`, `unknownToolNames`, off each load's
|
|
867
|
+
result as `loads`), and an `outcome`. The loop returns it as `metrics`. Every turn carries its own
|
|
868
|
+
window and every load its own counts, so a host that runs one loop per step gets the whole run from
|
|
869
|
+
`runMetrics` over all the loops' events, with no per-loop result to merge. It is a sibling of `fold`
|
|
870
|
+
rather than part of it, since `fold`'s blocks are for display and a summary is not one:
|
|
862
871
|
|
|
863
872
|
```ts
|
|
864
873
|
const metrics = runMetrics(history(runId), { contextLength: config.contextLength });
|
|
@@ -962,7 +971,11 @@ try {
|
|
|
962
971
|
|
|
963
972
|
`onMessage` hears the assistant message as it will be replayed — recovered calls folded in, and
|
|
964
973
|
arguments repaired, which the turn `onTurn` is handed does not show — together with that `Turn`,
|
|
965
|
-
and then each tool result, in transcript order.
|
|
974
|
+
and then each tool result, in transcript order. Beside an assistant message the model wrote comes a
|
|
975
|
+
fourth argument, `{ raw }`: the same message as the model produced it, its text whole with any call
|
|
976
|
+
written into it left there, and its calls' arguments exactly as streamed, under the same ids. Store
|
|
977
|
+
or show `raw` to keep what the model said; replay `message` to keep what an endpoint will take back.
|
|
978
|
+
The loop's own messages — a proxied run's preselected load — come without it. `onTurn` is not awaited and never sees a tool
|
|
966
979
|
result; this is the one to store from. What it returns is ignored, since `beforeStep` is the one way to rewrite
|
|
967
980
|
the transcript, and what it throws ends the run.
|
|
968
981
|
|
package/dist/index.d.ts
CHANGED
|
@@ -19,8 +19,8 @@ export { isGrammarError, relaxSchema, relaxTools, sanitizeSchema, sanitizeTools,
|
|
|
19
19
|
export { CACHE_PROMPT_FIELD, resetSlots, SESSION_HEADER, SLOT_FIELD, type SlotAllocator, slotAllocator, } from './endpoint/slots.ts';
|
|
20
20
|
export { HOOK_EVENTS, HookEvent, INJECT_EVENTS } from './hooks/hook-events.ts';
|
|
21
21
|
export { assembleContext, configureHooks, consult, type Gathered, gather, HOOK_CONTEXT_TOKENS, HOOK_PREFACE, type HookContext, type HookMessage, type HookNote, type HookOptions, type HookOutcome, type HookRunner, notify, resetHooks, turnIndex, turnMessages, UNTRUSTED_PREFACE, untrusted, withContext, } from './hooks/hooks.ts';
|
|
22
|
-
export { type AgentLoopHooks, type AgentLoopOptions, type AgentLoopRequest, type AgentLoopResult, PRESELECT_APPEND, PRESELECT_EXCLUSIVE, PRESELECTED_NOTICE, runAgentLoop, type StepWindow, } from './run/agent-loop.ts';
|
|
23
|
-
export { configureEvents, type EventBusOptions, emit, endRun, fold, history, type RunEvent, type RunEventInput, RunEventKind, type RunMetrics, type RunMetricsOptions, RunOutcome, type RunUsage, resetEvents, runMetrics, type TurnReport, watch, } from './run/events.ts';
|
|
22
|
+
export { type AgentLoopHooks, type AgentLoopOptions, type AgentLoopRequest, type AgentLoopResult, PRESELECT_APPEND, PRESELECT_EXCLUSIVE, PRESELECTED_NOTICE, runAgentLoop, type StepWindow, type WrittenMessage, } from './run/agent-loop.ts';
|
|
23
|
+
export { configureEvents, type EventBusOptions, emit, endRun, fold, history, type LoadCounts, type RunEvent, type RunEventInput, RunEventKind, type RunMetrics, type RunMetricsOptions, RunOutcome, type RunUsage, resetEvents, runMetrics, type TurnReport, watch, } from './run/events.ts';
|
|
24
24
|
export { buildBody } from './run/request-body.ts';
|
|
25
25
|
export { preview, type ToolDispatch } from './run/run-calls.ts';
|
|
26
26
|
export { resetAll } from './runtime/reset.ts';
|
|
@@ -37,7 +37,7 @@ export { type RunTurnOptions, runTurn } from './turn/run-turn.ts';
|
|
|
37
37
|
export { type AskJsonOptions, ask, askJson, askReply, clean, listLines, parseJson, resetHints, type SideTask, type SideTaskInput, type SideTaskOptions, type SideTaskReply, taskCall, tryAsk, } from './turn/side-task.ts';
|
|
38
38
|
export { AgentLoopError, type AgentLoopFailure, AgentLoopOverflow, errorMessage, failedRun, ToolIterationLimit, } from './wire/errors.ts';
|
|
39
39
|
export { backoffMs, ContextOverflow, EndpointSilent, isModelLoading, isOverflow, isTransient, SMALLEST_LIKELY_WINDOW, } from './wire/retry.ts';
|
|
40
|
-
export { CacheBreakReason, type InFlight, type PartialTurn, type Produced, type StreamTurnOptions, streamTurn, type Turn, type TurnUsage, } from './wire/stream.ts';
|
|
40
|
+
export { CacheBreakReason, type InFlight, type LoopUsage, type PartialTurn, type Produced, type StreamTurnOptions, streamTurn, type Turn, type TurnUsage, type UsageTotals, } from './wire/stream.ts';
|
|
41
41
|
export { ALL_FENCES, DEFAULT_FENCES, type Fence, FenceSplitter, type FenceSplitterOptions, SPLIT_OUTPUT, SPLIT_REASONING, type Split, stripThinking, THINK_FENCE, } from './wire/thinking.ts';
|
|
42
42
|
export { type ContextBreakdown, type ContextBreakdownOptions, compact, contextChars, contextTokens, estimateTokens, messageTokens, requestChars, requestTokens, type TokenEstimateOptions, } from './wire/tokens.ts';
|
|
43
43
|
export { ARGUMENTS_MALFORMED, ARGUMENTS_TRUNCATED, type EndedBy, parseToolArguments, type ResourceLink, type RunArtifact, recoverToolCalls, type ServerLogEntry, ServerLogLevel, type ToolAnswer, ToolArgumentsError, type ToolCall, type ToolCallOutcome, type ToolCallReport, type ToolCallRequest, type ToolCallResult, } from './wire/tool-calls.ts';
|
package/dist/run/agent-loop.d.ts
CHANGED
|
@@ -9,7 +9,7 @@ import { type ServerInstructionsOptions } from '../tools/instructions.ts';
|
|
|
9
9
|
import { type KeywordPreselectOptions } from '../tools/preselect.ts';
|
|
10
10
|
import { type ToolOrder } from '../tools/tool-loading.ts';
|
|
11
11
|
import { type SideTask } from '../turn/side-task.ts';
|
|
12
|
-
import { type
|
|
12
|
+
import { type LoopUsage, type Turn } from '../wire/stream.ts';
|
|
13
13
|
import { type EndedBy, type RunArtifact, type ToolCallOutcome, type ToolCallRequest, type ToolCallResult } from '../wire/tool-calls.ts';
|
|
14
14
|
import { type RunEventInput, type RunMetrics } from './events.ts';
|
|
15
15
|
import { type ToolDispatch } from './run-calls.ts';
|
|
@@ -23,6 +23,24 @@ import { type ToolDispatch } from './run-calls.ts';
|
|
|
23
23
|
* and two never did. What is here is the part that does not know what the run is for; the
|
|
24
24
|
* prompts, the tools and what a run means stay with the caller.
|
|
25
25
|
*/
|
|
26
|
+
/**
|
|
27
|
+
* What `onMessage` is handed beside an assistant message the model wrote: that message as the model
|
|
28
|
+
* produced it.
|
|
29
|
+
*
|
|
30
|
+
* @remarks
|
|
31
|
+
* The message `onMessage` is handed is what later requests replay — arguments repaired, `{}` for
|
|
32
|
+
* none, recovered calls taken out of the text and made calls — which is right to send and quietly
|
|
33
|
+
* wrong to show as the model's. Absent for a message the loop wrote itself, such as a proxied
|
|
34
|
+
* run's preselected load.
|
|
35
|
+
*/
|
|
36
|
+
export interface WrittenMessage {
|
|
37
|
+
/**
|
|
38
|
+
* The reply as it streamed: its text whole, a call written into it as text left there, and the
|
|
39
|
+
* calls it sent as calls with their arguments as streamed. A call that never got a name is left
|
|
40
|
+
* off, since nothing answers it. Ids are the turn's, so the tool results pair with either.
|
|
41
|
+
*/
|
|
42
|
+
raw: OpenAI.ChatCompletionAssistantMessageParam;
|
|
43
|
+
}
|
|
26
44
|
/**
|
|
27
45
|
* One step's request as `onRequest` is handed it: what the model is about to read.
|
|
28
46
|
*
|
|
@@ -459,8 +477,12 @@ export interface AgentLoopOptions {
|
|
|
459
477
|
* a proxied run writes for its `preselected` shortlist, told as step zero's before any request.
|
|
460
478
|
* What it returns is ignored: `beforeStep` is the one way to rewrite the transcript. What it
|
|
461
479
|
* throws ends the run, and it is not called again for that run.
|
|
480
|
+
*
|
|
481
|
+
* Beside an assistant message the model wrote comes `written`, whose `raw` is the message as the
|
|
482
|
+
* model produced it, for a host that shows or stores what was said rather than what is replayed.
|
|
483
|
+
* See `WrittenMessage`.
|
|
462
484
|
*/
|
|
463
|
-
onMessage?: (message: OpenAI.ChatCompletionMessageParam, step: number, turn?: Turn) => void | Promise<void>;
|
|
485
|
+
onMessage?: (message: OpenAI.ChatCompletionMessageParam, step: number, turn?: Turn, written?: WrittenMessage) => void | Promise<void>;
|
|
464
486
|
/**
|
|
465
487
|
* How many times an answer cut off at `maxTokens` is continued, zero for never. Opt-in because
|
|
466
488
|
* it spends another request, and on a server that does not continue a trailing assistant
|
|
@@ -497,8 +519,11 @@ export interface AgentLoopResult {
|
|
|
497
519
|
ended?: EndedBy;
|
|
498
520
|
/** The transcript, with every assistant turn and tool result the run added. No system prompt. */
|
|
499
521
|
messages: OpenAI.ChatCompletionMessageParam[];
|
|
500
|
-
/**
|
|
501
|
-
|
|
522
|
+
/**
|
|
523
|
+
* Summed over every turn of the run, in the spelling the `usage` events use, with whether any
|
|
524
|
+
* turn reported a cache count. The four counts are also under their older names.
|
|
525
|
+
*/
|
|
526
|
+
usage: LoopUsage;
|
|
502
527
|
/**
|
|
503
528
|
* Every call, `load_tools` included, in the order they were made, each under its call id — a
|
|
504
529
|
* proxied preselection's among them, first.
|
|
@@ -550,8 +575,10 @@ export interface AgentLoopResult {
|
|
|
550
575
|
*/
|
|
551
576
|
afterTurn: Promise<HookNote[]>;
|
|
552
577
|
/**
|
|
553
|
-
* The run summed and derived: what `runMetrics` makes of the events this loop emitted,
|
|
554
|
-
* `load_tools`
|
|
578
|
+
* The run summed and derived: what `runMetrics` makes of the events this loop emitted, with the
|
|
579
|
+
* `load_tools` counts at zero rather than absent in a run on demand, `wallMs` from the call to
|
|
580
|
+
* the return, and `outcome`. A host running several loops for one run gets its counts from
|
|
581
|
+
* `runMetrics` over all their events, without these results beside it.
|
|
555
582
|
*/
|
|
556
583
|
metrics: RunMetrics;
|
|
557
584
|
/**
|
package/dist/run/agent-loop.js
CHANGED
|
@@ -16,13 +16,13 @@ import { runTurn } from '../turn/run-turn.js';
|
|
|
16
16
|
import { taskCall, tryAsk } from '../turn/side-task.js';
|
|
17
17
|
import { AgentLoopError, AgentLoopOverflow, errorMessage, ToolIterationLimit, } from '../wire/errors.js';
|
|
18
18
|
import { ContextOverflow } from '../wire/retry.js';
|
|
19
|
-
import { addCounts, noUsage } from '../wire/stream.js';
|
|
19
|
+
import { addCounts, loopUsage, noUsage, reportsCache, } from '../wire/stream.js';
|
|
20
20
|
import { contextTokens, toolsChars } from '../wire/tokens.js';
|
|
21
21
|
import { recoverToolCalls, } from '../wire/tool-calls.js';
|
|
22
22
|
import { FinishReason, PartType, Role } from '../wire/wire.js';
|
|
23
23
|
import { RunEventKind, RunOutcome, runMetrics, stamp, } from './events.js';
|
|
24
24
|
import { buildBody } from './request-body.js';
|
|
25
|
-
import { assistantMessage, definedAs, loadShortlist, readCalls, runCalls, } from './run-calls.js';
|
|
25
|
+
import { assistantMessage, definedAs, loadShortlist, readCalls, runCalls, writtenMessage, } from './run-calls.js';
|
|
26
26
|
/** The `preselectRouting` that sends the first step the shortlist alone. */
|
|
27
27
|
export const PRESELECT_EXCLUSIVE = 'exclusive';
|
|
28
28
|
/** The `preselectRouting` that loads the shortlist after what was carried, and sends the first step like any other. */
|
|
@@ -127,7 +127,7 @@ export async function runAgentLoop(options) {
|
|
|
127
127
|
const standing = {
|
|
128
128
|
read: () => ({
|
|
129
129
|
messages: [...options.messages],
|
|
130
|
-
usage: noUsage(),
|
|
130
|
+
usage: loopUsage(noUsage(), false),
|
|
131
131
|
toolCalls: [],
|
|
132
132
|
artifacts: [],
|
|
133
133
|
loaded: [],
|
|
@@ -502,8 +502,8 @@ function definitionsByName(tools) {
|
|
|
502
502
|
* Offers each message written into the transcript to the host.
|
|
503
503
|
*
|
|
504
504
|
* @param onMessage - The host's callback, awaited. Absent offers nothing.
|
|
505
|
-
* @returns The offer, taking a message, its step
|
|
506
|
-
* host has taken the message, and rejects with what the host threw — once.
|
|
505
|
+
* @returns The offer, taking a message, its step, the turn it came with and what the model wrote:
|
|
506
|
+
* it resolves once the host has taken the message, and rejects with what the host threw — once.
|
|
507
507
|
*
|
|
508
508
|
* @remarks
|
|
509
509
|
* Once `onMessage` has thrown the run is ending on that, and the results still to be written are
|
|
@@ -511,12 +511,12 @@ function definitionsByName(tools) {
|
|
|
511
511
|
*/
|
|
512
512
|
function announcer(onMessage) {
|
|
513
513
|
let heard = true;
|
|
514
|
-
return async (message, step, turn) => {
|
|
514
|
+
return async (message, step, turn, written) => {
|
|
515
515
|
if (!onMessage || !heard) {
|
|
516
516
|
return;
|
|
517
517
|
}
|
|
518
518
|
try {
|
|
519
|
-
await onMessage(message, step, turn);
|
|
519
|
+
await onMessage(message, step, turn, written);
|
|
520
520
|
}
|
|
521
521
|
catch (error) {
|
|
522
522
|
heard = false;
|
|
@@ -530,19 +530,25 @@ function announcer(onMessage) {
|
|
|
530
530
|
* @param usage - The run's totals, this turn's already added.
|
|
531
531
|
* @param turn - The turn just finished, read for its usage and why it ended.
|
|
532
532
|
* @param ledger - The ledger as it stands. Carried on the event as it is, not copied.
|
|
533
|
+
* @param [contextLength] - The window the turn was served. Absent or zero leaves it off the report.
|
|
533
534
|
* @returns The event, for the caller to emit.
|
|
534
535
|
*/
|
|
535
|
-
const usageEvent = (usage, turn, ledger) =>
|
|
536
|
-
|
|
537
|
-
|
|
538
|
-
|
|
539
|
-
|
|
540
|
-
|
|
541
|
-
|
|
542
|
-
|
|
543
|
-
|
|
544
|
-
|
|
545
|
-
|
|
536
|
+
const usageEvent = (usage, turn, ledger, contextLength) => {
|
|
537
|
+
const { promptTokens, completionTokens, totalTokens, cachedTokens, cacheReported } = usage;
|
|
538
|
+
const window = contextLength ? { contextLength } : {};
|
|
539
|
+
return {
|
|
540
|
+
kind: RunEventKind.Usage,
|
|
541
|
+
usage: {
|
|
542
|
+
promptTokens,
|
|
543
|
+
completionTokens,
|
|
544
|
+
totalTokens,
|
|
545
|
+
cachedTokens,
|
|
546
|
+
cacheReported,
|
|
547
|
+
turn: { ...turn.usage, finishReason: turn.finishReason, ...window },
|
|
548
|
+
ledger,
|
|
549
|
+
},
|
|
550
|
+
};
|
|
551
|
+
};
|
|
546
552
|
/**
|
|
547
553
|
* The calls a turn made and the answer beside them, with calls the model wrote as text taken out
|
|
548
554
|
* of the answer and counted as calls.
|
|
@@ -552,21 +558,21 @@ const usageEvent = (usage, turn, ledger) => ({
|
|
|
552
558
|
* nothing is looked for in a turn that made a named call of its own or has no text.
|
|
553
559
|
* @param notice - Told when calls were recovered, and how many.
|
|
554
560
|
* @returns `calls`, the turn's own less any with no name, and `content`, its text as it was — or,
|
|
555
|
-
* where calls were recovered, those calls and the text without them.
|
|
561
|
+
* where calls were recovered, those calls and the text without them, and `recovered` true.
|
|
556
562
|
*/
|
|
557
563
|
function turnCalls(turn, recoverable, notice) {
|
|
558
564
|
// A call with no name is a fragment the server never finished sending: nothing to run, and
|
|
559
565
|
// an assistant message naming it would be answered by nothing.
|
|
560
566
|
const calls = turn.toolCalls.filter((call) => call.function.name);
|
|
561
567
|
if (!recoverable || calls.length || !turn.content) {
|
|
562
|
-
return { calls, content: turn.content };
|
|
568
|
+
return { calls, content: turn.content, recovered: false };
|
|
563
569
|
}
|
|
564
570
|
const recovered = recoverToolCalls(turn.content, { names: recoverable });
|
|
565
571
|
if (!recovered.toolCalls.length) {
|
|
566
|
-
return { calls, content: turn.content };
|
|
572
|
+
return { calls, content: turn.content, recovered: false };
|
|
567
573
|
}
|
|
568
574
|
notice(`recovered ${counted(recovered.toolCalls.length, 'tool call')} the model wrote as text; the server's tool-call parser does not match this model's template`);
|
|
569
|
-
return { calls: recovered.toolCalls, content: recovered.content };
|
|
575
|
+
return { calls: recovered.toolCalls, content: recovered.content, recovered: true };
|
|
570
576
|
}
|
|
571
577
|
/**
|
|
572
578
|
* The steps of `runAgentLoop`, throwing what they caught as it was caught.
|
|
@@ -629,6 +635,7 @@ async function runSteps(options, standing, pin) {
|
|
|
629
635
|
const byName = (names) => definedAs(definitions, names);
|
|
630
636
|
let messages = [...options.messages];
|
|
631
637
|
let usage = noUsage();
|
|
638
|
+
let cacheReported = false;
|
|
632
639
|
const toolCalls = [];
|
|
633
640
|
const artifacts = new Map();
|
|
634
641
|
// A box per request, emptied once its turn is in `messages`: a failure while the tools run had
|
|
@@ -639,7 +646,7 @@ async function runSteps(options, standing, pin) {
|
|
|
639
646
|
return {
|
|
640
647
|
messages,
|
|
641
648
|
...(partial ? { partial } : {}),
|
|
642
|
-
usage,
|
|
649
|
+
usage: loopUsage(usage, cacheReported),
|
|
643
650
|
toolCalls,
|
|
644
651
|
artifacts: [...artifacts.values()],
|
|
645
652
|
// As the result would have said it: proxied, nothing is loaded that the history does not hold.
|
|
@@ -837,20 +844,22 @@ async function runSteps(options, standing, pin) {
|
|
|
837
844
|
completion: turn.usage.completion,
|
|
838
845
|
};
|
|
839
846
|
usage = addCounts(usage, turn.usage);
|
|
840
|
-
|
|
847
|
+
cacheReported ||= reportsCache(turn.usage);
|
|
848
|
+
onEvent(usageEvent(loopUsage(usage, cacheReported), turn, ledger, config.contextLength));
|
|
841
849
|
if (turn.finishReason === FinishReason.Length) {
|
|
842
850
|
notice(`the model stopped at maxTokens (${config.maxTokens}); this turn is cut short`);
|
|
843
851
|
}
|
|
844
852
|
const canRecover = recover && (tools.length > 0 || hostTools.length > 0 || onDemand);
|
|
845
|
-
const { calls, content } = turnCalls(turn, canRecover ? recoverable : undefined, notice);
|
|
853
|
+
const { calls, content, recovered } = turnCalls(turn, canRecover ? recoverable : undefined, notice);
|
|
846
854
|
const shown = { ...turn, content, toolCalls: calls };
|
|
847
855
|
onTurn?.(shown, step);
|
|
848
856
|
// Read before the assistant message is written, so what is replayed on every later request
|
|
849
857
|
// is the repaired JSON.
|
|
850
|
-
const parsed = readCalls(calls, turn.finishReason);
|
|
858
|
+
const parsed = readCalls(calls, turn.finishReason, recovered);
|
|
851
859
|
const assistant = assistantMessage(content, parsed);
|
|
852
860
|
messages.push(assistant);
|
|
853
|
-
|
|
861
|
+
// Recovered calls were never calls on the wire: what the model said is the text they came from.
|
|
862
|
+
await announce(assistant, step, shown, { raw: writtenMessage(turn.content, recovered ? [] : calls) });
|
|
854
863
|
const isAnswer = calls.length === 0;
|
|
855
864
|
const ended = isAnswer ? undefined : await runCalls(calling, parsed, messages, step);
|
|
856
865
|
if (ended) {
|
|
@@ -873,7 +882,7 @@ async function runSteps(options, standing, pin) {
|
|
|
873
882
|
turn: shown,
|
|
874
883
|
...(ended ? { ended } : {}),
|
|
875
884
|
messages,
|
|
876
|
-
usage,
|
|
885
|
+
usage: loopUsage(usage, cacheReported),
|
|
877
886
|
toolCalls,
|
|
878
887
|
artifacts: [...artifacts.values()],
|
|
879
888
|
loaded: proxied ? [] : [...loaded],
|
package/dist/run/events.d.ts
CHANGED
|
@@ -121,6 +121,10 @@ export type RunEventKind = (typeof RunEventKind)[keyof typeof RunEventKind];
|
|
|
121
121
|
* What a run has spent, counted from the start of the run rather than for the turn that
|
|
122
122
|
* carried it: a client draws the latest one it has seen and needs no arithmetic of its own,
|
|
123
123
|
* and one lost to the backlog cap costs nothing because the next supersedes it.
|
|
124
|
+
*
|
|
125
|
+
* @remarks
|
|
126
|
+
* The counts are `UsageTotals`, the spelling `AgentLoopResult.usage` shares, with the cache's two
|
|
127
|
+
* fields optional so a host emitting usage of its own need not know them.
|
|
124
128
|
*/
|
|
125
129
|
export interface RunUsage {
|
|
126
130
|
promptTokens: number;
|
|
@@ -131,6 +135,12 @@ export interface RunUsage {
|
|
|
131
135
|
* emitting usage before this field existed still compiles; absent reads the same as zero.
|
|
132
136
|
*/
|
|
133
137
|
cachedTokens?: number;
|
|
138
|
+
/**
|
|
139
|
+
* Whether any turn so far reported a cache count, as `UsageTotals.cacheReported` says it — so a
|
|
140
|
+
* `cachedTokens` of zero reads as a miss or as nothing said. `runAgentLoop` always sets it;
|
|
141
|
+
* optional for the reason `cachedTokens` is, and absent reads as not known.
|
|
142
|
+
*/
|
|
143
|
+
cacheReported?: boolean;
|
|
134
144
|
/**
|
|
135
145
|
* The turn that carried this report, on its own rather than added in — the half `runMetrics`
|
|
136
146
|
* reads, since a total says nothing about which turn was slow or where the cache broke. Absent
|
|
@@ -148,6 +158,30 @@ export interface RunUsage {
|
|
|
148
158
|
export interface TurnReport extends TurnUsage {
|
|
149
159
|
/** Why the model stopped, as `Turn.finishReason` says it; `length` is a turn cut short. */
|
|
150
160
|
finishReason: string;
|
|
161
|
+
/**
|
|
162
|
+
* The window the turn was served, where the loop was told one, for `largestPromptShare`.
|
|
163
|
+
*
|
|
164
|
+
* @remarks
|
|
165
|
+
* On the turn rather than only in `runMetrics`' options, because a run of several loops can put
|
|
166
|
+
* them on models with different windows, and one window for the whole run misstates how close
|
|
167
|
+
* any of them came.
|
|
168
|
+
*/
|
|
169
|
+
contextLength?: number;
|
|
170
|
+
}
|
|
171
|
+
/**
|
|
172
|
+
* What `load_tools` calls resolved to: new tools, ones already loaded, and names in no catalogue.
|
|
173
|
+
*
|
|
174
|
+
* @remarks
|
|
175
|
+
* Carried on each `load_tools` result as `loads`, so `runMetrics` over every loop's events adds
|
|
176
|
+
* them up without the loops' results beside it.
|
|
177
|
+
*/
|
|
178
|
+
export interface LoadCounts {
|
|
179
|
+
/** Tools loaded that were not loaded already. */
|
|
180
|
+
toolsLoaded: number;
|
|
181
|
+
/** Tools asked for that were loaded already. */
|
|
182
|
+
redundantLoads: number;
|
|
183
|
+
/** Names asked for that are in no catalogue. */
|
|
184
|
+
unknownToolNames: number;
|
|
151
185
|
}
|
|
152
186
|
/**
|
|
153
187
|
* One thing that happened in a run, as a watcher receives it.
|
|
@@ -202,6 +236,11 @@ export interface RunEvent {
|
|
|
202
236
|
* `id` is.
|
|
203
237
|
*/
|
|
204
238
|
level?: string;
|
|
239
|
+
/**
|
|
240
|
+
* What the call loaded, on a `load_tools` `tool-result` and on no other event. Absent from one
|
|
241
|
+
* whose arguments did not parse, which loaded nothing. Optional for the reason `id` is.
|
|
242
|
+
*/
|
|
243
|
+
loads?: LoadCounts;
|
|
205
244
|
}
|
|
206
245
|
/**
|
|
207
246
|
* What `emit` is given: the run and the sequence are the bus's to assign.
|
|
@@ -334,13 +373,14 @@ export interface RunMetrics {
|
|
|
334
373
|
/** `server-log` events at `error` or above: `critical`, `alert` and `emergency` with it. */
|
|
335
374
|
serverErrors: number;
|
|
336
375
|
/**
|
|
337
|
-
* Tools `load_tools` loaded that were not loaded already
|
|
338
|
-
*
|
|
376
|
+
* Tools `load_tools` loaded that were not loaded already, summed from each result's `loads`.
|
|
377
|
+
* Absent where no result carried any; `runAgentLoop` fills all three with zero in a run on
|
|
378
|
+
* demand that loaded nothing.
|
|
339
379
|
*/
|
|
340
380
|
toolsLoaded?: number;
|
|
341
|
-
/** Tools the model asked `load_tools` for that it already had,
|
|
381
|
+
/** Tools the model asked `load_tools` for that it already had, summed the same way. */
|
|
342
382
|
redundantLoads?: number;
|
|
343
|
-
/** Names the model asked `load_tools` for that are in no catalogue,
|
|
383
|
+
/** Names the model asked `load_tools` for that are in no catalogue, summed the same way. */
|
|
344
384
|
unknownToolNames?: number;
|
|
345
385
|
promptTokens: number;
|
|
346
386
|
completionTokens: number;
|
|
@@ -382,7 +422,10 @@ export interface RunMetrics {
|
|
|
382
422
|
draftAcceptance?: number;
|
|
383
423
|
/** The biggest prompt any turn reported. */
|
|
384
424
|
largestPrompt?: number;
|
|
385
|
-
/**
|
|
425
|
+
/**
|
|
426
|
+
* The highest share of its window any turn's prompt took, for how close the run came: each
|
|
427
|
+
* turn over its own `contextLength`, or over the one `runMetrics` was told where it has none.
|
|
428
|
+
*/
|
|
386
429
|
largestPromptShare?: number;
|
|
387
430
|
/** How it ended, read off `done` and the last turn. */
|
|
388
431
|
outcome?: RunOutcome;
|
|
@@ -408,7 +451,10 @@ export declare const RunOutcome: {
|
|
|
408
451
|
export type RunOutcome = (typeof RunOutcome)[keyof typeof RunOutcome];
|
|
409
452
|
/** What `runMetrics` takes besides the events. */
|
|
410
453
|
export interface RunMetricsOptions {
|
|
411
|
-
/**
|
|
454
|
+
/**
|
|
455
|
+
* The window the run was served, for `largestPromptShare` over the turns that do not carry
|
|
456
|
+
* their own. Absent or zero leaves those out.
|
|
457
|
+
*/
|
|
412
458
|
contextLength?: number;
|
|
413
459
|
}
|
|
414
460
|
/**
|
|
@@ -425,6 +471,8 @@ export interface RunMetricsOptions {
|
|
|
425
471
|
* what it returns as blocks; a summary is another shape, and folding one in would give every
|
|
426
472
|
* consumer of `fold` a block it does not know how to draw. Derived from the `usage` reports' own
|
|
427
473
|
* `turn` rather than the running totals, so a run with several loops in it — a question per loop —
|
|
428
|
-
* adds up the same as one with a single loop.
|
|
474
|
+
* adds up the same as one with a single loop. That holds for everything here: what `load_tools`
|
|
475
|
+
* resolved to rides on its results, and each turn carries the window it was served, so the events
|
|
476
|
+
* of every loop of a run, concatenated, are a complete account of it.
|
|
429
477
|
*/
|
|
430
478
|
export declare function runMetrics(events: RunEvent[], { contextLength }?: RunMetricsOptions): RunMetrics;
|
package/dist/run/events.js
CHANGED
|
@@ -522,7 +522,9 @@ function outcomeOf({ ok, name }, last) {
|
|
|
522
522
|
* what it returns as blocks; a summary is another shape, and folding one in would give every
|
|
523
523
|
* consumer of `fold` a block it does not know how to draw. Derived from the `usage` reports' own
|
|
524
524
|
* `turn` rather than the running totals, so a run with several loops in it — a question per loop —
|
|
525
|
-
* adds up the same as one with a single loop.
|
|
525
|
+
* adds up the same as one with a single loop. That holds for everything here: what `load_tools`
|
|
526
|
+
* resolved to rides on its results, and each turn carries the window it was served, so the events
|
|
527
|
+
* of every loop of a run, concatenated, are a complete account of it.
|
|
526
528
|
*/
|
|
527
529
|
export function runMetrics(events, { contextLength } = {}) {
|
|
528
530
|
const metrics = {
|
|
@@ -570,6 +572,11 @@ export function runMetrics(events, { contextLength } = {}) {
|
|
|
570
572
|
if (event.ok === false) {
|
|
571
573
|
metrics.toolErrors[event.name] = (metrics.toolErrors[event.name] ?? 0) + 1;
|
|
572
574
|
}
|
|
575
|
+
if (event.loads) {
|
|
576
|
+
add('toolsLoaded', event.loads.toolsLoaded);
|
|
577
|
+
add('redundantLoads', event.loads.redundantLoads);
|
|
578
|
+
add('unknownToolNames', event.loads.unknownToolNames);
|
|
579
|
+
}
|
|
573
580
|
const called = pending.get(event.name)?.shift();
|
|
574
581
|
if (called !== undefined) {
|
|
575
582
|
add('toolMs', event.at - called);
|
|
@@ -625,6 +632,10 @@ export function runMetrics(events, { contextLength } = {}) {
|
|
|
625
632
|
}
|
|
626
633
|
if (turn.prompt > 0) {
|
|
627
634
|
metrics.largestPrompt = Math.max(metrics.largestPrompt ?? 0, turn.prompt);
|
|
635
|
+
const window = turn.contextLength ?? contextLength ?? 0;
|
|
636
|
+
if (window > 0) {
|
|
637
|
+
metrics.largestPromptShare = Math.max(metrics.largestPromptShare ?? 0, turn.prompt / window);
|
|
638
|
+
}
|
|
628
639
|
}
|
|
629
640
|
}
|
|
630
641
|
else if (event.kind === RunEventKind.Done) {
|
|
@@ -643,8 +654,5 @@ export function runMetrics(events, { contextLength } = {}) {
|
|
|
643
654
|
if (metrics.draftTotal && metrics.draftAccepted !== undefined) {
|
|
644
655
|
metrics.draftAcceptance = metrics.draftAccepted / metrics.draftTotal;
|
|
645
656
|
}
|
|
646
|
-
if (metrics.largestPrompt !== undefined && contextLength && contextLength > 0) {
|
|
647
|
-
metrics.largestPromptShare = metrics.largestPrompt / contextLength;
|
|
648
|
-
}
|
|
649
657
|
return metrics;
|
|
650
658
|
}
|
package/dist/run/run-calls.d.ts
CHANGED
|
@@ -2,7 +2,7 @@ import type OpenAI from 'openai';
|
|
|
2
2
|
import type { CatalogServer } from '../tools/catalog.ts';
|
|
3
3
|
import type { Turn } from '../wire/stream.ts';
|
|
4
4
|
import { type EndedBy, type RunArtifact, type ToolAnswer, type ToolCall, type ToolCallOutcome, type ToolCallReport, type ToolCallRequest, type ToolCallResult } from '../wire/tool-calls.ts';
|
|
5
|
-
import type { RunEventInput } from './events.ts';
|
|
5
|
+
import type { LoadCounts, RunEventInput } from './events.ts';
|
|
6
6
|
/**
|
|
7
7
|
* The tools between two turns: a step's calls read, run and written into the transcript.
|
|
8
8
|
*
|
|
@@ -31,12 +31,15 @@ export interface ReadCall {
|
|
|
31
31
|
error?: unknown;
|
|
32
32
|
/** The arguments as every later request replays them: the repaired JSON, or `{}` for none. */
|
|
33
33
|
normal: string;
|
|
34
|
+
/** Whether `args` took a repair to read, as `ToolCallRequest.repaired` says. */
|
|
35
|
+
repaired: boolean;
|
|
34
36
|
}
|
|
35
37
|
/**
|
|
36
38
|
* A turn's calls with their arguments read.
|
|
37
39
|
*
|
|
38
40
|
* @param calls - The turn's calls, in the order the model made them.
|
|
39
41
|
* @param finishReason - Why the turn ended, which tells a call cut off from one written wrongly.
|
|
42
|
+
* @param [recovered] - Whether the calls were recovered from the reply's text.
|
|
40
43
|
* @returns One entry per call, in the same order. Unreadable arguments do not throw: the entry
|
|
41
44
|
* carries the error in place of them.
|
|
42
45
|
*
|
|
@@ -44,7 +47,7 @@ export interface ReadCall {
|
|
|
44
47
|
* A server that parses replayed arguments refuses the almost-JSON a model wrote, so each is
|
|
45
48
|
* normalised here, and one that could not be read at all is replayed as no arguments.
|
|
46
49
|
*/
|
|
47
|
-
export declare const readCalls: (calls: ToolCall[], finishReason: Turn['finishReason']) => ReadCall[];
|
|
50
|
+
export declare const readCalls: (calls: ToolCall[], finishReason: Turn['finishReason'], recovered?: boolean) => ReadCall[];
|
|
48
51
|
/**
|
|
49
52
|
* The assistant message a turn is written into the transcript as, its calls' arguments repaired.
|
|
50
53
|
*
|
|
@@ -54,6 +57,15 @@ export declare const readCalls: (calls: ToolCall[], finishReason: Turn['finishRe
|
|
|
54
57
|
* @returns A new message, each call on it a copy with `normal` for its arguments.
|
|
55
58
|
*/
|
|
56
59
|
export declare const assistantMessage: (content: string, parsed: ReadCall[]) => OpenAI.ChatCompletionAssistantMessageParam;
|
|
60
|
+
/**
|
|
61
|
+
* The assistant message a turn is, as the model produced it: its text and calls as they streamed.
|
|
62
|
+
*
|
|
63
|
+
* @param content - The turn's text, recovered calls left in it. Empty is written as `null`.
|
|
64
|
+
* @param calls - The calls the model sent as calls, with their arguments as streamed. Not written
|
|
65
|
+
* to. None leaves `tool_calls` off the message rather than empty.
|
|
66
|
+
* @returns A new message, each call on it a copy.
|
|
67
|
+
*/
|
|
68
|
+
export declare const writtenMessage: (content: string, calls: ToolCall[]) => OpenAI.ChatCompletionAssistantMessageParam;
|
|
57
69
|
/**
|
|
58
70
|
* The definitions the host gave for these names, in the order the names are given.
|
|
59
71
|
*
|
|
@@ -89,11 +101,7 @@ export interface Calling {
|
|
|
89
101
|
/** The host's own tools by name: declared on every step, and never reached through `call_tool`. */
|
|
90
102
|
hostNames: ReadonlySet<string>;
|
|
91
103
|
loaded: Set<string>;
|
|
92
|
-
loads:
|
|
93
|
-
toolsLoaded: number;
|
|
94
|
-
redundantLoads: number;
|
|
95
|
-
unknownToolNames: number;
|
|
96
|
-
};
|
|
104
|
+
loads: LoadCounts;
|
|
97
105
|
used: Set<string>;
|
|
98
106
|
toolCalls: ToolCallOutcome[];
|
|
99
107
|
/** What the run's calls linked to, by uri, in the order first seen. */
|
package/dist/run/run-calls.js
CHANGED
|
@@ -1,3 +1,4 @@
|
|
|
1
|
+
import { isRecord } from '../core/guards.js';
|
|
1
2
|
import { CALL_TOOL, expandNames, inCatalog, LOAD_TOOLS, loadResult, proxiedCall, proxyLoadResult, requestedNames, shownCall, toolName, } from '../tools/tool-loading.js';
|
|
2
3
|
import { errorMessage } from '../wire/errors.js';
|
|
3
4
|
import { parseToolArguments, } from '../wire/tool-calls.js';
|
|
@@ -26,20 +27,40 @@ export const preview = (text, limit = 2000) => text.length > limit ? `${text.sli
|
|
|
26
27
|
const STOPPED_CALL = 'Stopped before this call finished.';
|
|
27
28
|
/** The same for a call of that step the loop never got to. */
|
|
28
29
|
const UNRUN_CALL = 'Not run: the run stopped first.';
|
|
30
|
+
/**
|
|
31
|
+
* Whether a call's arguments are, as written, the JSON of an object — or no arguments at all.
|
|
32
|
+
*
|
|
33
|
+
* @param raw - The arguments as the turn carried them: a string, or an object a server parsed.
|
|
34
|
+
* @returns False where reading them took `parseToolArguments`' repairs.
|
|
35
|
+
*/
|
|
36
|
+
function readsAsWritten(raw) {
|
|
37
|
+
if (typeof raw !== 'string' || !raw.trim()) {
|
|
38
|
+
return true;
|
|
39
|
+
}
|
|
40
|
+
try {
|
|
41
|
+
return isRecord(JSON.parse(raw));
|
|
42
|
+
}
|
|
43
|
+
catch {
|
|
44
|
+
return false;
|
|
45
|
+
}
|
|
46
|
+
}
|
|
29
47
|
/**
|
|
30
48
|
* One call with its arguments read, or with why they could not be.
|
|
31
49
|
*
|
|
32
50
|
* @param call - The call as the turn carried it.
|
|
33
51
|
* @param finishReason - Why the turn ended, which tells a call cut off from one written wrongly.
|
|
52
|
+
* @param recovered - Whether the call was recovered from the reply's text, which makes it repaired
|
|
53
|
+
* whatever its arguments.
|
|
34
54
|
* @returns The call beside its arguments and their repaired JSON, or beside the error and `{}`.
|
|
35
55
|
*/
|
|
36
|
-
function readCall(call, finishReason) {
|
|
56
|
+
function readCall(call, finishReason, recovered) {
|
|
37
57
|
try {
|
|
38
58
|
const args = parseToolArguments(call.function.arguments, { finishReason });
|
|
39
|
-
|
|
59
|
+
const repaired = recovered || !readsAsWritten(call.function.arguments);
|
|
60
|
+
return { call, args, normal: JSON.stringify(args), repaired };
|
|
40
61
|
}
|
|
41
62
|
catch (error) {
|
|
42
|
-
return { call, error, normal: '{}' };
|
|
63
|
+
return { call, error, normal: '{}', repaired: false };
|
|
43
64
|
}
|
|
44
65
|
}
|
|
45
66
|
/**
|
|
@@ -47,6 +68,7 @@ function readCall(call, finishReason) {
|
|
|
47
68
|
*
|
|
48
69
|
* @param calls - The turn's calls, in the order the model made them.
|
|
49
70
|
* @param finishReason - Why the turn ended, which tells a call cut off from one written wrongly.
|
|
71
|
+
* @param [recovered] - Whether the calls were recovered from the reply's text.
|
|
50
72
|
* @returns One entry per call, in the same order. Unreadable arguments do not throw: the entry
|
|
51
73
|
* carries the error in place of them.
|
|
52
74
|
*
|
|
@@ -54,7 +76,7 @@ function readCall(call, finishReason) {
|
|
|
54
76
|
* A server that parses replayed arguments refuses the almost-JSON a model wrote, so each is
|
|
55
77
|
* normalised here, and one that could not be read at all is replayed as no arguments.
|
|
56
78
|
*/
|
|
57
|
-
export const readCalls = (calls, finishReason) => calls.map((call) => readCall(call, finishReason));
|
|
79
|
+
export const readCalls = (calls, finishReason, recovered = false) => calls.map((call) => readCall(call, finishReason, recovered));
|
|
58
80
|
/**
|
|
59
81
|
* The assistant message a turn is written into the transcript as, its calls' arguments repaired.
|
|
60
82
|
*
|
|
@@ -75,6 +97,19 @@ export const assistantMessage = (content, parsed) => ({
|
|
|
75
97
|
}
|
|
76
98
|
: {}),
|
|
77
99
|
});
|
|
100
|
+
/**
|
|
101
|
+
* The assistant message a turn is, as the model produced it: its text and calls as they streamed.
|
|
102
|
+
*
|
|
103
|
+
* @param content - The turn's text, recovered calls left in it. Empty is written as `null`.
|
|
104
|
+
* @param calls - The calls the model sent as calls, with their arguments as streamed. Not written
|
|
105
|
+
* to. None leaves `tool_calls` off the message rather than empty.
|
|
106
|
+
* @returns A new message, each call on it a copy.
|
|
107
|
+
*/
|
|
108
|
+
export const writtenMessage = (content, calls) => ({
|
|
109
|
+
role: Role.Assistant,
|
|
110
|
+
content: content || null,
|
|
111
|
+
...(calls.length ? { tool_calls: calls.map((call) => ({ ...call, function: { ...call.function } })) } : {}),
|
|
112
|
+
});
|
|
78
113
|
/**
|
|
79
114
|
* The definitions the host gave for these names, in the order the names are given.
|
|
80
115
|
*
|
|
@@ -155,17 +190,28 @@ function answerOf(answer) {
|
|
|
155
190
|
const shownAs = (run, { call, args, normal }) => run.proxies && call.function.name === CALL_TOOL
|
|
156
191
|
? shownCall(CALL_TOOL, args ? normal : call.function.arguments)
|
|
157
192
|
: { name: call.function.name, input: call.function.arguments };
|
|
193
|
+
/**
|
|
194
|
+
* Adds one call's load counts to the run's.
|
|
195
|
+
*
|
|
196
|
+
* @param into - The run's counts. Written to.
|
|
197
|
+
* @param counts - The call's. Not written to.
|
|
198
|
+
*/
|
|
199
|
+
function addLoads(into, counts) {
|
|
200
|
+
into.toolsLoaded += counts.toolsLoaded;
|
|
201
|
+
into.redundantLoads += counts.redundantLoads;
|
|
202
|
+
into.unknownToolNames += counts.unknownToolNames;
|
|
203
|
+
}
|
|
158
204
|
/**
|
|
159
205
|
* Answers a `load_tools`, counting what it loaded.
|
|
160
206
|
*
|
|
161
207
|
* @param run - The run. Written to: the names this call loads are added to `loaded`, and `loads`
|
|
162
208
|
* counts the new ones, the ones already loaded and the ones not in the catalogue.
|
|
163
209
|
* @param args - The call's arguments, read as `requestedNames` reads them.
|
|
164
|
-
* @returns The text the model reads
|
|
165
|
-
* where it named even one tool that was loaded already.
|
|
210
|
+
* @returns The text the model reads; `ok`, false where nothing it named was loadable and true
|
|
211
|
+
* where it named even one tool that was loaded already; and `counts`, this call's alone.
|
|
166
212
|
*/
|
|
167
213
|
function answerLoad(run, args) {
|
|
168
|
-
const { catalog, proxied, definitions, loaded
|
|
214
|
+
const { catalog, proxied, definitions, loaded } = run;
|
|
169
215
|
const resolved = expandNames(requestedNames(args), catalog);
|
|
170
216
|
const content = proxied
|
|
171
217
|
? proxyLoadResult(resolved, catalog, definedAs(definitions, resolved.matched), loaded)
|
|
@@ -173,14 +219,15 @@ function answerLoad(run, args) {
|
|
|
173
219
|
// Proxied, a load is only of what the result could define: a catalogued name the host
|
|
174
220
|
// gave no definition for was not loaded, and is not counted or remembered as though it was.
|
|
175
221
|
const hits = proxied ? resolved.matched.filter((hit) => definitions.has(hit)) : resolved.matched;
|
|
222
|
+
const counts = { toolsLoaded: 0, redundantLoads: 0, unknownToolNames: resolved.unknown.length };
|
|
176
223
|
for (const hit of hits) {
|
|
177
|
-
|
|
224
|
+
counts[loaded.has(hit) ? 'redundantLoads' : 'toolsLoaded']++;
|
|
178
225
|
}
|
|
179
|
-
loads
|
|
226
|
+
addLoads(run.loads, counts);
|
|
180
227
|
for (const hit of hits) {
|
|
181
228
|
loaded.add(hit);
|
|
182
229
|
}
|
|
183
|
-
return { content, ok: hits.length > 0 };
|
|
230
|
+
return { content, ok: hits.length > 0, counts };
|
|
184
231
|
}
|
|
185
232
|
/**
|
|
186
233
|
* Makes a call at most once per key, sharing the in-flight promise so two identical calls in one
|
|
@@ -272,7 +319,7 @@ function prepare(run, entry) {
|
|
|
272
319
|
}
|
|
273
320
|
// Built for every call rather than only the dispatched ones, so a host hears of the calls
|
|
274
321
|
// the loop answers itself in the same shape. Arguments that could not be read are none.
|
|
275
|
-
const request = { id: call.id, name, args: inner, raw };
|
|
322
|
+
const request = { id: call.id, name, args: inner, raw, repaired: entry.repaired };
|
|
276
323
|
// By the name the model called, not the one a `call_tool` wraps: `load_tools` is not in
|
|
277
324
|
// the catalogue, and one reached through `call_tool` is refused like any other.
|
|
278
325
|
const load = onDemand && call.function.name === LOAD_TOOLS;
|
|
@@ -353,6 +400,7 @@ async function runCall(run, prepared, answered, step) {
|
|
|
353
400
|
let links = [];
|
|
354
401
|
let end = false;
|
|
355
402
|
let ok = true;
|
|
403
|
+
let loads;
|
|
356
404
|
// Answered from an earlier identical call of the step, whose links are already counted.
|
|
357
405
|
let shared = false;
|
|
358
406
|
try {
|
|
@@ -360,7 +408,7 @@ async function runCall(run, prepared, answered, step) {
|
|
|
360
408
|
throw unreadable;
|
|
361
409
|
}
|
|
362
410
|
if (load) {
|
|
363
|
-
({ content, ok } = answerLoad(run, args));
|
|
411
|
+
({ content, ok, counts: loads } = answerLoad(run, args));
|
|
364
412
|
}
|
|
365
413
|
else {
|
|
366
414
|
if (refused) {
|
|
@@ -393,7 +441,14 @@ async function runCall(run, prepared, answered, step) {
|
|
|
393
441
|
finally {
|
|
394
442
|
close();
|
|
395
443
|
}
|
|
396
|
-
run.onEvent({
|
|
444
|
+
run.onEvent({
|
|
445
|
+
kind: RunEventKind.ToolResult,
|
|
446
|
+
id: call.id,
|
|
447
|
+
name,
|
|
448
|
+
ok,
|
|
449
|
+
text: preview(content),
|
|
450
|
+
...(loads ? { loads } : {}),
|
|
451
|
+
});
|
|
397
452
|
const isFirstAnswer = shared === false;
|
|
398
453
|
if (isFirstAnswer) {
|
|
399
454
|
collect(run, links, { callId: call.id, tool: name, step });
|
|
@@ -520,11 +575,12 @@ export async function loadShortlist(run, shortlist, messages) {
|
|
|
520
575
|
// host that shortlisted more has already chosen.
|
|
521
576
|
const content = proxyLoadResult(expandNames(names, catalog, names.length), catalog, shortlist);
|
|
522
577
|
run.onEvent({ kind: RunEventKind.ToolCall, id, name: LOAD_TOOLS, text: preview(args) });
|
|
523
|
-
run.onToolCall?.({ id, name: LOAD_TOOLS, args: { names }, raw: args });
|
|
524
|
-
|
|
578
|
+
run.onToolCall?.({ id, name: LOAD_TOOLS, args: { names }, raw: args, repaired: false });
|
|
579
|
+
const loads = { toolsLoaded: names.length, redundantLoads: 0, unknownToolNames: 0 };
|
|
580
|
+
run.onEvent({ kind: RunEventKind.ToolResult, id, name: LOAD_TOOLS, ok: true, text: preview(content), loads });
|
|
525
581
|
run.onToolResult?.({ id, name: LOAD_TOOLS, ok: true, content });
|
|
526
582
|
run.toolCalls.push({ id, name: LOAD_TOOLS, ok: true });
|
|
527
|
-
run.loads
|
|
583
|
+
addLoads(run.loads, loads);
|
|
528
584
|
for (const name of names) {
|
|
529
585
|
run.loaded.add(name);
|
|
530
586
|
}
|
package/dist/wire/errors.d.ts
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
import type OpenAI from 'openai';
|
|
2
2
|
import { ContextOverflow } from './retry.ts';
|
|
3
|
-
import type {
|
|
3
|
+
import type { LoopUsage, PartialTurn } from './stream.ts';
|
|
4
4
|
import type { RunArtifact, ToolCallOutcome } from './tool-calls.ts';
|
|
5
5
|
/**
|
|
6
6
|
* What went wrong, as a string.
|
|
@@ -40,8 +40,11 @@ export interface AgentLoopFailure {
|
|
|
40
40
|
* watching had been shown, and whether it belongs in the history is the host's call.
|
|
41
41
|
*/
|
|
42
42
|
partial?: PartialTurn;
|
|
43
|
-
/**
|
|
44
|
-
|
|
43
|
+
/**
|
|
44
|
+
* Summed over the turns that came back, as the result's `usage` would have said it. A request
|
|
45
|
+
* that failed or was stopped adds nothing.
|
|
46
|
+
*/
|
|
47
|
+
usage: LoopUsage;
|
|
45
48
|
/** Every call that was made, in order. One the run was stopped during is `ok: false`. */
|
|
46
49
|
toolCalls: ToolCallOutcome[];
|
|
47
50
|
/**
|
|
@@ -70,7 +73,7 @@ export declare class AgentLoopError extends Error implements AgentLoopFailure {
|
|
|
70
73
|
readonly name: string;
|
|
71
74
|
readonly messages: OpenAI.ChatCompletionMessageParam[];
|
|
72
75
|
readonly partial?: PartialTurn;
|
|
73
|
-
readonly usage:
|
|
76
|
+
readonly usage: LoopUsage;
|
|
74
77
|
readonly toolCalls: ToolCallOutcome[];
|
|
75
78
|
readonly artifacts: RunArtifact[];
|
|
76
79
|
readonly loaded: string[];
|
|
@@ -105,7 +108,7 @@ export declare class ToolIterationLimit extends AgentLoopError {
|
|
|
105
108
|
export declare class AgentLoopOverflow extends ContextOverflow implements AgentLoopFailure {
|
|
106
109
|
readonly messages: OpenAI.ChatCompletionMessageParam[];
|
|
107
110
|
readonly partial?: PartialTurn;
|
|
108
|
-
readonly usage:
|
|
111
|
+
readonly usage: LoopUsage;
|
|
109
112
|
readonly toolCalls: ToolCallOutcome[];
|
|
110
113
|
readonly artifacts: RunArtifact[];
|
|
111
114
|
readonly loaded: string[];
|
package/dist/wire/stream.d.ts
CHANGED
|
@@ -145,6 +145,60 @@ export interface TurnUsage {
|
|
|
145
145
|
* nothing reported them.
|
|
146
146
|
*/
|
|
147
147
|
export declare const noUsage: () => TurnUsage;
|
|
148
|
+
/**
|
|
149
|
+
* A run's token totals, in the spelling every run-level usage shares.
|
|
150
|
+
*
|
|
151
|
+
* @remarks
|
|
152
|
+
* The spelling of `RunUsage`, which is what the `usage` events carry and what hosts persist, so a
|
|
153
|
+
* host reads a loop's result and its events with one mapping rather than two. A single turn's own
|
|
154
|
+
* report, `TurnUsage`, keeps its shorter names: it is a different thing, a turn's measurements,
|
|
155
|
+
* and every field on it is that turn's.
|
|
156
|
+
*/
|
|
157
|
+
export interface UsageTotals {
|
|
158
|
+
promptTokens: number;
|
|
159
|
+
completionTokens: number;
|
|
160
|
+
totalTokens: number;
|
|
161
|
+
/** How much of `promptTokens` came from the endpoint's prompt cache. Zero where none said. */
|
|
162
|
+
cachedTokens: number;
|
|
163
|
+
/**
|
|
164
|
+
* Whether any turn of the run reported a cache count, which is what tells a `cachedTokens` of
|
|
165
|
+
* zero that is a miss from one the endpoint never reports. A turn reported one where its usage
|
|
166
|
+
* carries `uncached`.
|
|
167
|
+
*/
|
|
168
|
+
cacheReported: boolean;
|
|
169
|
+
}
|
|
170
|
+
/**
|
|
171
|
+
* What a loop spent, as `AgentLoopResult.usage` and a failed run's `usage` carry it.
|
|
172
|
+
*
|
|
173
|
+
* @remarks
|
|
174
|
+
* `UsageTotals`, with the four counts also under the names this field had before it: a host that
|
|
175
|
+
* reads `usage.prompt` keeps compiling and reads the same number as `usage.promptTokens`.
|
|
176
|
+
*/
|
|
177
|
+
export interface LoopUsage extends TurnUsage, UsageTotals {
|
|
178
|
+
/** @deprecated Read `promptTokens`, the same number. */
|
|
179
|
+
prompt: number;
|
|
180
|
+
/** @deprecated Read `completionTokens`, the same number. */
|
|
181
|
+
completion: number;
|
|
182
|
+
/** @deprecated Read `totalTokens`, the same number. */
|
|
183
|
+
total: number;
|
|
184
|
+
/** @deprecated Read `cachedTokens`, the same number, and `cacheReported` for whether it was reported. */
|
|
185
|
+
cached: number;
|
|
186
|
+
}
|
|
187
|
+
/**
|
|
188
|
+
* Whether a turn's usage says the endpoint reported its cache.
|
|
189
|
+
*
|
|
190
|
+
* @param usage - One turn's report.
|
|
191
|
+
* @returns True where it carries `uncached`, which is present only where a cache count was.
|
|
192
|
+
*/
|
|
193
|
+
export declare const reportsCache: (usage: TurnUsage) => boolean;
|
|
194
|
+
/**
|
|
195
|
+
* A loop's summed counts in both spellings, for its result and its `usage` events.
|
|
196
|
+
*
|
|
197
|
+
* @param counts - The four counts summed over the loop's turns. Not written to.
|
|
198
|
+
* @param cacheReported - Whether any of those turns reported a cache count.
|
|
199
|
+
* @returns A new usage holding the four counts under both names, and the flag.
|
|
200
|
+
*/
|
|
201
|
+
export declare const loopUsage: (counts: TurnUsage, cacheReported: boolean) => LoopUsage;
|
|
148
202
|
/**
|
|
149
203
|
* The four token counts of two usages, added.
|
|
150
204
|
*
|
package/dist/wire/stream.js
CHANGED
|
@@ -42,6 +42,31 @@ export const CacheBreakReason = {
|
|
|
42
42
|
* nothing reported them.
|
|
43
43
|
*/
|
|
44
44
|
export const noUsage = () => ({ prompt: 0, completion: 0, total: 0, cached: 0 });
|
|
45
|
+
/**
|
|
46
|
+
* Whether a turn's usage says the endpoint reported its cache.
|
|
47
|
+
*
|
|
48
|
+
* @param usage - One turn's report.
|
|
49
|
+
* @returns True where it carries `uncached`, which is present only where a cache count was.
|
|
50
|
+
*/
|
|
51
|
+
export const reportsCache = (usage) => usage.uncached !== undefined;
|
|
52
|
+
/**
|
|
53
|
+
* A loop's summed counts in both spellings, for its result and its `usage` events.
|
|
54
|
+
*
|
|
55
|
+
* @param counts - The four counts summed over the loop's turns. Not written to.
|
|
56
|
+
* @param cacheReported - Whether any of those turns reported a cache count.
|
|
57
|
+
* @returns A new usage holding the four counts under both names, and the flag.
|
|
58
|
+
*/
|
|
59
|
+
export const loopUsage = (counts, cacheReported) => ({
|
|
60
|
+
prompt: counts.prompt,
|
|
61
|
+
completion: counts.completion,
|
|
62
|
+
total: counts.total,
|
|
63
|
+
cached: counts.cached,
|
|
64
|
+
promptTokens: counts.prompt,
|
|
65
|
+
completionTokens: counts.completion,
|
|
66
|
+
totalTokens: counts.total,
|
|
67
|
+
cachedTokens: counts.cached,
|
|
68
|
+
cacheReported,
|
|
69
|
+
});
|
|
45
70
|
/**
|
|
46
71
|
* The four token counts of two usages, added.
|
|
47
72
|
*
|
|
@@ -114,9 +114,24 @@ export interface ToolCallRequest {
|
|
|
114
114
|
args: Record<string, unknown>;
|
|
115
115
|
/**
|
|
116
116
|
* The arguments as the model wrote them, before any repair. Through `call_tool`, the inner
|
|
117
|
-
* arguments as JSON text.
|
|
117
|
+
* arguments as JSON text. For a call recovered from the reply's text, the arguments recovery read
|
|
118
|
+
* out of it, as JSON text: the words the model wrote are the reply's content, which `onMessage`'s
|
|
119
|
+
* `raw` message carries.
|
|
118
120
|
*/
|
|
119
121
|
raw: string;
|
|
122
|
+
/**
|
|
123
|
+
* Whether `args` took a repair to read: true where `raw` does not parse, as written, to the JSON
|
|
124
|
+
* object `args` holds — near-JSON, or JSON inside a string — and for every call recovered from
|
|
125
|
+
* text, which the model never sent as a call. False for blank arguments, which every reader takes
|
|
126
|
+
* as none, and for arguments that could not be read at all. Through `call_tool`, whether the
|
|
127
|
+
* `call_tool`'s own arguments did.
|
|
128
|
+
*
|
|
129
|
+
* @remarks
|
|
130
|
+
* A host that shows or stores what the model said reads `raw` where this is true. Whitespace is
|
|
131
|
+
* not a repair, though the replayed arguments are re-serialised either way. `runAgentLoop` always
|
|
132
|
+
* sets it; optional so a request built before it existed still compiles.
|
|
133
|
+
*/
|
|
134
|
+
repaired?: boolean;
|
|
120
135
|
}
|
|
121
136
|
/** What one tool call did, in the order the model asked. */
|
|
122
137
|
export interface ToolCallOutcome {
|
package/llms.txt
CHANGED
|
@@ -176,6 +176,7 @@ The loop above a turn: send, run the tools the model asked for, send again, unti
|
|
|
176
176
|
- `PRESELECTED_NOTICE` — What opens the notice naming the loop's own picks, ahead of the names.
|
|
177
177
|
- `runAgentLoop` — Runs a question to its answer: one `runTurn` per step, the tools it asks for between them, until a turn asks for none or a call's answer ends the run.
|
|
178
178
|
- `StepWindow` (type) — How full the window is as a step is about to be sent, as `beforeStep` is told it.
|
|
179
|
+
- `WrittenMessage` (type) — What `onMessage` is handed beside an assistant message the model wrote: that message as the model produced it.
|
|
179
180
|
|
|
180
181
|
### events
|
|
181
182
|
|
|
@@ -187,6 +188,7 @@ What a run is doing, while it is doing it.
|
|
|
187
188
|
- `endRun` — Forgets a run that will not be emitting `done` — one whose process is tearing down, or whose loop threw where it could not be caught.
|
|
188
189
|
- `fold` — A run's events with each stretch of `thinking` or of `output` joined into one block, for a reader that takes a run in snapshots.
|
|
189
190
|
- `history` — The backlog alone, for a caller that wants a snapshot rather than a subscription.
|
|
191
|
+
- `LoadCounts` (type) — What `load_tools` calls resolved to: new tools, ones already loaded, and names in no catalogue.
|
|
190
192
|
- `RunEvent` (type) — One thing that happened in a run, as a watcher receives it.
|
|
191
193
|
- `RunEventInput` (type) — What `emit` is given: the run and the sequence are the bus's to assign.
|
|
192
194
|
- `RunEventKind` — Any one of the kinds in `RunEventKind`.
|
|
@@ -388,12 +390,14 @@ Reading one streamed turn back into a message.
|
|
|
388
390
|
|
|
389
391
|
- `CacheBreakReason` — Any one of the reasons in `CacheBreakReason`.
|
|
390
392
|
- `InFlight` (type) — Where the request in flight can be read from, after it has thrown.
|
|
393
|
+
- `LoopUsage` (type) — What a loop spent, as `AgentLoopResult.usage` and a failed run's `usage` carry it.
|
|
391
394
|
- `PartialTurn` (type) — What a turn had said when it ended without coming back: the answer, the scratchpad and the calls, as far as each had arrived.
|
|
392
395
|
- `Produced` (type) — Whether the model has said anything a second attempt would say twice.
|
|
393
396
|
- `StreamTurnOptions` (type) — What `streamTurn` takes besides the request body.
|
|
394
397
|
- `streamTurn` — Runs one turn as a stream, reporting tokens as they arrive and assembling them back into a message.
|
|
395
398
|
- `Turn` (type) — One streamed turn, put back together into the shape a loop and a transcript work with.
|
|
396
399
|
- `TurnUsage` (type) — What a turn cost, and how it went.
|
|
400
|
+
- `UsageTotals` (type) — A run's token totals, in the spelling every run-level usage shares.
|
|
397
401
|
|
|
398
402
|
### thinking
|
|
399
403
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@cubicecho/agent-core",
|
|
3
|
-
"version": "4.
|
|
3
|
+
"version": "4.20.0",
|
|
4
4
|
"description": "The endpoint-agnostic half of an OpenAI-compatible agent loop: tool-schema compatibility, on-demand tool loading, one-shot side tasks, run events, and a pooled client.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"openai",
|