@modelprofile.com/flexharness-agent 8.3.0 → 8.4.0

package/ts_agent/readme.md

@@ -88,7 +88,7 @@ The following Node examples reuse `setup` and, where needed, `tools` from the qu
 | `messages` | Current AI SDK message history after projection or compaction. Pass it into another run to continue. |
 | `steps` | Completed model steps, including steps from validation-triggered attempts. A step can call several tools. |
 | `finishReason` | The model's final finish reason; inspect this together with application validation. |
-| `usage` | Input, output and total tokens, plus cache-read and cache-write tokens. |
+| `usage` | Input, output and total tokens, plus cache-read and cache-write tokens, summed over every model call of the run the provider reported: its model steps, retried calls included, and the reported calls of a compaction its context overflow caused. |
 | `toolCalls` | Tool-call IDs, names and inputs, with available outputs or errors. |
 
 ```typescript
@@ -154,9 +154,68 @@ Pass these callbacks to `runAgent` or `AgentSession.create()`:
 | `onToolCallUpdate(event)` | The call identity and a transient streamed `output`. |
 | `onToolCallFinish(event)` | The call identity plus either `success: true, output` or `success: false, error`. |
 | `onRetry(event)` | Before each wait to retry a model call: `attempt`, `maxAttempts`, `delayMs` and `reason` (`rate_limit`, `overloaded`, `unavailable`). |
+| `onUsage(event)` | Once per model call, as soon as its usage is known; see [Count the usage of every run](#count-the-usage-of-every-run). |
 
 Tool updates are transient; the finish callback carries the authoritative final output. Use `subscribe()` for committed session changes (`committed`, `updated`, `archived`), and retain its returned unsubscribe function. Session listeners are delivered in order per listener, have bounded queues and timeouts, and are removed on failure. Streaming callbacks and session-change listeners serve different purposes.
 
+## Count the usage of every run
+
+A run that throws or is aborted has used tokens too, and its promise carries no result. `onUsage` reports each model call once, as soon as its usage is known, whatever the run's outcome. Sum it to count a run's usage:
+
+```typescript
+import type { IAgentUsage } from '@modelprofile.com/flexharness-agent';
+
+const used: IAgentUsage = { inputTokens: 0, outputTokens: 0, totalTokens: 0, cacheReadTokens: 0, cacheWriteTokens: 0 };
+let usageComplete = true;
+try {
+  await runAgent({
+    ...setup,
+    tools,
+    prompt: 'Convert 10 km to miles.',
+    abort: AbortSignal.timeout(60_000),
+    onUsage: (event) => {
+      if (event.status === 'unreported') {
+        usageComplete = false;
+        return;
+      }
+      for (const key of Object.keys(used) as (keyof IAgentUsage)[]) used[key] += event.usage[key];
+    },
+  });
+} finally {
+  console.log(used, usageComplete ? 'complete' : 'lower bound');
+}
+```
+
+| Event field | Meaning |
+| --- | --- |
+| `source` | `generation`: a model step of the generation. `compaction`: a model call of a context compaction. |
+| `generationId` | The generation the call belongs to. A compaction carries the generation whose context overflow caused it; a compaction by `compact()` or event retention has none. |
+| `provider`, `requestedModelId` | The model the call was made with: the language model's `provider` and `modelId`. Every event carries both, whatever its status, so key usage caps on them. |
+| `status: 'reported'`, `usage`, `responseModelId` | The provider reported the call's usage. A count the provider leaves out is zero. `responseModelId` is the model id the provider's response named (a provider may answer with a dated model version), or `requestedModelId` when it named none. |
+| `status: 'unreported'`, `reason` | The call ended before the provider reported its usage: `aborted` (the call was aborted), `failed` (the call or its response failed) or `missing` (the response carried no usage). The provider may still have consumed tokens for it; their number is unknown. |
+
+When a run returns, its reported calls sum to `result.usage`; count one or the other, not both. A call is reported when the provider's response ends, before its tool calls run, so a run aborted during a tool call still reports the call that requested it. Retried calls and validation retries are included. `onUsage` must not throw; an error it throws is reported like a session listener error and does not change the run's outcome. `runAgent` delivers every event of the run before its promise settles. `AgentSession.create()` accepts the same callback for every generation and compaction of the session; `generate()` and `scheduleGenerate()` without a `transaction` reject as soon as their abort signal fires and may report the interrupted call afterwards; `close()` waits for that report. With a `transaction`, they settle only after every call of the generation has been reported.
+
+### What is counted
+
+- Every model step of a generation, whatever its outcome, including retried calls and validation retries.
+- The model calls of a context compaction, when the compactor reports them. `contextCompactor` and `onContextOverflow` receive `reportUsage` in their options; it reports into `onUsage` with `source: 'compaction'`. `compactMessages()` from `@modelprofile.com/flexharness/compaction` reports each attempt of its model call when it receives `reportUsage`, so pass the handler's options through:
+
+  ```typescript
+  import { compactMessages } from '@modelprofile.com/flexharness/compaction';
+
+  const session = await AgentSession.create({
+    ...setup,
+    contextCompactor: (messages, _events, options) => compactMessages(setup.model, messages, options),
+    onContextOverflow: (messages, options) => compactMessages(setup.model, messages, options),
+    onUsage: (event) => console.log(event),
+  });
+  ```
+
+  A handler that makes its own model calls reports each one through `reportUsage` before its promise settles, a call that fails or is aborted as `unreported`. `AgentModelCallUsageRecorder` does the bookkeeping for one language model: call `start()` when a call begins, `end(responseModelId, usage)` with the AI SDK's `onLanguageModelCallEnd` values, and `settle(reason)` when a call ends otherwise.
+
+Not counted: model calls a handler makes without reporting them through `reportUsage`, and model calls outside the session, such as tools that call models themselves. Report those in your own accounting.
+
 ## Validate an answer and request corrections
 
 `validateCompletion` returns `void` to accept a result or a string to add a corrective user message and generate again. `maxValidationRetries` defaults to `0`: a failed validation throws unless retries are configured.
@@ -243,9 +302,9 @@ Use an application-owned durable adapter for persistent sessions. Its `save(sess
 ### Bound model context and active events
 
 - `contextBuilder({ events })` controls the model-message projection.
-- `contextCompactor(messages, events, { reason, abortSignal })` returns replacement model messages. Provide it to use `session.compact()` or automatic event retention.
+- `contextCompactor(messages, events, { reason, abortSignal, reportUsage })` returns replacement model messages. Provide it to use `session.compact()` or automatic event retention. Report the usage of its model calls through `reportUsage`; see [What is counted](#what-is-counted).
 - `eventRetention: { maxEvents }` triggers compaction and archival when the active event count exceeds the threshold. It also requires an event store with `archive()` support.
-- Context overflow invokes the configured compactor, or the `onContextOverflow` handler. Without either, generation throws `ContextOverflowError`. `maxContextOverflowRetries` defaults to `3`.
+- Context overflow invokes the configured compactor, or the `onContextOverflow(messages, { abortSignal, reportUsage })` handler. Without either, generation throws `ContextOverflowError`. `maxContextOverflowRetries` defaults to `3`.
 - Open transactions and uncertain tool intents constrain when compaction and archival can proceed. Resolve them before manual compaction.
 
 Compaction changes the active model context; archival moves covered events out of the active event set. Implement archive retention in your chosen store when you need a complete audit history.

package/ts_agent/runner.ts

@@ -37,4 +37,11 @@ export function runAgent(options: IAgentRunOptions): Promise<IAgentRunResult> {
   return runAgentWithSession(options, (sessionOptions) => RunnerSession.create(sessionOptions));
 }
 
-export type { IAgentRunOptions, IAgentRunResult } from './smartagent.interfaces.js';
+export type {
+  IAgentRunOptions,
+  IAgentRunResult,
+  IAgentUsage,
+  TAgentModelCallUnreportedReason,
+  TAgentModelCallUsage,
+  TAgentModelCallUsageEvent,
+} from './smartagent.interfaces.js';

package/ts_agent/runtime.run.ts

@@ -52,6 +52,7 @@ export async function runAgentWithSession(
     onToolCallUpdate: options.onToolCallUpdate,
     onToolCallFinish: options.onToolCallFinish,
     onRetry: options.onRetry,
+    onUsage: options.onUsage,
     onToolCall: options.onToolCall,
     onToolResult: options.onToolResult,
     onContextOverflow: options.onContextOverflow,

package/ts_agent/smartagent.interfaces.ts

@@ -54,8 +54,69 @@ export type TAgentToolCallFinishEvent =
       error: string;
     });
 
+/** Token usage of model calls. A count a provider leaves out counts as zero. */
+export interface IAgentUsage {
+  inputTokens: number;
+  outputTokens: number;
+  totalTokens: number;
+  cacheReadTokens: number;
+  cacheWriteTokens: number;
+}
+
+interface IAgentModelCallIdentity {
+  /** The provider of the model that was called, as the language model names it. */
+  provider: string;
+  /**
+   * The model id the call was made with: the language model's `modelId`. Every call carries it,
+   * whatever its outcome, so key usage caps on `provider` and `requestedModelId`.
+   */
+  requestedModelId: string;
+}
+
+/** Why a model call ended without the provider reporting its usage. */
+export type TAgentModelCallUnreportedReason = 'aborted' | 'failed' | 'missing';
+
+/**
+ * The usage outcome of one model call, as the component that made the call reports it.
+ * `reported`: the provider reported the call's usage. `responseModelId` is the model id the
+ * provider's response named, or `requestedModelId` when the response named none.
+ * `unreported`: the call ended before the provider reported its usage: `aborted` (the call was
+ * aborted), `failed` (the call or its response failed) or `missing` (the response carried no usage).
+ * The provider may still have consumed tokens for it; their number is unknown.
+ */
+export type TAgentModelCallUsage =
+  | (IAgentModelCallIdentity & {
+      status: 'reported';
+      responseModelId: string;
+      usage: IAgentUsage;
+    })
+  | (IAgentModelCallIdentity & {
+      status: 'unreported';
+      reason: TAgentModelCallUnreportedReason;
+    });
+
+/** Receives the usage outcome of each model call. Must not throw. */
+export type TAgentModelCallUsageReporter = (call: TAgentModelCallUsage) => void;
+
+/**
+ * The usage outcome of one model call of a session, reported once per call through `onUsage`.
+ * `source: 'generation'`: a model step of the generation `generationId`.
+ * `source: 'compaction'`: a model call of a context compaction. `generationId` names the generation
+ * whose context overflow caused it; it is absent for `compact()` and event retention.
+ */
+export type TAgentModelCallUsageEvent = TAgentModelCallUsage & (
+  | { source: 'generation'; generationId: string }
+  | { source: 'compaction'; generationId?: string }
+);
+
 export interface IAgentContextOverflowInvocationOptions {
   abortSignal?: AbortSignal;
+  /**
+   * Reports the usage of each model call the handler makes, into the session's `onUsage` as
+   * `source: 'compaction'`. The session always supplies it. Report every call before the returned
+   * promise settles, including calls that fail or are aborted.
+   */
+  reportUsage?: TAgentModelCallUsageReporter;
 }
 
 export interface IAgentContextBuildOptions {
@@ -69,6 +130,12 @@ export type TAgentContextBuilder = (
 export interface IAgentContextCompactionOptions {
   abortSignal?: AbortSignal;
   reason: 'context-overflow' | 'retention' | 'manual';
+  /**
+   * Reports the usage of each model call the compactor makes, into the session's `onUsage` as
+   * `source: 'compaction'`. The session always supplies it. Report every call before the returned
+   * promise settles, including calls that fail or are aborted.
+   */
+  reportUsage?: TAgentModelCallUsageReporter;
 }
 
 export type TAgentContextCompactor = (
@@ -153,6 +220,16 @@ export interface IAgentSessionOptions {
   onToolCallFinish?: (event: TAgentToolCallFinishEvent) => void;
   /** Called before the engine waits to retry a rate-limited or unavailable model call. */
   onRetry?: (event: IAgentRetryEvent) => void;
+  /**
+   * Called once for every model call of the session, as soon as its usage is known: when the
+   * provider reports it, or when the call ends without it. This covers the generation's model
+   * steps, the model call of `compactMessages()` when a compactor passes it `reportUsage`, and every
+   * call a `contextCompactor` or `onContextOverflow` handler reports through `reportUsage`. The
+   * reported calls of a generation, including those of a compaction its context overflow caused,
+   * sum to the generation's `usage` when it returns. Must not throw; an error it throws is reported
+   * like a session listener error and does not change the generation's outcome.
+   */
+  onUsage?: (event: TAgentModelCallUsageEvent) => void;
   /** @deprecated Use onToolCallStart instead. */
   onToolCall?: (toolName: string, input: unknown) => void;
   /** @deprecated Use onToolCallFinish instead. */
@@ -303,14 +380,11 @@ export interface IAgentRunResult {
   steps: number;
   /** Finish reason from the final step */
   finishReason: string;
-  /** Accumulated token usage across all steps */
-  usage: {
-    inputTokens: number;
-    outputTokens: number;
-    totalTokens: number;
-    cacheReadTokens: number;
-    cacheWriteTokens: number;
-  };
+  /**
+   * Token usage the provider reported for every model call of the run: its model steps, retried
+   * calls included, and the calls of a compaction its context overflow caused that were reported
+   */
+  usage: IAgentUsage;
   /** Tool calls observed during the run, including inputs and outputs/errors when available */
   toolCalls: IAgentToolCallRecord[];
 }

package/ts_agent/smartagent.usage.ts

@@ -0,0 +1,66 @@
+import type { LanguageModelUsage } from './plugins.js';
+import type {
+  IAgentUsage,
+  TAgentModelCallUnreportedReason,
+  TAgentModelCallUsageReporter,
+} from './smartagent.interfaces.js';
+
+/** The identity of the language model whose calls a recorder follows. */
+export interface IAgentModelCallUsageRecorderModel {
+  readonly provider: string;
+  readonly modelId: string;
+}
+
+/**
+ * Follows the calls of one language model and reports each call's usage exactly once.
+ * Call `start()` when a model call begins, `end()` when its provider reports usage, and `settle()`
+ * when the call ends otherwise. A call that is still open when the next one starts ended without
+ * usage and is reported as `unreported` with reason `missing`.
+ */
+export class AgentModelCallUsageRecorder {
+  private open = false;
+
+  constructor(
+    private readonly model: IAgentModelCallUsageRecorderModel,
+    private readonly report: TAgentModelCallUsageReporter,
+  ) {}
+
+  /** A model call begins. */
+  public start(): void {
+    this.settle('missing');
+    this.open = true;
+  }
+
+  /** The provider reported the usage of the current call. A count it leaves out counts as zero. */
+  public end(responseModelId: string, providerUsage: LanguageModelUsage): void {
+    this.open = false;
+    const inputTokens = providerUsage.inputTokens ?? 0;
+    const outputTokens = providerUsage.outputTokens ?? 0;
+    const usage: IAgentUsage = {
+      inputTokens,
+      outputTokens,
+      totalTokens: inputTokens + outputTokens,
+      cacheReadTokens: providerUsage.inputTokenDetails.cacheReadTokens ?? 0,
+      cacheWriteTokens: providerUsage.inputTokenDetails.cacheWriteTokens ?? 0,
+    };
+    this.report({
+      status: 'reported',
+      provider: this.model.provider,
+      requestedModelId: this.model.modelId,
+      responseModelId,
+      usage,
+    });
+  }
+
+  /** The current call ended without usage. Does nothing when no call is open. */
+  public settle(reason: TAgentModelCallUnreportedReason): void {
+    if (!this.open) return;
+    this.open = false;
+    this.report({
+      status: 'unreported',
+      provider: this.model.provider,
+      requestedModelId: this.model.modelId,
+      reason,
+    });
+  }
+}