@rarebit-one/harness-kernel 0.10.0 → 0.11.0-next.0

package/README.md

@@ -46,9 +46,9 @@ ESM-only, Node >= 22.
 | Area | Exports |
 |------|---------|
 | **Model seam** | `ModelInvocation` — one arrow for every model kind, with declared capabilities, a uniform result envelope and `probe()` |
-| **Providers** | `Provider` interface + `selectProvider()` over Anthropic, OpenAI, OpenRouter, a self-hosted `local` (any OpenAI-compatible server), and an offline `mock`; `chatModel()` puts any of them on the seam |
-| **Agent loop** | `runAgent()` — a provider-neutral tool-use loop with step and wall-clock budgets, cancellable through an `AbortSignal` |
-| **Loop** | `Loop` + `nativeLoop` — the control loop as a seam; the kernel ships one and it is the loop `runAgent` has always run |
+| **Providers** | `Provider` interface + `selectProvider()` over Anthropic, OpenAI, OpenRouter, a self-hosted `local` (any OpenAI-compatible server), an offline `mock` (by request) and a `ScriptedProvider` for recorded evals; `chatModel()` puts any of them on the seam; typed per-call `ProviderOptions` (prompt caching, token cap, response format, OpenRouter routing) |
+| **Agent loop** | `runConversation()` — a provider-neutral tool-use loop over a seeded conversation, with step and wall-clock budgets, terminal (final-answer) tools, summed usage and the served model in the result; `runAgent()` is its one-prompt form |
+| **Loop** | `Loop` + `nativeLoop` — the control loop as a seam; the kernel ships one and it is the loop `runAgent` and `runConversation` have always run |
 | **Run events** | `RunEvent` — an ordered, structured account of a run (turns, tool calls, budgets, outcome), emitted to an optional sink |
 | **Tools** | `Tool` (+ `ToolContext`), `primitiveTools()` (`run_code`, `read_file`, `list_files`, `http_fetch`), `connectorTools()`, `modelAsTool()`, metadata + projections (`replaySafeTools`, …) |
 | **Connectors** | `connectMcp()` — MCP clients over stdio / streamable HTTP / SSE |
@@ -297,9 +297,145 @@ const text = await runAgent({
 })
 ```
 
-With no provider API key configured, `selectProvider` falls back to the offline
-`mock` provider — so the whole loop runs end to end with no keys and no network.
-That is deliberate: the test suite depends on it, and so can yours.
+**`selectProvider` fails loud.** A preferred provider with no credential, or no
+preference and no credential at all, throws a `ProviderCredentialsError` naming
+the env var and the BYO slot that would have satisfied it; it never falls
+through to another configured provider, and a name it does not know (a typo, a
+provider with no adapter) is refused the same way rather than read as "no
+preference". `providerAvailability()` gives the same
+answer as a reason instead of a throw, and `NativeEngine.supports()` uses it, so
+a run with no key is refused before a sandbox is prepared. The offline `mock`
+is reached only by request (`selectProvider("mock")`) or by explicit opt-in
+(`{ allowMockFallback: true }`), which is what a keyless test rig sets. Until
+0.11 a keyless run landed on the mock silently and reported success; nothing on
+the wire told that run from a real one.
+
+### Conversations and terminal tools
+
+`runAgent` is single-shot. A conversational application seeds the loop with its
+stored history and gets the whole transcript back:
+
+```ts
+import { runConversation, terminalTool } from "@rarebit-one/harness-kernel"
+
+const reply = terminalTool({
+  name: "reply",
+  description: "Send the final answer to the user.",
+  inputSchema: {
+    type: "object",
+    properties: { text: { type: "string" }, quick_replies: { type: "array", items: { type: "string" } } },
+    required: ["text"],
+  },
+})
+
+const result = await runConversation({
+  provider,
+  system,
+  messages: [...storedHistory, { role: "user", text: incoming }],
+  tools: [...lookupTools, reply],
+  providerOptions: { promptCache: {} },
+})
+
+result.stopReason // "end_turn" | "terminal_tool" | "steps_exhausted" | "timed_out" | "canceled"
+result.terminal?.input // { text, quick_replies } when the model called `reply`
+result.messages // store this; seed the next turn with it
+result.usage // summed over every model call
+result.provider, result.model // what actually served the run
+```
+
+A tool with `meta.dispatch: "terminal"` is a **final-answer tool**: the loop
+still runs its executor (so it can validate, or send), appends the tool result
+so the transcript carries no dangling call, and then ends the run with
+`stopReason: "terminal_tool"` and the call on `result.terminal`. An executor
+that throws turns the call into an ordinary tool error the model gets to
+correct. `terminalTool()` builds one with a no-op executor; the structured
+answer is the call's `input`, which the kernel does not validate against the
+schema (run your own validator on it, or supply an executor that throws).
+
+The seed must end with a user message or with the tool results the model is
+waiting for; the loop copies it and never mutates the array it was handed.
+`result.events` is this call's event stream (the same events reached `emit`),
+recorded for a caller with nowhere else to put them; the kernel still stores
+nothing between calls.
+
+`dispatch: "external"` is reserved for a tool the *caller* executes: the loop
+would return the pending call and resume when given its result. The native
+loop refuses it today (a custom `Loop` may honour it); native suspend/resume is
+a later release.
+
+### Provider options and prompt caching
+
+`providerOptions` rides on every model call a run makes and each adapter maps it
+onto its own wire format. An option an adapter cannot honour throws a
+`ProviderOptionsError` rather than being dropped.
+
+```ts
+providerOptions: {
+  maxTokens: 1024,                       // per-call completion cap
+  promptCache: { ttl: "1h" },            // system + tools + last message cached
+  responseFormat: { type: "json_object" }, // OpenAI-compatible adapters only
+  openrouter: {                          // OpenRouter adapter only
+    models: ["anthropic/claude-sonnet-4.5", "openai/gpt-4.1"], // fallbacks
+    provider: { data_collection: "deny", zdr: true },         // data policy
+    usage: { include: true },
+  },
+}
+```
+
+`promptCache: {}` means "cache this prompt the usual way". On **Anthropic** that
+is a `cache_control` breakpoint on the system prompt, the last tool definition
+and the last message (`trailingMessages` up to 2, `ttl` `5m` or `1h`). On
+**OpenRouter** the breakpoints go on the system prompt and the trailing
+messages as content parts, which the upstreams that cache (Anthropic, Gemini)
+honour. On **OpenAI** caching is automatic, so only `promptCache.key` does
+anything (it becomes `prompt_cache_key`). The `local` adapter accepts it and
+lets the server's own prefix cache do the work.
+
+Every result reports what served it: `ConverseResult.model` is the model id the
+response named (with a fallback list it can differ from the one requested),
+`responseId` is the provider's own id for the response (a generation id on
+OpenRouter), and, behind OpenRouter, `upstream` names who actually processed the
+prompt, which is what a zero-data-retention audit needs. `model.turn` events
+carry `model`, `responseId` and `upstream` too; `ConversationResult.responseIds`
+lists every id in call order and `ConversationResult.upstream` names who served
+the last turn, so a cross-vendor fallback chain can still be audited.
+
+The package also exports its own `package.json`
+(`import pkg from "@rarebit-one/harness-kernel/package.json" with { type: "json" }`)
+so a service can report the kernel version it runs.
+
+Usage carries what the provider prices: beside the token counts, `TokenUsage`
+has `cacheWriteTokens`, `reasoningTokens` and `reportedCost`
+(`{ amount, currency: "USD" }`, from OpenRouter's `usage.cost`) when the
+response reports them, on every `model.turn` event and summed by amount on the
+result, so a per-turn cost cap can abort the run through its `signal`. Absent
+means unknown, never free.
+
+Two more per-call knobs: `maxRetries` overrides the SDK retry count for one
+call (set `0` alongside an OpenRouter `models` chain, or the SDK and the router
+both retry), and the `openai` adapter sends the cap as `max_completion_tokens`
+(its current field; reasoning models reject `max_tokens`) while OpenRouter and
+local servers get `max_tokens`. `connectorTools(..., { toolName })` names MCP
+tools your way when the default `<connector>__<tool>` prefix is in the way.
+
+### Recorded evals with `ScriptedProvider`
+
+```ts
+import { ScriptedProvider } from "@rarebit-one/harness-kernel"
+
+const provider = new ScriptedProvider([
+  { text: "", toolCalls: [{ id: "c1", name: "lookup", input: { q: "balance" } }] },
+  (req) => ({ text: "", toolCalls: [{ id: "r1", name: "reply", input: { text: "S$120" } }] }),
+])
+const result = await runConversation({ provider, system, messages, tools })
+provider.requests // every request the loop built, for assertions
+provider.remaining // 0 when the whole script was consumed; a longer loop throws
+```
+
+It plays a recorded script of model turns in order, records each request it was
+given, honours `signal`, and throws `ScriptExhaustedError` rather than invent a
+turn, so an eval that drifts from its recording fails instead of passing
+quietly.
 
 **Self-hosted (`local`).** `selectProvider("local")` runs against any
 OpenAI-compatible server — llama-server, vLLM, LM Studio, Ollama's OpenAI

package/dist/agent.d.ts

@@ -1,9 +1,12 @@
-import { type RunEventSink } from "./events.js";
-import { type LoopContext, type LoopRequest, type LoopResult } from "./loop.js";
+import { type RunEvent, type RunEventSink, type RunOutcome, type StopReason } from "./events.js";
+import { type LoopContext, type LoopRequest, type LoopResult, type TerminalCall } from "./loop.js";
 import { type ChatModel } from "./models/chat.js";
-import type { Provider } from "./providers/types.js";
+import type { TokenUsage } from "./models/types.js";
+import type { ProviderOptions } from "./providers/options.js";
+import type { AgentMessage, Provider } from "./providers/types.js";
 import { type Tool } from "./tools/registry.js";
-export interface RunAgentOptions {
+/** What {@link runAgent} and {@link runConversation} share. */
+export interface LoopRunOptions {
     /**
      * The chat model driving the loop. A bare {@link Provider} is accepted and
      * adapted onto the model seam internally, so existing callers are unaffected;
@@ -12,8 +15,13 @@ export interface RunAgentOptions {
      */
     provider: Provider | ChatModel;
     system: string;
-    userPrompt: string;
     tools: Tool[];
+    /**
+     * Per-call provider options — token cap, prompt caching, response format,
+     * OpenRouter routing — sent on every model call the run makes. See
+     * `providers/options.ts`.
+     */
+    providerOptions?: ProviderOptions;
     maxSteps?: number;
     /**
      * Wall-clock budget for the whole loop in ms. Enforced both between steps and
@@ -39,13 +47,62 @@ export interface RunAgentOptions {
      */
     emit?: RunEventSink;
 }
+/** A single-shot run: one user prompt in, the final prose out. */
+export interface RunAgentOptions extends LoopRunOptions {
+    userPrompt: string;
+}
+/** A conversational run: the history so far in, the whole transcript out. */
+export interface RunConversationOptions extends LoopRunOptions {
+    /**
+     * The conversation so far, oldest first, ending with the new user message
+     * (or with the tool results the model is waiting for). The array is not
+     * mutated; the result carries a new one.
+     */
+    messages: AgentMessage[];
+}
+/**
+ * What {@link runConversation} returns: everything a conversational caller
+ * needs to store the turn, bill it, and answer with it.
+ */
+export interface ConversationResult {
+    /** The final assistant prose. Empty when a terminal tool answered without prose. */
+    text: string;
+    /** The seed followed by every message the loop appended. Store this; seed the next turn with it. */
+    messages: AgentMessage[];
+    /**
+     * This run's full event stream, in order. The same events reached `emit` as
+     * they happened; this is a per-call recording for a caller with nowhere else
+     * to put them, not a store — nothing is kept between calls.
+     */
+    events: RunEvent[];
+    /** Token usage summed over every model call that reported it; absent if none did. */
+    usage?: TokenUsage;
+    steps: number;
+    outcome: RunOutcome;
+    stopReason: StopReason;
+    /** The terminal-tool call that ended the run, when one did. Its `input` is the structured answer. */
+    terminal?: TerminalCall;
+    /** Which adapter served the run (`"anthropic"`, `"openrouter"`, `"mock"`, a custom model's id). */
+    provider: string;
+    /** The model id the last response named, else the adapter's configured model. */
+    model?: string;
+    /** Every response id the provider reported, in call order. */
+    responseIds: string[];
+    /**
+     * Behind an aggregator, who actually served the last response (OpenRouter's
+     * response-level `provider`, e.g. "Anthropic"). This is what a data-policy
+     * audit reads: `provider` names the adapter, `upstream` names the vendor the
+     * prompt reached. Absent for direct adapters.
+     */
+    upstream?: string;
+}
 /**
  * Resolve a partial budget to the concrete numbers a {@link LoopRequest} needs.
  *
  * One place, deliberately. A {@link Loop} is handed numbers rather than
  * optionals, so if each caller resolved its own defaults a second loop
  * implementation could silently run to a different ceiling than the one the
- * kernel ships. `runAgent` and `NativeEngine` both come here.
+ * kernel ships. `runAgent`, `runConversation` and `NativeEngine` all come here.
  */
 export declare function resolveLoopLimits(limits?: {
     maxSteps?: number | undefined;
@@ -66,15 +123,30 @@ export declare function maxToolResultBytes(): number;
  * executes any requested tools in the sandbox, feeds the results back, and
  * repeats until the model stops calling tools (or the step budget is hit). The
  * model orchestrates; deterministic work happens in the tools.
+ *
+ * This is the single-shot form: one user prompt in, the final prose out. It is
+ * {@link runConversation} with a one-message seed, kept because most callers
+ * want exactly that and nothing more.
  */
 export declare function runAgent(options: RunAgentOptions): Promise<string>;
+/**
+ * Run the tool-use loop over a conversation: the stored history plus the new
+ * user turn in, the whole transcript out, with the structured answer when a
+ * terminal tool gave one.
+ *
+ * The kernel stores nothing between calls. A conversational application keeps
+ * `result.messages` wherever it keeps its state and seeds the next call with
+ * it; what the kernel owes it is a faithful transcript, a summed bill, the
+ * reason the run stopped, and the name of what served it.
+ */
+export declare function runConversation(options: RunConversationOptions): Promise<ConversationResult>;
 /**
  * The native tool-use loop, as a {@link Loop}.
  *
  * Exported for `loop.ts` to wrap; applications reach it through `nativeLoop`
  * rather than calling this directly. It is the whole of the kernel's control
- * flow, and the only place any of it lives — `runAgent` and `nativeLoop` both
- * come here, so the two cannot drift.
+ * flow, and the only place any of it lives — `runAgent`, `runConversation` and
+ * `nativeLoop` all come here, so they cannot drift.
  */
 export declare function runNativeLoop(req: LoopRequest, loopCtx: LoopContext): Promise<LoopResult>;
 //# sourceMappingURL=agent.d.ts.map

package/dist/agent.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"agent.d.ts","sourceRoot":"","sources":["../src/agent.ts"],"names":[],"mappings":"AAAA,OAAO,EAEL,KAAK,YAAY,EAGlB,MAAM,aAAa,CAAA;AACpB,OAAO,EAGL,KAAK,WAAW,EAChB,KAAK,WAAW,EAChB,KAAK,UAAU,EAChB,MAAM,WAAW,CAAA;AAClB,OAAO,EAAe,KAAK,SAAS,EAAE,MAAM,kBAAkB,CAAA;AAG9D,OAAO,KAAK,EAAgB,QAAQ,EAAc,MAAM,sBAAsB,CAAA;AAC9E,OAAO,EAAe,KAAK,IAAI,EAAE,MAAM,qBAAqB,CAAA;AAG5D,MAAM,WAAW,eAAe;IAC9B;;;;;OAKG;IACH,QAAQ,EAAE,QAAQ,GAAG,SAAS,CAAA;IAC9B,MAAM,EAAE,MAAM,CAAA;IACd,UAAU,EAAE,MAAM,CAAA;IAClB,KAAK,EAAE,IAAI,EAAE,CAAA;IACb,QAAQ,CAAC,EAAE,MAAM,CAAA;IACjB;;;;;OAKG;IACH,aAAa,CAAC,EAAE,MAAM,CAAA;IACtB;;;;OAIG;IACH,MAAM,CAAC,EAAE,WAAW,CAAA;IACpB,4DAA4D;IAC5D,KAAK,CAAC,EAAE,MAAM,CAAA;IACd,GAAG,CAAC,EAAE,CAAC,IAAI,EAAE,MAAM,KAAK,IAAI,CAAA;IAC5B;;;;;OAKG;IACH,IAAI,CAAC,EAAE,YAAY,CAAA;CACpB;AAYD;;;;;;;GAOG;AACH,wBAAgB,iBAAiB,CAAC,MAAM,CAAC,EAAE;IACzC,QAAQ,CAAC,EAAE,MAAM,GAAG,SAAS,CAAA;IAC7B,aAAa,CAAC,EAAE,MAAM,GAAG,SAAS,CAAA;CACnC,GAAG;IAAE,QAAQ,EAAE,MAAM,CAAC;IAAC,aAAa,EAAE,MAAM,CAAA;CAAE,CAK9C;AAED;;;;;GAKG;AACH,wBAAgB,kBAAkB,IAAI,MAAM,CAG3C;AAqDD;;;;;GAKG;AACH,wBAAsB,QAAQ,CAAC,OAAO,EAAE,eAAe,GAAG,OAAO,CAAC,MAAM,CAAC,CAsBxE;AAED;;;;;;;GAOG;AACH,wBAAsB,aAAa,CAAC,GAAG,EAAE,WAAW,EAAE,OAAO,EAAE,WAAW,GAAG,OAAO,CAAC,UAAU,CAAC,CAgP/F"}
+{"version":3,"file":"agent.d.ts","sourceRoot":"","sources":["../src/agent.ts"],"names":[],"mappings":"AAAA,OAAO,EAEL,KAAK,QAAQ,EACb,KAAK,YAAY,EACjB,KAAK,UAAU,EACf,KAAK,UAAU,EAEhB,MAAM,aAAa,CAAA;AACpB,OAAO,EAGL,KAAK,WAAW,EAChB,KAAK,WAAW,EAChB,KAAK,UAAU,EACf,KAAK,YAAY,EAClB,MAAM,WAAW,CAAA;AAClB,OAAO,EAA4B,KAAK,SAAS,EAAE,MAAM,kBAAkB,CAAA;AAE3E,OAAO,KAAK,EAAiB,UAAU,EAAS,MAAM,mBAAmB,CAAA;AACzE,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,wBAAwB,CAAA;AAC7D,OAAO,KAAK,EAAE,YAAY,EAAE,QAAQ,EAAwB,MAAM,sBAAsB,CAAA;AACxF,OAAO,EAAe,KAAK,IAAI,EAAE,MAAM,qBAAqB,CAAA;AAG5D,+DAA+D;AAC/D,MAAM,WAAW,cAAc;IAC7B;;;;;OAKG;IACH,QAAQ,EAAE,QAAQ,GAAG,SAAS,CAAA;IAC9B,MAAM,EAAE,MAAM,CAAA;IACd,KAAK,EAAE,IAAI,EAAE,CAAA;IACb;;;;OAIG;IACH,eAAe,CAAC,EAAE,eAAe,CAAA;IACjC,QAAQ,CAAC,EAAE,MAAM,CAAA;IACjB;;;;;OAKG;IACH,aAAa,CAAC,EAAE,MAAM,CAAA;IACtB;;;;OAIG;IACH,MAAM,CAAC,EAAE,WAAW,CAAA;IACpB,4DAA4D;IAC5D,KAAK,CAAC,EAAE,MAAM,CAAA;IACd,GAAG,CAAC,EAAE,CAAC,IAAI,EAAE,MAAM,KAAK,IAAI,CAAA;IAC5B;;;;;OAKG;IACH,IAAI,CAAC,EAAE,YAAY,CAAA;CACpB;AAED,kEAAkE;AAClE,MAAM,WAAW,eAAgB,SAAQ,cAAc;IACrD,UAAU,EAAE,MAAM,CAAA;CACnB;AAED,6EAA6E;AAC7E,MAAM,WAAW,sBAAuB,SAAQ,cAAc;IAC5D;;;;OAIG;IACH,QAAQ,EAAE,YAAY,EAAE,CAAA;CACzB;AAED;;;GAGG;AACH,MAAM,WAAW,kBAAkB;IACjC,oFAAoF;IACpF,IAAI,EAAE,MAAM,CAAA;IACZ,oGAAoG;IACpG,QAAQ,EAAE,YAAY,EAAE,CAAA;IACxB;;;;OAIG;IACH,MAAM,EAAE,QAAQ,EAAE,CAAA;IAClB,qFAAqF;IACrF,KAAK,CAAC,EAAE,UAAU,CAAA;IAClB,KAAK,EAAE,MAAM,CAAA;IACb,OAAO,EAAE,UAAU,CAAA;IACnB,UAAU,EAAE,UAAU,CAAA;IACtB,qGAAqG;IACrG,QAAQ,CAAC,EAAE,YAAY,CAAA;IACvB,mGAAmG;IACnG,QAAQ,EAAE,MAAM,CAAA;IAChB,iFAAiF;IACjF,KAAK,CAAC,EAAE,MAAM,CAAA;IACd,8DAA8D;IAC9D,WAAW,EAAE,MAAM,EAAE,CAAA;IACrB;;;;;OAKG;IACH,QAAQ,CAAC,EAAE,MAAM,CAAA;CAClB;AAYD;;;;;;;GAOG;AACH,wBAAgB,iBAAiB,CAAC,MAAM,CAAC,EAAE;IACzC,QAAQ,CAAC,EAAE,MAAM,GAAG,SAAS,CAAA;IAC7B,aAAa,CAAC,EAAE,MAAM,GAAG,SAAS,CAAA;CACnC,GAAG;IAAE,QAAQ,EAAE,MAAM,CAAC;IAAC,aAAa,EAAE,MAAM,CAAA;CAAE,CAK9C;AAED;;;;;GAKG;AACH,wBAAgB,kBAAkB,IAAI,MAAM,CAG3C;AAsHD;;;;;;;;;GASG;AACH,wBAAsB,QAAQ,CAAC,OAAO,EAAE,eAAe,GAAG,OAAO,CAAC,MAAM,CAAC,CAOxE;AAED;;;;;;;;;GASG;AACH,wBAAsB,eAAe,CACnC,OAAO,EAAE,sBAAsB,GAC9B,OAAO,CAAC,kBAAkB,CAAC,CA8C7B;AAED;;;;;;;GAOG;AACH,wBAAsB,aAAa,CAAC,GAAG,EAAE,WAAW,EAAE,OAAO,EAAE,WAAW,GAAG,OAAO,CAAC,UAAU,CAAC,CAoV/F"}