orbitx 0.2.6 → 0.2.8

package/docs/agents.md

@@ -11,26 +11,36 @@
 ```ts
 new BaseAgent({
   instruction: string;
+  safetyPolicies?: string;   // default: "" — see "Guardrail system prompt" below
   allowedTools: MCPTool<any>[];
   aiProvider: AIProvider | AgentProviderEntry[];   // see Providers doc
   mcpClient: MCPClient;
   skills?: Skill[];
   maxMemorizeToken?: number;   // default: derived from the main provider's contextWindow * safeUsageRatio
   initData?: {
-    memory: string;
-    messagesFull: Message[];
-    fullInputMissTokens: number;
-    fullInputHitTokens: number;
-    fullOutputTokens: number;
-    messagesCompact: Message[];
-    currentInputMissTokens: number;
-    currentInputHitTokens: number;
-    currentOutputTokens: number;
+    compactMemory: string;
+    retiredMessages: Message[];   // complete, ever-appended history of retired messages — each Message carries its own usage/timestamp
+    messagesCompact: Message[];   // the working context currently replayed to the provider every turn
+  };
+  features?: {
+    executeProviderFromMCPTool?: boolean;   // default: false — see "executeProvider feature flag" below
   };
 });
 ```
 
-`SimpleAgent`'s constructor is the same shape minus `mcpClient`/`allowedTools` (it builds those from `tools`), plus `tools?: MCPTool<any>[]` and `maxMemorizeToken` defaulting to `16000` instead of being derived.
+### Guardrail system prompt
+
+`safetyPolicies` is a separate `role: "system"` message, sent as its own leading entry ahead of the regular system prompt (see [Instructions, skills, and memory](#instructions-skills-and-memory-in-the-system-prompt)) on every main-provider `chat()` call in the run loop. Keep house rules/guardrails here rather than folding them into `instruction`, since it's a separate, cache-stable prefix ahead of the rest of the system prompt.
+
+It is **not** prepended on an `executeProvider` call (see below) — stuffing a policy paragraph ahead of a narrow, single-purpose call (e.g. "describe this image") measurably degrades that call's output, and not every registered provider role even accepts a `system` message the way the main LLM does. A pre-dispatch content-safety check is the intended replacement there; not implemented yet.
+
+### `executeProvider` feature flag
+
+A tool's `execute()` can reach a registered `AIProvider` directly via `mcp.executeProvider(toolCallId, type, input)` (see [Tools](./tools.md#mcpexecuteprovider--calling-a-provider-from-inside-a-tool)) — a bigger trust surface than a tool that only returns data to the model, since it lets a tool spend real provider calls (and, per the point above, without a guardrail system prompt ahead of them). It's off by default: `BaseAgent#executeProvider` throws immediately unless `features.executeProviderFromMCPTool` was set `true` on construction. This gates every role uniformly (`image-describer` included) — enable it once you're ready to trust the tools you've allowed with direct provider access.
+
+`SimpleAgent`'s constructor is the same shape minus `mcpClient`/`allowedTools` (it builds those from `tools`), plus `tools?: MCPTool<any>[]` and `maxMemorizeToken` defaulting to `16000` instead of being derived. `safetyPolicies` and `features` pass straight through to the underlying `BaseAgent`.
+
+Token accounting no longer lives in `initData` as separate counters — every `Message` carries its own `usage` (see [Token accounting](#token-accounting)), so `retiredMessages`/`messagesCompact` alone are enough to resume with full token history intact.
 
 ## The run loop
 
@@ -60,6 +70,8 @@ Practical implication: don't treat an early queued call's resolved `run()` promi
 
 `immediateStop()` requests the same stop but returns right away without waiting for confirmation, **and** aborts the in-flight `chat()` call to the main provider immediately — the underlying HTTP request is cancelled rather than left to run to completion, so no further output tokens get generated (and billed) past that point. Input tokens, and any output already generated before the abort landed, are already committed — cancelling can't retroactively make those free. This is the one to reach for when a call has to be killed regardless of how far through it is (e.g. a credit balance that just went negative). A tool dispatch already in progress isn't aborted (tools have no generic cancellation hook) — the loop stops as soon as that call returns. Abort support is real for Anthropic, OpenAI, and DeepSeek (both streaming and non-streaming); for Ollama it only takes effect on streaming calls — see the note in `ollama-provider.ts`.
 
+If you're deciding *when* to call `immediateStop()` from outside — e.g. against an external credit/cost system — watch `usage` on the `StreamCallback` chunks (see [Streaming](./streaming.md)) rather than waiting for `run()` to resolve: it's the same callback passed into `run()`, forwarded straight through to the provider, so external code sees the same numbers `BaseAgent` itself uses to build `Message.usage`. Note this is only genuinely live for Anthropic — the other three providers only report usage once a turn is already complete, so for them there's no earlier signal to react to, cancellation-timing-wise, than the final chunk.
+
 For a **streaming** call, an abort doesn't throw the turn away: `chat()` resolves normally with whatever text, thinking, and tool calls were already accumulated before the cutoff — the same as if the model had naturally stopped there — so anything already sent to `streamCallback` still ends up recorded in message history instead of vanishing. A tool call the model was still in the middle of generating when the abort hit is dropped even then (its arguments may be truncated JSON); only tool calls that had already fully finished are included — e.g. if the model requested 3 tool calls and the abort landed while the 3rd was still streaming, only the first 2 come back. For a **non-streaming** call there's nothing to salvage (the API only returns a response once, in full) — abort there still surfaces as a thrown error, and `BaseAgent` discards that turn entirely rather than recording a broken one.
 
 `stop()` is a deprecated alias for `safeStop()` and will be removed in the 1.0.0 major release.
@@ -74,37 +86,50 @@ OrbitX never truncates or deletes conversation history. Compaction happens throu
 
 Calling `compact_memory`:
 
-1. Replaces the running `memory` string with `new_memory`, which is injected into every future system prompt (`MEMORY:` section).
-2. Clears `messagesCompact` — the *working* context sent to the provider each turn.
-3. Leaves `messagesFull` — the complete, ever-appended history — untouched, so nothing is lost; it's just no longer replayed to the model every turn.
-4. Rolls the current token counters into the lifetime totals (see [Token accounting](#token-accounting)) and resets them to zero.
+1. Replaces the running `compactMemory` string with `new_memory`, which is injected into every future system prompt (`MEMORY:` section). The cached system prompt is invalidated at the same time, so the very next turn's system prompt is rebuilt with the new summary rather than replaying the stale one.
+2. Moves everything currently in `messagesCompact` — the *working* context sent to the provider each turn — into `retiredMessages`, then clears `messagesCompact`. Nothing is lost, it's just retired from what gets replayed to the model every turn. Nothing needs to be separately "rolled into a lifetime total" — every message already carries its own `usage`, so `getFullTotalToken()` (see [Token accounting](#token-accounting)) just sums `retiredMessages` + `messagesCompact` directly.
+
+Before each turn, `BaseAgent` also checks the current context size: the summed `inputMissTokens` of every message's `"main"` usage entry across `messagesCompact` (see [Token accounting](#token-accounting)) — a cheap proxy for context size, not a precise replay-cost measurement. When that sum exceeds `maxMemorizeToken`, the agent doesn't compact on the model's behalf — instead it appends a `role: "user"` message telling the model to call `compact_memory` before doing anything else, and loops back for another turn. This keeps compaction on the same native tool-call path the model already uses for everything else, rather than a side-channel the model never sees; a model that ignores the nudge (rare, since the instruction is explicit) will simply see it repeated on the next turn until it complies.
 
-`BaseAgent` also tracks two token totals per run: `currentInputMissTokens` (uncached input) and `currentLastOutputTokens` (the last turn's output). When their sum exceeds `maxMemorizeToken`, the agent doesn't compact on the model's behalf — instead it appends a `role: "user"` message telling the model to call `compact_memory` before doing anything else, and loops back for another turn. This keeps compaction on the same native tool-call path the model already uses for everything else, rather than a hidden side-channel; a model that ignores the nudge (rare, since the instruction is explicit) will simply see it repeated on the next turn until it complies.
+This nudge message (and its matching `StreamCallback` chunk) is marked `hidden: true`, since it's `BaseAgent` talking to the model on the caller's behalf, not something a human said or the model produced — a chat UI built on this SDK should filter it out of what it shows an end user, the same way it wouldn't show raw system-prompt text. See [Streaming](./streaming.md) for the full `Message`/chunk field reference.
 
 `maxMemorizeToken`, if not passed explicitly, is derived as `contextWindow * safeUsageRatio` (default ratio `0.5`) from the main provider's `getCapabilities()` — so it scales automatically with whatever model you plug in.
 
 ## Persisting and resuming state
 
-`getCurrentAgentStates()` returns exactly the shape expected by the `initData` constructor option — `memory`, `messagesFull`, `messagesCompact`, the six main-loop token counters, and the two image-usage counters (`imageInputMissTokens`, `imageOutputTokens` — see [Token accounting](#token-accounting)). Serialize it (it's plain JSON) after any `run()` call, and pass it back into a fresh `BaseAgent`/`SimpleAgent` to resume — same conversation, same memory, same token history. See the example in [Getting Started](./getting-started.md#recovering-an-agents-state).
+`getCurrentAgentStates()` returns exactly the shape expected by the `initData` constructor option — `compactMemory`, `retiredMessages`, `messagesCompact`. Serialize it (it's plain JSON) after any `run()` call, and pass it back into a fresh `BaseAgent`/`SimpleAgent` to resume — same conversation, same memory, same token history, since every message already carries its own `usage`. See the example in [Getting Started](./getting-started.md#recovering-an-agents-state).
 
-The two image counters are optional on `initData` — a snapshot taken before image-usage tracking existed still loads, with both defaulting to `0`.
+A snapshot taken with an older SDK version, before per-message `usage` existed, still loads fine — messages with no `usage` field just contribute `0` to every token total below.
 
 ## Token accounting
 
-Four read methods, all returning `{ total, inputHit, inputMiss, output }`:
+Token usage lives on `Message.usage` (`MessageUsage[]`), not on separate counters — `BaseAgent` populates it as `chat()` responses come back, and the four read methods below are pure derived sums over `retiredMessages`/`messagesCompact`. There's no rollover bookkeeping to keep in sync on compaction (see [Memory compaction](#memory-compaction)): a message lives in exactly one of `retiredMessages` (retired) or `messagesCompact` (active) at a time — they're disjoint, not overlapping — so "already compacted away" is simply `retiredMessages` itself, with no slicing required. The complete history at any point is the concatenation of the two, in that order, not `retiredMessages` alone.
+
+Each entry in `Message.usage` is one of:
+
+```ts
+{ type: ProviderType; unit: "tokens"; inputMissTokens: number; inputCacheTokens: number; outputTokens: number }
+{ type: ProviderType; unit: "cost"; cost: number }
+```
+
+`type` is the provider role that produced the entry (see [Provider roles](./providers.md#provider-roles) — `"main"`, `"image-describer"`, or anything else registered/looked up). `unit` distinguishes real token billing from anything that isn't naturally token-shaped — an image/audio/3D-generation call billed as a flat price reports `unit: "cost"` instead.
+
+**Attribution** — a `type: "main"` entry is set only on the assistant message a `chat()` call produced, straight from that call's own `inputMissTokens`/`inputCacheTokens`/`outputTokens`. There's no splitting or attribution onto the user/tool messages that triggered the call, and no per-message context-size measurement — a user or tool message never carries a `"main"` entry. These are real, provider-reported numbers (Anthropic's `cache_read_input_tokens`/`cache_creation_input_tokens`, DeepSeek's `prompt_cache_hit_tokens`/`prompt_cache_miss_tokens`, OpenAI's `prompt_tokens_details.cached_tokens`), not a client-side guess — see [Providers](./providers.md) for what each one actually reports and Ollama's lack of any cache concept (`inputCacheTokens` always `0` there).
+
+The compaction trigger (see [Memory compaction](#memory-compaction)) sums `inputMissTokens` across every assistant message's `"main"` entry in `messagesCompact` as a cheap proxy for context size — not a precise measurement of what replaying `messagesCompact` right now would actually cost, since each entry reflects the call that produced that one assistant message, not the whole context at that point.
+
+Every other `type` entry comes from a tool calling `mcp.executeProvider(toolCallId, type, input)` (see [Tools](./tools.md#mcpexecuteprovider--calling-a-provider-from-inside-a-tool)) — but the entry itself is recorded by `BaseAgent`, the instant the real provider call resolves, never by the tool's own return value. This is deliberate: a tool can't under-report (or hide) what it spent, because it never holds the usage number to begin with — `BaseAgent` keys it by `toolCallId` and moves it onto that call's result `Message.usage` only after `execute()` returns, regardless of what the tool itself returns. `read-image`/`browser-screenshot`, for example, trigger a `type: "image-describer", unit: "tokens"` entry with no hit/miss split (each call is a fresh, standalone chat issued directly against the resolved provider by `BaseAgent`, so every input token counts as `inputMissTokens`).
 
-- `getCurrentTotalTokens()` — main-loop tokens used since the last memory compaction.
-- `getFullTotalTokens()` — main-loop tokens used across all *prior* compacted segments.
-- `getImageTotalTokens()` — tokens used by every `ImageDescriber` call this agent has made (see [Providers](./providers.md#two-provider-roles-main--image)), tracked separately since a one-off image-description call has no conversation history to compact and so doesn't fit the current/full split.
-- `getTotalTokens()` — the sum of all three (lifetime total for this agent instance, main loop and image calls combined).
+`executeProvider` picks `unit: "tokens"` vs `unit: "cost"` dynamically from what the provider's `ChatResponse` actually reported, rather than assuming one or the other: if the response set `cost` (a flat, non-token price — see [`ChatResponse`](./providers.md#writing-a-custom-provider)), the entry is `unit: "cost"`; otherwise it falls back to the token fields (`inputMissTokens`/`inputCacheTokens`/`outputTokens`, each defaulting to `0` if the provider omitted them). Every built-in provider (Anthropic/OpenAI/DeepSeek/Ollama) is token-billed and always takes the token path — `cost` only comes into play for a custom provider registered under an image/video/audio-generation-style role that bills per-request instead of per-token.
 
-`inputHit`/`inputMiss` on the main-loop methods track cached vs. uncached input tokens, since the compacted-context-replay pattern means most of a long conversation's prefix is a cache hit on providers that support prompt caching — this is a heuristic `BaseAgent` computes itself from the growing/reset conversation history: whatever was `inputMiss` last turn rolls into `inputHit` this turn, and `inputMiss` becomes this turn's raw input token count.
+**The four read methods:**
 
-`getImageTotalTokens()`'s `inputHit` is always `0` — each `ImageDescriber` call is a fresh, standalone chat with no prior turns to replay, so there's nothing for that heuristic to apply to. Every input token an image call reports counts as `inputMiss`.
+- `getCompactTotalTokens(type: ProviderType)` / `getFullTotalToken(type: ProviderType)` — return `{ total, inputHit, inputMiss, output }`, summing `unit: "tokens"` entries matching `type` over `messagesCompact` only, or `retiredMessages` + `messagesCompact` (this agent's whole lifetime), respectively.
+- `getCompactTotalCost(type: ProviderType)` / `getFullTotalCost(type: ProviderType)` — return a plain `number`, summing `unit: "cost"` entries matching `type` over the same two scopes.
 
 ## Instructions, skills, and memory in the system prompt
 
-The system prompt is built once (lazily, on first use) and never changes for the lifetime of an agent instance — a stable prefix, which matters for provider-side prompt caching. It's assembled from: a fixed "you have tools" preamble, each skill's name/description/instructions (see [Skills](./skills.md)), the current `MEMORY:` block, and your `instruction` string, in that order.
+The system prompt is built lazily on first use and cached — a stable prefix between compactions, which matters for provider-side prompt caching. It's assembled from: a fixed "you have tools" preamble, each skill's name/description/instructions (see [Skills](./skills.md)), the current `MEMORY:` block (sourced from `compactMemory`), and your `instruction` string, in that order. Each `compact_memory` call invalidates the cache (see [Memory compaction](#memory-compaction)), so the next turn's system prompt is rebuilt with the fresh `MEMORY:` block instead of the one that was current when the prompt was first built.
 
 ## Multi-agent: `WorkerAgent` *(experimental)*
 

package/docs/api-reference.md

@@ -6,12 +6,11 @@ Flat index of every public export from `orbitx` (see `src/index.ts`), grouped by
 
 - `AIProvider` (abstract base class)
 - `OpenAIProvider`, `AnthropicProvider`, `DeepSeekProvider`, `OllamaProvider`
-- `resolveAgentProviders()`
-- Types: `AgentProviderRole`, `AgentProviderEntry`, `AgentProvidersInput`, `ResolvedAgentProviders`
-- Types: `ChatResponse`, `Message`, `StreamCallback`, `ToolSchema`, `ToolCallRequest`, `MessageContentPart`, `ProviderCapabilities`
+- `resolveAgentProviders()`, `ProviderRegistry`
+- Types: `AgentProviderEntry`, `AgentProvidersInput`
+- Types: `ChatResponse`, `Message`, `MessageUsage`, `MessageUsageTokens`, `MessageUsageCost`, `StreamCallback`, `ToolSchema`, `ToolCallRequest`, `MessageContentPart`, `ProviderCapabilities`, `ProviderType` — see [Providers](./providers.md#provider-roles)
 - `resolveThinkEffortLevel()`, `clampThinkEffort()` — the universal 0-1 `thinkEffort` → provider-native-level mapping helpers; see [Providers](./providers.md#think-effort)
 - Type: `ThinkEffortLevel`
-- `ImageDescriber`, type `ImageDescription` — used internally by `BaseAgent` to describe image tool output via the `image` provider role; exported for direct use if you want to describe an image outside the agent loop. `describe()` returns an `ImageDescription` (`{ description, inputTokens, outputTokens }`), not a bare string.
 - `toOpenAIFunctionTools()` — translates a `ToolSchema[]` into OpenAI's function-calling format; exported for building custom providers.
 
 ## Agents — see [Agents](./agents.md)
@@ -23,21 +22,21 @@ Flat index of every public export from `orbitx` (see `src/index.ts`), grouped by
 
 ## Tools — see [Tools](./tools.md)
 
-- `MCPTool`, `MCPCustomClass`, `generateRefId()`, `normalizeToolOutput()`
+- `MCPTool`, `generateRefId()`, `normalizeToolOutput()`
 - Types: `MCP`, `MCPToolOutput`
 
-Built-in tool factories, grouped by domain (each domain also exports a `*Tools()` array and a `*Interaction` EventEmitter):
+Built-in tool factories, grouped by domain (each domain also exports a `*Tools()` array):
 
 | Domain | Exports |
 |---|---|
-| Filesystem | `FsReadFileTool`, `FsWriteFileTool`, `FsEditFileTool`, `FsListDirTool`, `FsCreateDirTool`, `FsDeleteTool`, `FsMoveTool`, `FsStatTool`, `FsTools`, `FsInteraction`, type `FsEvent` |
-| Bash | `BashRunTool`, `BashWaitTool`, `BashLogsTool`, `BashListTool`, `BashWriteInputTool`, `BashTerminateTool`, `BashTools`, `BashInteraction`, type `BashEvent` |
-| Browser | `BrowserCreateSessionTool`, `BrowserRemoveSessionTool`, `BrowserGetSessionsTool`, `BrowserNavigateTool`, `BrowserConsoleTool`, `BrowserInjectTool`, `BrowserReadTool`, `BrowserClickTool`, `BrowserFillTool`, `BrowserScrollInfoTool`, `BrowserScrollTool`, `BrowserNetworkStatusTool`, `BrowserNetworkTool`, `BrowserSubmitFormTool`, `BrowserScreenshotTool`, `BrowserTools`, `BrowserInteraction`, type `BrowserEvent` |
-| Todo | `TodoCreateListTool`, `TodoRemoveListTool`, `TodoGetListsTool`, `TodoGetListTool`, `TodoCreateTaskTool`, `TodoRemoveTaskTool`, `TodoCheckTaskTool`, `TodoTools`, `TodoInteraction`, types `TodoTask`, `TodoList`, `TodoEvent` |
-| Present | `PresentAddTool`, `PresentClearTool`, `PresentGetListTool`, `PresentTools`, `getPresentFolder()`, `PresentInteraction`, type `PresentEvent` |
-| Question/Answer | `QuestionAnswerTool`, `QuestionAnswerTools`, `QuestionAnswerInteraction`, type `QuestionAnswerEvent` |
+| Filesystem | `FsReadFileTool`, `FsWriteFileTool`, `FsEditFileTool`, `FsListDirTool`, `FsCreateDirTool`, `FsDeleteTool`, `FsMoveTool`, `FsStatTool`, `FsTools` |
+| Bash | `BashRunTool`, `BashWaitTool`, `BashLogsTool`, `BashListTool`, `BashWriteInputTool`, `BashTerminateTool`, `BashTools` |
+| Browser | `BrowserCreateSessionTool`, `BrowserRemoveSessionTool`, `BrowserGetSessionsTool`, `BrowserNavigateTool`, `BrowserConsoleTool`, `BrowserInjectTool`, `BrowserReadTool`, `BrowserClickTool`, `BrowserFillTool`, `BrowserScrollInfoTool`, `BrowserScrollTool`, `BrowserNetworkStatusTool`, `BrowserNetworkTool`, `BrowserSubmitFormTool`, `BrowserScreenshotTool`, `BrowserTools` |
+| Todo | `TodoCreateListTool`, `TodoRemoveListTool`, `TodoGetListsTool`, `TodoGetListTool`, `TodoCreateTaskTool`, `TodoRemoveTaskTool`, `TodoCheckTaskTool`, `TodoTools`, types `TodoTask`, `TodoList` |
+| Present | `PresentAddTool`, `PresentClearTool`, `PresentGetListTool`, `PresentTools`, `getPresentFolder()` |
+| Question/Answer | `QuestionAnswerTool`, `QuestionAnswerTools` |
 | Utility | `GetCurrentTimeTool`, `DelayTool`, `ReadImageTool`, `UtilTools` |
-| Multi-agent *(experimental)* | `AgentTools(availableAgents, options?)`, `AgentListTool`, `AgentHireTool`, `AgentPromptTool`, `AgentReportTool`, `AgentRegistry`, `AgentInteraction`, type `AgentInfo`, type `AgentEvent`, type `AgentToolsOptions` — see [Agents](./agents.md#multi-agent-workeragent-experimental) |
+| Multi-agent *(experimental)* | `AgentTools(availableAgents, options?)`, `AgentListTool`, `AgentHireTool`, `AgentPromptTool`, `AgentReportTool`, `AgentRegistry`, type `AgentInfo`, type `AgentToolsOptions` — see [Agents](./agents.md#multi-agent-workeragent-experimental) |
 
 ## Skills — see [Skills](./skills.md)
 
@@ -51,4 +50,5 @@ Built-in tool factories, grouped by domain (each domain also exports a `*Tools()
 - `MCPComputer` *(experimental)*
 - `MCPStorage`, `MCPFSStorage`
 - `MCPRNG`
-- `MCPFilter`
+- `MCPOutputFilter` (`MCPFilter` is a deprecated alias, removed in `1.0.0`)
+- `MCPExecutionPolicy`, `MCPBypassExecutionPolicy`, type `MCPToolCallRequest`

package/docs/getting-started.md

@@ -49,7 +49,7 @@ Every provider implements the same `AIProvider` interface, so switching models i
 
 ## Recovering an agent's state
 
-`getCurrentAgentStates()` returns a plain JSON-serializable snapshot: message history, running memory summary, and token counters. Persist it anywhere; pass it back in as `initData` to resume the exact same agent later (after a process restart, in a new request handler, etc.):
+`getCurrentAgentStates()` returns a plain JSON-serializable snapshot: message history (each message carrying its own token usage) and running memory summary. Persist it anywhere; pass it back in as `initData` to resume the exact same agent later (after a process restart, in a new request handler, etc.):
 
 ```ts
 import { writeFileSync, existsSync, readFileSync } from "fs";
@@ -69,7 +69,7 @@ const agent = new SimpleAgent({
 writeFileSync("./state.json", JSON.stringify(agent.getCurrentAgentStates()));
 ```
 
-Messages are only ever appended, never rewritten. When the conversation grows past the memory threshold, OrbitX keeps the full history on disk (`messagesFull`) but resets the model's *working* context from a running summary, so an agent can keep going indefinitely without losing the ability to act on what happened earlier. Mechanics are covered in [Agents](./agents.md#memory-compaction).
+Messages are only ever appended, never rewritten. When the conversation grows past the memory threshold, OrbitX retires the working context into `retiredMessages` and resets the model's *active* context (`messagesCompact`) from a running summary (`compactMemory`), so an agent can keep going indefinitely without losing the ability to act on what happened earlier — nothing on disk is ever deleted, it just moves from one array to the other. Mechanics are covered in [Agents](./agents.md#memory-compaction).
 
 ## Next steps
 

package/docs/mcp-architecture.md

@@ -24,14 +24,19 @@ Registers tools and listens on its `connection` for incoming tool-call requests,
 ### `MCPClient`
 
 ```ts
-new MCPClient(envID: string, connection: MCPConnection | MCPConnection[], storage?: MCPStorage, rng?: MCPRNG, mcpFilter?: MCPFilter);
+new MCPClient(envID: string, connection: MCPConnection | MCPConnection[], storage?: MCPStorage, rng?: MCPRNG, mcpFilter?: MCPOutputFilter, executionPolicy?: MCPExecutionPolicy);
 mcpClient.getTools(): Promise<ToolSchema[]>;
-mcpClient.callTool(name: string, inputs: Record<string, any>): Promise<MCPToolOutput>;
+mcpClient.callTool(name: string, inputs: Record<string, any>, toolCallId?: string): Promise<MCPToolOutput>;
+mcpClient.setExecuteProviderHandler(handler: (toolCallId: string, type: ProviderType, input: Record<string, any>) => Promise<{ output: Record<string, any> }>): void;
 ```
 
+`toolCallId` on `callTool()` and the handler installed via `setExecuteProviderHandler()` are what `BaseAgent` wires up automatically so `mcp.executeProvider(...)` works inside a tool's `execute()` — see [Custom `MCP` subclasses](#custom-mcp-subclasses) below. You don't need to call either yourself unless you're driving `MCPClient` outside of `BaseAgent`.
+
 This is what you pass into `BaseAgent({ mcpClient, ... })`. `envID` scopes storage/RNG state per logical environment/session — pass a stable id (e.g. a user or conversation id) if you want isolated tool state per agent instance sharing a server. Accepting an array of connections lets one client fan out across multiple servers.
 
-`mcpFilter` (an `MCPFilter`) redacts sensitive substrings/patterns from tool output before it reaches the model — see [`MCPFilter`](#mcpfilter-redacting-tool-output) below.
+`mcpFilter` (an `MCPOutputFilter`) redacts sensitive substrings/patterns from tool output before it reaches the model — see [`MCPOutputFilter`](#mcpoutputfilter-redacting-tool-output) below.
+
+`executionPolicy` (an `MCPExecutionPolicy`) decides whether a tool call is allowed to run at all, *before* it's dispatched — see [`MCPExecutionPolicy`](#mcpexecutionpolicy-gating-tool-calls) below.
 
 #### Registering tools directly on the client (bypassing the connection)
 
@@ -89,15 +94,56 @@ Requires the `aliafsordeh/orbitx-sandbox:0.1` image (pull it, or build it yourse
 - **`MCPStorage`** — abstract `{ get(key): Promise<string>; set(key, value): Promise<void> }`. `MCPFSStorage` is the default filesystem-backed implementation (`new MCPFSStorage(path?)`, defaults to a randomly-named folder under the OS temp directory). `MCPStorage` and `MCPFSStorage` are both exported if you want to implement your own backend (Redis, a database, etc.) or point the default one at a specific path.
 - **`MCPRNG`** — deterministic-ish id generator backed by an `MCPStorage` instance (used internally for things like `generateRefId()`, the helper behind ref ids in `BrowserReadTool`'s output).
 
-## `MCPFilter` — redacting tool output
+## `MCPOutputFilter` — redacting tool output
 
 ```ts
-new MCPFilter(values: (string | RegExp)[]);
+new MCPOutputFilter(values: (string | RegExp)[]);
 mcpFilter.filter(input: any): any;   // replaces every match with "FILTERED_OUTPUT"
 ```
 
 Pass literal strings (e.g. an API key you never want echoed back to the model) or regexes (e.g. `/sk-[a-zA-Z0-9]{20,}/`) as the `mcpFilter` argument to `MCPClient` to scrub tool output before it's returned — useful when a tool might read a file or environment variable containing a secret.
 
+> `MCPOutputFilter` was named `MCPFilter` before. The old name is still exported as a deprecated alias (`export const MCPFilter = MCPOutputFilter`) and will be removed in `1.0.0` — switch to `MCPOutputFilter` in new code.
+
+## `MCPExecutionPolicy` — gating tool calls
+
+```ts
+abstract class MCPExecutionPolicy {
+    abstract authorize(request: MCPToolCallRequest): Promise<boolean> | boolean;
+}
+// MCPToolCallRequest = { toolName: string; inputs: Record<string, any>; envID: string; toolCallId?: string };
+```
+
+Checked by `MCPClient.callTool()` before every tool call is dispatched — whether the tool is registered locally on the client or routed across a connection to a remote `MCPServer`. Returning (or resolving to) `false` throws before the call reaches the tool, and that error surfaces to the model as a normal tool-error result (`BaseAgent`'s dispatch loop catches it the same way it catches any other `callTool()` rejection).
+
+`MCPClient` defaults to `MCPBypassExecutionPolicy`, a built-in no-op that authorizes every call — so this is entirely opt-in. Implement your own subclass of `MCPExecutionPolicy` to add an allowlist, rate limiting, per-tool confirmation, or any other trust policy, and pass it as the `executionPolicy` argument to `MCPClient`'s constructor:
+
+```ts
+import { MCPClient, MCPExecutionPolicy, type MCPToolCallRequest } from "orbitx";
+
+class AllowlistPolicy extends MCPExecutionPolicy {
+    #allowed: Set<string>;
+    constructor(allowed: string[]) {
+        super();
+        this.#allowed = new Set(allowed);
+    }
+    authorize({ toolName }: MCPToolCallRequest) {
+        return this.#allowed.has(toolName);
+    }
+}
+
+const mcpClient = new MCPClient(envID, connection, storage, rng, undefined, new AllowlistPolicy(["get_current_time"]));
+```
+
 ## Custom `MCP` subclasses
 
-Both `MCPServer` and `MCPClient` extend the abstract `MCP` class (`getStorage()`, `getRNG()`). Tool `execute` functions receive the calling `MCP` instance as their third argument, so a custom `MCPCustomClass`-based tool (see [Tools](./tools.md#the-mcptool-shape)) can reach storage/RNG scoped to whichever client dispatched the call.
+Both `MCPServer` and `MCPClient` extend the abstract `MCP` class — `getStorage()`, `getRNG()`, and `executeProvider(toolCallId, type, input)`. Tool `execute` functions receive the calling `MCP` instance directly, as their 4th argument (after a `toolCallId` — see [Tools](./tools.md#the-mcptool-shape)), so a tool can reach storage/RNG scoped to whichever client dispatched the call.
+
+`executeProvider` is where the two subclasses genuinely differ:
+
+- **`MCPClient`** answers it in-process, by delegating to a handler `BaseAgent` installs on the client via `setExecuteProviderHandler(...)` at construction time. `MCPClient` itself never imports anything from `ai-provider.ts` — it just holds an opaque callback.
+- **`MCPServer`** has no such handler — it never touches a provider directly. It proxies the request over its `MCPConnection` (an `executeProvider`/`executeProviderResponse` message pair, mirroring `toolCall`/`toolCallCallback` but initiated by the server instead of the client) to whichever `MCPClient` is attached, which runs the same installed handler and answers back.
+
+So a **server-registered** tool — the kind meant to run inside `MCPComputer`'s sandbox, or in a genuinely separate process over IPC/WS — can still call `mcp.executeProvider(...)` and get a real result, without a provider instance (or the credentials it holds) ever crossing into that process. Only a scoped request/response for one call does. This is deliberate: filesystem/browser-touching tools like `ReadImageTool`/`BrowserScreenshotTool` are exactly the kind this doc already recommends sandboxing (see above), and they both need a provider to do their job.
+
+That accessor also carries provider access (`agent.getProvider(type)`/`getProviders(type)`) — but **only for a tool dispatched by `MCPClient.callTool`'s local branch** (a tool registered directly on the client, per the section above). `MCPServer`'s dispatch (both the `toolCall` handler above and, transitively, anything a remote/sandboxed tool triggers) builds an accessor with no provider registry behind it at all, so `getProvider`/`getProviders` there always resolve `undefined`. This is deliberate, not a gap to work around: provider instances hold API keys, and `MCPServer` is precisely the thing that runs inside a separate process or an `MCPComputer` sandbox — credentials never need to, and never do, cross that boundary. A tool that needs to call a provider directly (e.g. to describe an image it just read) has to be registered on the `MCPClient`, in the same process as the agent that owns those providers.

package/docs/providers.md

@@ -12,10 +12,12 @@ abstract class AIProvider {
   ): Promise<ChatResponse>;
 
   abstract getCapabilities(): ProviderCapabilities;
+
+  abstract setOption(key: string, value: unknown): void;
 }
 ```
 
-- `chat` sends the full message history (plus a provider-agnostic `tools` schema, when supplied) and returns a `ChatResponse` (`{ content, inputTokens, outputTokens, toolCalls? }`). If `streamCallback` is passed, text is streamed to it incrementally as it arrives — see [Streaming](./streaming.md).
+- `chat` sends the full message history (plus a provider-agnostic `tools` schema, when supplied) and returns a `ChatResponse` (`{ content, inputMissTokens, inputCacheTokens, outputTokens, toolCalls? }`) — `inputMissTokens`/`inputCacheTokens` are real, provider-reported non-cached/cache-hit input token counts (see "User tracking & prompt caching" below), not a client-side estimate. If `streamCallback` is passed, text is streamed to it incrementally as it arrives — see [Streaming](./streaming.md).
 - `signal`, when passed and later aborted, cancels the in-flight request to the provider's API immediately rather than just abandoning it client-side — `BaseAgent.immediateStop()` uses this so the call stops generating (and billing for) further output right away. It doesn't retroactively waive input tokens or output already generated before the abort. Anthropic, OpenAI, and DeepSeek honor it for both streaming and non-streaming calls (their SDKs forward it straight to `fetch`). Ollama only honors it for streaming calls — its client has no way to abort a non-streaming request — so a `chat()` call made without a `streamCallback` against `OllamaProvider` can't be cancelled mid-flight; see the retry-loop guard in `ollama-provider.ts`.
   - **Streaming calls resolve normally on abort**, returning a `ChatResponse` built from whatever content/thinking/tool-calls were accumulated before the cutoff, instead of throwing — so text already delivered via `streamCallback` still ends up recorded rather than discarded. A tool call still being generated when the abort lands is dropped (its arguments may be truncated JSON); only tool calls that had already fully finished streaming are included in `toolCalls`.
   - **Non-streaming calls still throw on abort** — there's no partial response to salvage since the API only returns once, in full — and `BaseAgent` discards that turn.
@@ -23,15 +25,48 @@ abstract class AIProvider {
 
 ```ts
 interface ProviderCapabilities {
-  supportsTools: boolean;     // must be true — BaseAgent requires native tool-calling
-  supportsImages: boolean;    // whether this provider/model accepts image content
-  contextWindow: number;      // total context window, in tokens
-  safeUsageRatio?: number;    // fraction of contextWindow safe to fill before compacting (default 0.5)
+  supportsTools: boolean;      // must be true — BaseAgent requires native tool-calling
+  supportsImages: boolean;     // whether this provider/model accepts image content
+  contextWindow: number;       // total context window, in tokens
+  safeUsageRatio?: number;     // fraction of contextWindow safe to fill before compacting (default 0.5)
+  supportsThinking?: boolean;  // whether reasoning/thinking text is surfaced — see "Think effort" below
+  maxOutputTokens?: number;    // the model's max output tokens per response, where confirmed — not yet enforced anywhere, reserved for a future release
+  supportsVideo?: boolean;             // whether this provider/model accepts video content
+  supportsAudio?: boolean;             // whether this provider/model accepts audio content
+  supportsImageGeneration?: boolean;   // whether this provider/model can generate new images
+  supportsImageEditing?: boolean;      // whether this provider/model can edit an existing image
+  supportsVideoGeneration?: boolean;   // whether this provider/model can generate new video
+  supportsVideoEditing?: boolean;      // whether this provider/model can edit existing video
+  supportsAudioGeneration?: boolean;   // whether this provider/model can generate new audio (speech, music, sound effects)
+  supportsAudioDesign?: boolean;       // whether this provider/model can design/produce sound effects or soundscapes
+  supportsAudioClone?: boolean;        // whether this provider/model can clone a voice from a reference sample
+  supports3DModelGeneration?: boolean; // whether this provider/model can generate a new 3D model
 }
 ```
 
+These flags describe what the model itself can natively do — they're unrelated to `ProviderType` (see "Provider roles" below), which is just a role name used to look a provider up in a `ProviderRegistry`. A provider registered under the `"image-generation"` role doesn't need `supportsImageGeneration: true` (or vice versa) — the role name and the capability flag are independent. All four built-in providers are chat/vision LLMs, not generation APIs, so every one of the new flags above is always `false` on them.
+
 `BaseAgent.run()` throws immediately if `getCapabilities().supportsTools` is `false` — the legacy JSON-in-text tool-call convention has been removed from the main loop. All providers default `supportsTools` to `true` except Ollama, which defaults it to `false` (many locally-hosted models don't support native function calling; pass `{ supportsTools: true }` explicitly if your model does).
 
+## Changing an option without reconstructing
+
+`setOption(key, value)` mutates an already-constructed provider in place — e.g. `provider.setOption("thinkEffort", 0.8)` — instead of building a new one. Every built-in provider only allows keys it already reads fresh on every `chat()` call (never something baked into derived state at construction, like `contextWindow`); an unrecognized key throws.
+
+| Provider | Settable keys |
+|---|---|
+| Anthropic | `thinkEffort`, `userId`, `disablePromptCaching`, `maxTokens`, `supportsTools` |
+| OpenAI | `thinkEffort`, `userId`, `supportsTools` |
+| DeepSeek | `thinkEffort`, `userId`, `supportsTools` |
+| Ollama | `thinkEffort`, `supportsTools` |
+
+## Usage on the streaming path
+
+OpenAI and DeepSeek's Chat-Completions-style streaming protocol doesn't compute token usage until a turn is fully complete — it arrives once, in a special final chunk (`stream_options: { include_usage: true }`), alongside `done: true`. Every chunk before that has `usage: { inputMissTokens: 0, inputCacheTokens: 0, outputTokens: 0 }` — not because those numbers are unknown-but-estimated, but because the API genuinely hasn't computed them yet. `OpenAIProvider` and `DeepSeekProvider` both trust this real, provider-reported final chunk directly — no local estimation involved on either. DeepSeek's final chunk includes its own `prompt_cache_hit_tokens`/`prompt_cache_miss_tokens` fields (falling back to `prompt_tokens_details.cached_tokens` if those aren't present), parsed the same way real cache data is read from any other provider.
+
+If a call is cut off before that final chunk arrives (e.g. `BaseAgent.immediateStop()` aborting the connection to stop paying for further output), the real usage number simply never gets computed or sent for either provider — not partially, not late, just never — so `inputMissTokens`/`inputCacheTokens`/`outputTokens` stay `0` on `ChatResponse` too in that case. Anthropic doesn't have this problem (it streams real usage incrementally, see [Streaming](./streaming.md#usage-chunks)); Ollama has the same final-chunk-only limitation but isn't covered by what's described here.
+
+This is the only usage source DeepSeek's streaming path has, so it's treated as the real number, not flagged as a lesser "estimate" — whatever lands on the final `done: true` chunk (and the returned `ChatResponse`) is what a consumer should use for billing/accounting, the same as it would for any other provider's real, API-reported usage. The two sides of that number are revealed at different points, though: `outputTokens` grows on every streamed chunk as content arrives, since it's cheap to recompute from what's been generated so far; `inputMissTokens`/`inputCacheTokens` are only computed once, on that final chunk, since the input side never changes mid-call and there's no reason to resend the same number on every content delta.
+
 ## Think effort
 
 Reasoning/thinking models each expose a different native "how hard should the model think" knob — OpenAI and Ollama use `low`/`medium`/`high`, DeepSeek uses `none`/`low`/`high`/`max`, Anthropic uses a numeric thinking-token budget. OrbitX replaces all of that with one universal option on every provider's constructor:
@@ -60,6 +95,19 @@ Per-provider specifics:
 
 See [Streaming](./streaming.md#thinking-chunks) for how (and whether) a given provider's reasoning text is observable while it streams.
 
+## User tracking & prompt caching
+
+Every provider's constructor accepts `userId?: string` — an opaque end-user identifier forwarded to the underlying API for that provider's own abuse-monitoring/analytics, same universal-option pattern as `thinkEffort`: pass it uniformly and providers that have no equivalent field just ignore it.
+
+| Provider | Where `userId` goes | Notes |
+|---|---|---|
+| Anthropic | `metadata.user_id` on every request | Anthropic's documented field for this purpose. |
+| OpenAI | `user_id` on every request | Sent as an extra body field alongside the standard params. |
+| DeepSeek | `user_id` on every request | Same mechanism as OpenAI (DeepSeek's API is OpenAI-compatible). |
+| Ollama | Ignored | Accepted in the options type for interface parity only — Ollama's API has no equivalent field. |
+
+**Anthropic prompt caching is on by default.** Every request marks the last content block with an ephemeral `cache_control` breakpoint, so a growing conversation gets served from cache instead of being fully reprocessed each turn — pass `disablePromptCaching: true` to send every request uncached. DeepSeek's cache is automatic server-side (no `cache_control` equivalent to toggle) and reports its own explicit `prompt_cache_hit_tokens`/`prompt_cache_miss_tokens`. OpenAI's cache is also automatic, reporting hits via `prompt_tokens_details.cached_tokens`. Ollama has no cache concept at all — `inputCacheTokens` is always `0`.
+
 ## Built-in providers
 
 ### Ollama
@@ -71,6 +119,7 @@ new OllamaProvider(model: string, host = "http://localhost:11434", options?: {
   supportsTools?: boolean;   // default false
   contextWindow?: number;    // default 32_000
   thinkEffort?: number;      // 0-1, see "Think effort" above
+  userId?: string;           // accepted for interface parity only — ignored, Ollama has no equivalent field
 });
 ```
 
@@ -86,12 +135,13 @@ new DeepSeekProvider(apiKey: string, model = "deepseek-v4-flash", options?: {
   supportsImages?: boolean;  // default: true only for deepseek-v4-flash-vision-exp
   contextWindow?: number;    // default 1_000_000 (deepseek-v4-flash / deepseek-v4-flash-vision-exp / deepseek-v4-pro)
   thinkEffort?: number;      // 0-1, see "Think effort" above
+  userId?: string;           // see "User tracking & prompt caching" above
 });
 ```
 
 Uses the `openai` SDK pointed at `https://api.deepseek.com` (DeepSeek's API is OpenAI-compatible). `getCapabilities().supportsImages` is `false` for every model except `deepseek-v4-flash-vision-exp`, which is `deepseek-v4-flash` plus native image input (same context window, same reasoning support) — pass `supportsImages: true` explicitly if a future model adds vision support before this SDK's allowlist is updated.
 
-Token usage (`inputTokens`/`outputTokens`) is only accurate on the non-streaming path (`chat()` called without `streamCallback`), which reads `response.usage` directly. On the streaming path this provider deliberately does not set `stream_options: { include_usage: true }`, so DeepSeek never sends a usage-bearing chunk and both counts report as `0`. This matters most for `ImageDescriber`, which always calls `chat()` without a `streamCallback` (see [Token accounting](./agents.md#token-accounting)) — so image-description usage is captured correctly even though live agent streaming isn't.
+`stream_options: { include_usage: true }` is set on every streaming call, so `inputMissTokens`/`inputCacheTokens`/`outputTokens` are accurate on both the streaming and non-streaming paths — DeepSeek sends a usage-bearing final chunk (including its own `prompt_cache_hit_tokens`/`prompt_cache_miss_tokens` split) the same way OpenAI does.
 
 ### Anthropic
 
@@ -101,9 +151,12 @@ import { AnthropicProvider } from "orbitx";
 new AnthropicProvider(apiKey: string, model = "claude-sonnet-5", options?: {
   supportsTools?: boolean;    // default true
   supportsImages?: boolean;   // default true
-  contextWindow?: number;     // default from a per-model table, else 200_000
+  contextWindow?: number;     // default from a per-model table, else 200_000; an explicit value here always wins over extendedContext below
   maxTokens?: number;         // output token cap, default 4096
   thinkEffort?: number;       // 0-1, see "Think effort" above
+  userId?: string;            // see "User tracking & prompt caching" above
+  disablePromptCaching?: boolean;  // default false — prompt caching is on by default, see above
+  extendedContext?: boolean | string;  // default false/undefined — see "Context window" below
 });
 ```
 
@@ -120,27 +173,47 @@ new OpenAIProvider(apiKey: string, model = "gpt-5", options?: {
   contextWindow?: number;     // default from a per-model table, else 128_000
   baseURL?: string;           // point at an OpenAI-compatible endpoint
   thinkEffort?: number;       // 0-1, see "Think effort" above — ignored on non-reasoning models
+  userId?: string;            // see "User tracking & prompt caching" above
 });
 ```
 
 Uses the Chat Completions API (not the Responses API). `baseURL` makes this usable against any OpenAI-compatible provider, not just OpenAI itself.
 
-## Two provider roles: main + image
+## Provider roles
+
+An agent's providers are registered under open-ended role names, not a fixed pair. `"main"` — the provider driving the agent loop itself — is the only one that's required; every other role is optional and looked up on demand by whatever tool needs it.
 
-An agent has exactly two provider "roles":
+```ts
+export type ProviderType =
+  | "main"
+  | "image-describer"
+  | "image-generation"
+  | "image-editing"
+  | "video-describer"
+  | "video-generation"
+  | "video-editing"
+  | "audio-describer"
+  | "audio-generation"
+  | "audio-design"
+  | "audio-clone"
+  | "3d-model-generator"
+  | "llm-low"
+  | "llm-medium"
+  | "llm-high"
+  | (string & {});
+```
 
-- **`main`** — drives the agent loop itself (required).
-- **`image`** — describes image tool output (e.g. a screenshot from `BrowserScreenshotTool`) into text, so the main conversation doesn't have to carry raw image bytes. Optional.
+`ProviderType` (exported from `orbitx`) is the documented starting vocabulary for every role name — not a closed union, and purely for naming/registry lookup. It has no relationship to `ProviderCapabilities` (see above) — a role name doesn't imply anything about what the provider registered under it can actually do. The trailing `(string & {})` (not bare `string`) is what makes the open-endedness work: a plain `| string` union collapses to just `string` and an editor stops suggesting the literals at all, while intersecting with `{}` keeps TS from widening it away, so a custom name still type-checks the moment nothing here fits. `llm-low`/`medium`/`high` are a self-declared overall quality/durability tier (cheap-and-good-enough vs. worth relying on for longer/harder tasks) — not a specific claim about reasoning power.
 
-Pass a single provider to use it for both roles (if it supports images, it's also used as the image describer; otherwise images are dropped with an explanatory error message back to the model). Pass an array to split them:
+Pass a single provider to use it as `main` (same as today); pass an array of `{ type, provider, default? }` entries to register more:
 
 ```ts
 import { BaseAgent, DeepSeekProvider, OllamaProvider } from "orbitx";
 
 const agent = new BaseAgent({
   aiProvider: [
-    { type: "main", aiProvider: new DeepSeekProvider("api-key", "deepseek-v4-flash") },
-    { type: "image", aiProvider: new OllamaProvider("llava:latest") },
+    { type: "main", provider: new DeepSeekProvider("api-key", "deepseek-v4-flash") },
+    { type: "image-describer", provider: new OllamaProvider("llava:latest") },
   ],
   instruction: "You are a helpful assistant.",
   mcpClient,
@@ -148,8 +221,34 @@ const agent = new BaseAgent({
 });
 ```
 
-This resolution is done by `resolveAgentProviders()` (`AgentProvidersInput = AIProvider | AgentProviderEntry[]`), exported from `orbitx` if you need it directly. There is no third "utils" role — every tool/skill is included in the system prompt from the start, so there's nothing left for a separate pass to decide.
+`provider` can also be an array — a pool of candidates for that role — with `default` (0-based, defaults to `0`) picking which one is used when code needs exactly one. The rest of the pool is there to be enumerated later; nothing built-in does smarter selection than "the default" yet.
+
+### Resolving a provider by role
+
+`ProviderRegistry.getProvider(type)` tries an exact role-name match first (that role's pool default). If nothing is registered under `type`, it falls back to scanning every provider registered under *any* role for the first one whose `getCapabilities()` reports the matching capability flag — not a generic capability bag keyed by `ProviderType` (that's gone; `ProviderType` is purely a registry role name and has no relationship to what a model can do), but a fixed, small mapping from a handful of well-known role names to their one corresponding `ProviderCapabilities` flag:
+
+| Role (`ProviderType`) | Capability flag checked in the fallback |
+|---|---|
+| `"image-describer"` | `supportsImages` |
+| `"image-generation"` | `supportsImageGeneration` |
+| `"image-editing"` | `supportsImageEditing` |
+| `"video-describer"` | `supportsVideo` |
+| `"video-generation"` | `supportsVideoGeneration` |
+| `"video-editing"` | `supportsVideoEditing` |
+| `"audio-describer"` | `supportsAudio` |
+| `"audio-generation"` | `supportsAudioGeneration` |
+| `"audio-design"` | `supportsAudioDesign` |
+| `"audio-clone"` | `supportsAudioClone` |
+| `"3d-model-generator"` | `supports3DModelGeneration` |
+
+This is why a single vision-capable `main` provider "just works" as an image describer with zero extra config: nothing needs to be registered under `"image-describer"` explicitly — the fallback finds `main` because its `getCapabilities().supportsImages` is `true`. Role names outside this table (`"main"`, `"llm-low"`/`"medium"`/`"high"`, or any custom string) have no capability equivalent, so the fallback never applies to them — an unregistered role like that returns `undefined` from `getProvider` regardless of what any registered provider can do.
+
+`BaseAgent` is the only thing that ever calls `getProvider` directly — a tool never touches a `ProviderRegistry`, or a raw provider, itself. It reaches this indirectly via `mcp.executeProvider(toolCallId, type, input)` (see [Tools](./tools.md#mcpexecuteprovider--calling-a-provider-from-inside-a-tool)), which works the same way whether the tool is registered on the local `MCPClient` or a sandboxed/remote `MCPServer` — the request is always resolved on `BaseAgent`'s side, never handed across as a live object.
+
+`resolveAgentProviders()` (`AgentProvidersInput = AIProvider | AgentProviderEntry[]`) builds the `ProviderRegistry` this all runs on — both are exported from `orbitx` if you need them directly (`getMain()`, `getProvider(type)`, `getProviders(type)` for the full pool under an exact role). There is no separate "utils" role — every tool/skill is included in the system prompt from the start, so there's nothing left for a separate pass to decide.
 
 ## Writing a custom provider
 
 Extend `AIProvider` and implement `chat`/`getCapabilities`. Follow an existing provider (e.g. `src/core/ollama-provider.ts`) as a template: translate `Message[]` to your API's wire format, stream text chunks to `streamCallback` as they arrive, accumulate tool-call deltas and emit them on the final chunk, and translate `ToolSchema[]` into your API's function-calling format (see `src/core/tool-schema-translator.ts` for existing OpenAI/Anthropic translators you can reuse or reference).
+
+`inputMissTokens`/`inputCacheTokens`/`outputTokens` on the returned `ChatResponse` are optional — every built-in provider is token-billed and always sets them, but a provider registered under a per-request-billed role (e.g. `image-generation`, `audio-clone`) should omit them and set `cost` instead. `BaseAgent#executeProvider` (see [Tools](./tools.md#mcpexecuteprovider--calling-a-provider-from-inside-a-tool)) checks `cost` first and records a `unit: "cost"` usage entry when it's present, falling back to the token fields (defaulting any missing one to `0`) otherwise — see [Token accounting](./agents.md#token-accounting).

package/docs/streaming.md

@@ -11,12 +11,14 @@ type StreamCallback = (chunk: {
   toolCallId?: string;            // on "tool"-role chunks
   toolName?: string;              // on "tool"-role chunks
   thinking?: string;               // reasoning-text delta, mirroring `content` — see "Thinking chunks" below
+  usage?: { inputMissTokens: number; inputCacheTokens: number; outputTokens: number };  // see "Usage chunks" below
+  hidden?: boolean;                // true on messages BaseAgent injected itself
 }) => Promise<void> | void;
 ```
 
 ## What you actually receive, turn by turn
 
-1. **`role: "user"`** — one chunk, `done: true`, echoing the prompt that was just added to history (either the one you passed to `run()`, or a queued one — see [Agents](./agents.md#the-run-loop)).
+1. **`role: "user"`** — one chunk, `done: true`, echoing the prompt that was just added to history (either the one you passed to `run()`, or a queued one — see [Agents](./agents.md#the-run-loop)). One exception: the forced-compaction nudge (see [Agents](./agents.md#memory-compaction)) is also a `role: "user"` chunk, but carries `hidden: true` since it's `BaseAgent` talking to the model, not the actual user — check that flag before showing a `"user"` chunk to an end user.
 2. **`role: "assistant"`** — the model's reply, streamed incrementally: zero or more `done: false` chunks each carrying the next slice of `content` as it arrives from the provider, followed by one final `done: true` chunk. `content` is delta text, not the accumulated total — append it yourself if you need the running message.
    - If the model requested tool calls this turn, each one is emitted individually, mid-stream, as soon as *that* call finishes generating — `toolCalls: [thatOneCall]` on a `done: false` chunk. See "Incremental tool-call chunks" below.
 3. **`role: "tool"`** — one chunk per dispatched tool call, `done: true`, with `content` set to the tool's result text and `toolCallId`/`toolName` identifying which call it answers. Emitted after the tool has actually finished executing (there is no streaming/progress signal for tool execution itself through this callback — see below for how to get that separately).
@@ -49,6 +51,21 @@ Detection differs by provider, since not every API has an explicit "this call is
 | OpenAI / DeepSeek | Inferred: the API streams one call's arguments at a time with no end marker, so a call is treated as finished the instant the *next* call's first delta (carrying a fresh `id`) arrives, or the stream ends (for the last/only call). |
 | Ollama | Ollama sends each tool call already fully formed (not built up token by token), so it's emitted the moment it's seen — deduplicated in case the same call reappears in a later chunk. |
 
+## Usage chunks
+
+Every `assistant`-role chunk carries `usage` — not just the ones with a fresh number to report. Where a value isn't known yet, it's an explicit `0`, never an absent/undefined field, so consuming code can always read `chunk.usage.outputTokens` etc. without separately guarding for `usage` or its fields being missing. This is the same `streamCallback` passed into `run()`, forwarded straight through to the provider, so external code (e.g. something watching an external credit/cost budget, deciding when to call `immediateStop()`) sees it too — no separate channel.
+
+Every number here is real, straight from whatever the provider reports — nothing in this SDK estimates or guesses a token count. What differs is *when* each field stops being a placeholder `0` and starts reflecting the API's real number:
+
+| Provider | Mid-stream `usage` | Final `usage` (on `done: true`) |
+|---|---|---|
+| Anthropic | `inputMissTokens`/`inputCacheTokens` are real from the very first chunk (before any content) — that's simply when its protocol first has them, not something computed early on purpose. `outputTokens` updates on every delta. | Same numbers, just the last update. |
+| OpenAI | All three fields are `0` on every chunk before the last one (`include_usage` is honored correctly, but only delivers on completion — there's nothing to report early). | Real, billed numbers for all three fields. If the call is cut off before this chunk arrives, `inputMissTokens`/`inputCacheTokens`/`outputTokens` come back `0` on `ChatResponse` too — the API never computed them, so there's nothing to report. |
+| DeepSeek | Same as OpenAI — `0` on every chunk before the last one. | Real, billed numbers, including its own `prompt_cache_hit_tokens`/`prompt_cache_miss_tokens` for the cache split. Same `0`-on-abort behavior as OpenAI if the call is cut off first. |
+| Ollama | All three fields are `0` on every chunk before the last one — no live signal exists for this provider at all. | Real. |
+
+Practical implication: for Anthropic and DeepSeek you get a live, reactable signal before a turn finishes (immediately for Anthropic's input side, throughout for either provider's output side); for OpenAI there's no live signal at all today — you only find out after the turn finishes, or get a hard `0` if it was cut off first.
+
 ## Minimal consumer
 
 ```ts
@@ -110,7 +127,7 @@ Once a turn finishes, its accumulated reasoning is available two ways on both `C
 
 ## What's *not* in the stream today
 
-- **No live per-tool progress channel through `StreamCallback`.** For domains that emit their own progress (browser navigation, long-running bash processes), subscribe to that domain's `*Interaction` `EventEmitter` directly instead (e.g. `BrowserInteraction`, `BashInteraction`) — see [Tools](./tools.md).
+- **No live per-tool progress channel through `StreamCallback`.** There's no built-in way to observe a long-running tool call (browser navigation, a bash process) mid-flight — `execute()` runs to completion and its return value is the only thing surfaced.
 - **Tool calls still only appear once, fully assembled** — see above. `thinkEffort`/thinking chunks don't change that.
 
 If your use case depends on either of these, check the project's issue tracker / recent changes before assuming they're unavailable — this area of the SDK is actively evolving.