orbitx 0.2.6 → 0.2.7

package/docs/providers.md

@@ -12,10 +12,12 @@ abstract class AIProvider {
   ): Promise<ChatResponse>;
 
   abstract getCapabilities(): ProviderCapabilities;
+
+  abstract setOption(key: string, value: unknown): void;
 }
 ```
 
-- `chat` sends the full message history (plus a provider-agnostic `tools` schema, when supplied) and returns a `ChatResponse` (`{ content, inputTokens, outputTokens, toolCalls? }`). If `streamCallback` is passed, text is streamed to it incrementally as it arrives — see [Streaming](./streaming.md).
+- `chat` sends the full message history (plus a provider-agnostic `tools` schema, when supplied) and returns a `ChatResponse` (`{ content, inputMissTokens, inputCacheTokens, outputTokens, toolCalls? }`) — `inputMissTokens`/`inputCacheTokens` are real, provider-reported non-cached/cache-hit input token counts (see "User tracking & prompt caching" below), not a client-side estimate. If `streamCallback` is passed, text is streamed to it incrementally as it arrives — see [Streaming](./streaming.md).
 - `signal`, when passed and later aborted, cancels the in-flight request to the provider's API immediately rather than just abandoning it client-side — `BaseAgent.immediateStop()` uses this so the call stops generating (and billing for) further output right away. It doesn't retroactively waive input tokens or output already generated before the abort. Anthropic, OpenAI, and DeepSeek honor it for both streaming and non-streaming calls (their SDKs forward it straight to `fetch`). Ollama only honors it for streaming calls — its client has no way to abort a non-streaming request — so a `chat()` call made without a `streamCallback` against `OllamaProvider` can't be cancelled mid-flight; see the retry-loop guard in `ollama-provider.ts`.
   - **Streaming calls resolve normally on abort**, returning a `ChatResponse` built from whatever content/thinking/tool-calls were accumulated before the cutoff, instead of throwing — so text already delivered via `streamCallback` still ends up recorded rather than discarded. A tool call still being generated when the abort lands is dropped (its arguments may be truncated JSON); only tool calls that had already fully finished streaming are included in `toolCalls`.
   - **Non-streaming calls still throw on abort** — there's no partial response to salvage since the API only returns once, in full — and `BaseAgent` discards that turn.
@@ -23,15 +25,48 @@ abstract class AIProvider {
 
 ```ts
 interface ProviderCapabilities {
-  supportsTools: boolean;     // must be true — BaseAgent requires native tool-calling
-  supportsImages: boolean;    // whether this provider/model accepts image content
-  contextWindow: number;      // total context window, in tokens
-  safeUsageRatio?: number;    // fraction of contextWindow safe to fill before compacting (default 0.5)
+  supportsTools: boolean;      // must be true — BaseAgent requires native tool-calling
+  supportsImages: boolean;     // whether this provider/model accepts image content
+  contextWindow: number;       // total context window, in tokens
+  safeUsageRatio?: number;     // fraction of contextWindow safe to fill before compacting (default 0.5)
+  supportsThinking?: boolean;  // whether reasoning/thinking text is surfaced — see "Think effort" below
+  maxOutputTokens?: number;    // the model's max output tokens per response, where confirmed — not yet enforced anywhere, reserved for a future release
+  supportsVideo?: boolean;             // whether this provider/model accepts video content
+  supportsAudio?: boolean;             // whether this provider/model accepts audio content
+  supportsImageGeneration?: boolean;   // whether this provider/model can generate new images
+  supportsImageEditing?: boolean;      // whether this provider/model can edit an existing image
+  supportsVideoGeneration?: boolean;   // whether this provider/model can generate new video
+  supportsVideoEditing?: boolean;      // whether this provider/model can edit existing video
+  supportsAudioGeneration?: boolean;   // whether this provider/model can generate new audio (speech, music, sound effects)
+  supportsAudioDesign?: boolean;       // whether this provider/model can design/produce sound effects or soundscapes
+  supportsAudioClone?: boolean;        // whether this provider/model can clone a voice from a reference sample
+  supports3DModelGeneration?: boolean; // whether this provider/model can generate a new 3D model
 }
 ```
 
+These flags describe what the model itself can natively do — they're unrelated to `ProviderType` (see "Provider roles" below), which is just a role name used to look a provider up in a `ProviderRegistry`. A provider registered under the `"image-generation"` role doesn't need `supportsImageGeneration: true` (or vice versa) — the role name and the capability flag are independent. All four built-in providers are chat/vision LLMs, not generation APIs, so every one of the new flags above is always `false` on them.
+
 `BaseAgent.run()` throws immediately if `getCapabilities().supportsTools` is `false` — the legacy JSON-in-text tool-call convention has been removed from the main loop. All providers default `supportsTools` to `true` except Ollama, which defaults it to `false` (many locally-hosted models don't support native function calling; pass `{ supportsTools: true }` explicitly if your model does).
 
+## Changing an option without reconstructing
+
+`setOption(key, value)` mutates an already-constructed provider in place — e.g. `provider.setOption("thinkEffort", 0.8)` — instead of building a new one. Every built-in provider only allows keys it already reads fresh on every `chat()` call (never something baked into derived state at construction, like `contextWindow`); an unrecognized key throws.
+
+| Provider | Settable keys |
+|---|---|
+| Anthropic | `thinkEffort`, `userId`, `disablePromptCaching`, `maxTokens`, `supportsTools` |
+| OpenAI | `thinkEffort`, `userId`, `supportsTools` |
+| DeepSeek | `thinkEffort`, `userId`, `supportsTools` |
+| Ollama | `thinkEffort`, `supportsTools` |
+
+## Usage on the streaming path
+
+OpenAI and DeepSeek's Chat-Completions-style streaming protocol doesn't compute token usage until a turn is fully complete — it arrives once, in a special final chunk (`stream_options: { include_usage: true }`), alongside `done: true`. Every chunk before that has `usage: { inputMissTokens: 0, inputCacheTokens: 0, outputTokens: 0 }` — not because those numbers are unknown-but-estimated, but because the API genuinely hasn't computed them yet. `OpenAIProvider` and `DeepSeekProvider` both trust this real, provider-reported final chunk directly — no local estimation involved on either. DeepSeek's final chunk includes its own `prompt_cache_hit_tokens`/`prompt_cache_miss_tokens` fields (falling back to `prompt_tokens_details.cached_tokens` if those aren't present), parsed the same way real cache data is read from any other provider.
+
+If a call is cut off before that final chunk arrives (e.g. `BaseAgent.immediateStop()` aborting the connection to stop paying for further output), the real usage number simply never gets computed or sent for either provider — not partially, not late, just never — so `inputMissTokens`/`inputCacheTokens`/`outputTokens` stay `0` on `ChatResponse` too in that case. Anthropic doesn't have this problem (it streams real usage incrementally, see [Streaming](./streaming.md#usage-chunks)); Ollama has the same final-chunk-only limitation but isn't covered by what's described here.
+
+This is the only usage source DeepSeek's streaming path has, so it's treated as the real number, not flagged as a lesser "estimate" — whatever lands on the final `done: true` chunk (and the returned `ChatResponse`) is what a consumer should use for billing/accounting, the same as it would for any other provider's real, API-reported usage. The two sides of that number are revealed at different points, though: `outputTokens` grows on every streamed chunk as content arrives, since it's cheap to recompute from what's been generated so far; `inputMissTokens`/`inputCacheTokens` are only computed once, on that final chunk, since the input side never changes mid-call and there's no reason to resend the same number on every content delta.
+
 ## Think effort
 
 Reasoning/thinking models each expose a different native "how hard should the model think" knob — OpenAI and Ollama use `low`/`medium`/`high`, DeepSeek uses `none`/`low`/`high`/`max`, Anthropic uses a numeric thinking-token budget. OrbitX replaces all of that with one universal option on every provider's constructor:
@@ -60,6 +95,19 @@ Per-provider specifics:
 
 See [Streaming](./streaming.md#thinking-chunks) for how (and whether) a given provider's reasoning text is observable while it streams.
 
+## User tracking & prompt caching
+
+Every provider's constructor accepts `userId?: string` — an opaque end-user identifier forwarded to the underlying API for that provider's own abuse-monitoring/analytics, same universal-option pattern as `thinkEffort`: pass it uniformly and providers that have no equivalent field just ignore it.
+
+| Provider | Where `userId` goes | Notes |
+|---|---|---|
+| Anthropic | `metadata.user_id` on every request | Anthropic's documented field for this purpose. |
+| OpenAI | `user_id` on every request | Sent as an extra body field alongside the standard params. |
+| DeepSeek | `user_id` on every request | Same mechanism as OpenAI (DeepSeek's API is OpenAI-compatible). |
+| Ollama | Ignored | Accepted in the options type for interface parity only — Ollama's API has no equivalent field. |
+
+**Anthropic prompt caching is on by default.** Every request marks the last content block with an ephemeral `cache_control` breakpoint, so a growing conversation gets served from cache instead of being fully reprocessed each turn — pass `disablePromptCaching: true` to send every request uncached. DeepSeek's cache is automatic server-side (no `cache_control` equivalent to toggle) and reports its own explicit `prompt_cache_hit_tokens`/`prompt_cache_miss_tokens`. OpenAI's cache is also automatic, reporting hits via `prompt_tokens_details.cached_tokens`. Ollama has no cache concept at all — `inputCacheTokens` is always `0`.
+
 ## Built-in providers
 
 ### Ollama
@@ -71,6 +119,7 @@ new OllamaProvider(model: string, host = "http://localhost:11434", options?: {
   supportsTools?: boolean;   // default false
   contextWindow?: number;    // default 32_000
   thinkEffort?: number;      // 0-1, see "Think effort" above
+  userId?: string;           // accepted for interface parity only — ignored, Ollama has no equivalent field
 });
 ```
 
@@ -86,12 +135,13 @@ new DeepSeekProvider(apiKey: string, model = "deepseek-v4-flash", options?: {
   supportsImages?: boolean;  // default: true only for deepseek-v4-flash-vision-exp
   contextWindow?: number;    // default 1_000_000 (deepseek-v4-flash / deepseek-v4-flash-vision-exp / deepseek-v4-pro)
   thinkEffort?: number;      // 0-1, see "Think effort" above
+  userId?: string;           // see "User tracking & prompt caching" above
 });
 ```
 
 Uses the `openai` SDK pointed at `https://api.deepseek.com` (DeepSeek's API is OpenAI-compatible). `getCapabilities().supportsImages` is `false` for every model except `deepseek-v4-flash-vision-exp`, which is `deepseek-v4-flash` plus native image input (same context window, same reasoning support) — pass `supportsImages: true` explicitly if a future model adds vision support before this SDK's allowlist is updated.
 
-Token usage (`inputTokens`/`outputTokens`) is only accurate on the non-streaming path (`chat()` called without `streamCallback`), which reads `response.usage` directly. On the streaming path this provider deliberately does not set `stream_options: { include_usage: true }`, so DeepSeek never sends a usage-bearing chunk and both counts report as `0`. This matters most for `ImageDescriber`, which always calls `chat()` without a `streamCallback` (see [Token accounting](./agents.md#token-accounting)) — so image-description usage is captured correctly even though live agent streaming isn't.
+`stream_options: { include_usage: true }` is set on every streaming call, so `inputMissTokens`/`inputCacheTokens`/`outputTokens` are accurate on both the streaming and non-streaming paths — DeepSeek sends a usage-bearing final chunk (including its own `prompt_cache_hit_tokens`/`prompt_cache_miss_tokens` split) the same way OpenAI does.
 
 ### Anthropic
 
@@ -101,9 +151,12 @@ import { AnthropicProvider } from "orbitx";
 new AnthropicProvider(apiKey: string, model = "claude-sonnet-5", options?: {
   supportsTools?: boolean;    // default true
   supportsImages?: boolean;   // default true
-  contextWindow?: number;     // default from a per-model table, else 200_000
+  contextWindow?: number;     // default from a per-model table, else 200_000; an explicit value here always wins over extendedContext below
   maxTokens?: number;         // output token cap, default 4096
   thinkEffort?: number;       // 0-1, see "Think effort" above
+  userId?: string;            // see "User tracking & prompt caching" above
+  disablePromptCaching?: boolean;  // default false — prompt caching is on by default, see above
+  extendedContext?: boolean | string;  // default false/undefined — see "Context window" below
 });
 ```
 
@@ -120,27 +173,47 @@ new OpenAIProvider(apiKey: string, model = "gpt-5", options?: {
   contextWindow?: number;     // default from a per-model table, else 128_000
   baseURL?: string;           // point at an OpenAI-compatible endpoint
   thinkEffort?: number;       // 0-1, see "Think effort" above — ignored on non-reasoning models
+  userId?: string;            // see "User tracking & prompt caching" above
 });
 ```
 
 Uses the Chat Completions API (not the Responses API). `baseURL` makes this usable against any OpenAI-compatible provider, not just OpenAI itself.
 
-## Two provider roles: main + image
+## Provider roles
+
+An agent's providers are registered under open-ended role names, not a fixed pair. `"main"` — the provider driving the agent loop itself — is the only one that's required; every other role is optional and looked up on demand by whatever tool needs it.
 
-An agent has exactly two provider "roles":
+```ts
+export type ProviderType =
+  | "main"
+  | "image-describer"
+  | "image-generation"
+  | "image-editing"
+  | "video-describer"
+  | "video-generation"
+  | "video-editing"
+  | "audio-describer"
+  | "audio-generation"
+  | "audio-design"
+  | "audio-clone"
+  | "3d-model-generator"
+  | "llm-low"
+  | "llm-medium"
+  | "llm-high"
+  | (string & {});
+```
 
-- **`main`** — drives the agent loop itself (required).
-- **`image`** — describes image tool output (e.g. a screenshot from `BrowserScreenshotTool`) into text, so the main conversation doesn't have to carry raw image bytes. Optional.
+`ProviderType` (exported from `orbitx`) is the documented starting vocabulary for every role name — not a closed union, and purely for naming/registry lookup. It has no relationship to `ProviderCapabilities` (see above) — a role name doesn't imply anything about what the provider registered under it can actually do. The trailing `(string & {})` (not bare `string`) is what makes the open-endedness work: a plain `| string` union collapses to just `string` and an editor stops suggesting the literals at all, while intersecting with `{}` keeps TS from widening it away, so a custom name still type-checks the moment nothing here fits. `llm-low`/`medium`/`high` are a self-declared overall quality/durability tier (cheap-and-good-enough vs. worth relying on for longer/harder tasks) — not a specific claim about reasoning power.
 
-Pass a single provider to use it for both roles (if it supports images, it's also used as the image describer; otherwise images are dropped with an explanatory error message back to the model). Pass an array to split them:
+Pass a single provider to use it as `main` (same as today); pass an array of `{ type, provider, default? }` entries to register more:
 
 ```ts
 import { BaseAgent, DeepSeekProvider, OllamaProvider } from "orbitx";
 
 const agent = new BaseAgent({
   aiProvider: [
-    { type: "main", aiProvider: new DeepSeekProvider("api-key", "deepseek-v4-flash") },
-    { type: "image", aiProvider: new OllamaProvider("llava:latest") },
+    { type: "main", provider: new DeepSeekProvider("api-key", "deepseek-v4-flash") },
+    { type: "image-describer", provider: new OllamaProvider("llava:latest") },
   ],
   instruction: "You are a helpful assistant.",
   mcpClient,
@@ -148,8 +221,34 @@ const agent = new BaseAgent({
 });
 ```
 
-This resolution is done by `resolveAgentProviders()` (`AgentProvidersInput = AIProvider | AgentProviderEntry[]`), exported from `orbitx` if you need it directly. There is no third "utils" role — every tool/skill is included in the system prompt from the start, so there's nothing left for a separate pass to decide.
+`provider` can also be an array — a pool of candidates for that role — with `default` (0-based, defaults to `0`) picking which one is used when code needs exactly one. The rest of the pool is there to be enumerated later; nothing built-in does smarter selection than "the default" yet.
+
+### Resolving a provider by role
+
+`ProviderRegistry.getProvider(type)` tries an exact role-name match first (that role's pool default). If nothing is registered under `type`, it falls back to scanning every provider registered under *any* role for the first one whose `getCapabilities()` reports the matching capability flag — not a generic capability bag keyed by `ProviderType` (that's gone; `ProviderType` is purely a registry role name and has no relationship to what a model can do), but a fixed, small mapping from a handful of well-known role names to their one corresponding `ProviderCapabilities` flag:
+
+| Role (`ProviderType`) | Capability flag checked in the fallback |
+|---|---|
+| `"image-describer"` | `supportsImages` |
+| `"image-generation"` | `supportsImageGeneration` |
+| `"image-editing"` | `supportsImageEditing` |
+| `"video-describer"` | `supportsVideo` |
+| `"video-generation"` | `supportsVideoGeneration` |
+| `"video-editing"` | `supportsVideoEditing` |
+| `"audio-describer"` | `supportsAudio` |
+| `"audio-generation"` | `supportsAudioGeneration` |
+| `"audio-design"` | `supportsAudioDesign` |
+| `"audio-clone"` | `supportsAudioClone` |
+| `"3d-model-generator"` | `supports3DModelGeneration` |
+
+This is why a single vision-capable `main` provider "just works" as an image describer with zero extra config: nothing needs to be registered under `"image-describer"` explicitly — the fallback finds `main` because its `getCapabilities().supportsImages` is `true`. Role names outside this table (`"main"`, `"llm-low"`/`"medium"`/`"high"`, or any custom string) have no capability equivalent, so the fallback never applies to them — an unregistered role like that returns `undefined` from `getProvider` regardless of what any registered provider can do.
+
+`BaseAgent` is the only thing that ever calls `getProvider` directly — a tool never touches a `ProviderRegistry`, or a raw provider, itself. It reaches this indirectly via `mcp.executeProvider(toolCallId, type, input)` (see [Tools](./tools.md#mcpexecuteprovider--calling-a-provider-from-inside-a-tool)), which works the same way whether the tool is registered on the local `MCPClient` or a sandboxed/remote `MCPServer` — the request is always resolved on `BaseAgent`'s side, never handed across as a live object.
+
+`resolveAgentProviders()` (`AgentProvidersInput = AIProvider | AgentProviderEntry[]`) builds the `ProviderRegistry` this all runs on — both are exported from `orbitx` if you need them directly (`getMain()`, `getProvider(type)`, `getProviders(type)` for the full pool under an exact role). There is no separate "utils" role — every tool/skill is included in the system prompt from the start, so there's nothing left for a separate pass to decide.
 
 ## Writing a custom provider
 
 Extend `AIProvider` and implement `chat`/`getCapabilities`. Follow an existing provider (e.g. `src/core/ollama-provider.ts`) as a template: translate `Message[]` to your API's wire format, stream text chunks to `streamCallback` as they arrive, accumulate tool-call deltas and emit them on the final chunk, and translate `ToolSchema[]` into your API's function-calling format (see `src/core/tool-schema-translator.ts` for existing OpenAI/Anthropic translators you can reuse or reference).
+
+`inputMissTokens`/`inputCacheTokens`/`outputTokens` on the returned `ChatResponse` are optional — every built-in provider is token-billed and always sets them, but a provider registered under a per-request-billed role (e.g. `image-generation`, `audio-clone`) should omit them and set `cost` instead. `BaseAgent#executeProvider` (see [Tools](./tools.md#mcpexecuteprovider--calling-a-provider-from-inside-a-tool)) checks `cost` first and records a `unit: "cost"` usage entry when it's present, falling back to the token fields (defaulting any missing one to `0`) otherwise — see [Token accounting](./agents.md#token-accounting).

package/docs/streaming.md

@@ -11,12 +11,14 @@ type StreamCallback = (chunk: {
   toolCallId?: string;            // on "tool"-role chunks
   toolName?: string;              // on "tool"-role chunks
   thinking?: string;               // reasoning-text delta, mirroring `content` — see "Thinking chunks" below
+  usage?: { inputMissTokens: number; inputCacheTokens: number; outputTokens: number };  // see "Usage chunks" below
+  hidden?: boolean;                // true on messages BaseAgent injected itself
 }) => Promise<void> | void;
 ```
 
 ## What you actually receive, turn by turn
 
-1. **`role: "user"`** — one chunk, `done: true`, echoing the prompt that was just added to history (either the one you passed to `run()`, or a queued one — see [Agents](./agents.md#the-run-loop)).
+1. **`role: "user"`** — one chunk, `done: true`, echoing the prompt that was just added to history (either the one you passed to `run()`, or a queued one — see [Agents](./agents.md#the-run-loop)). One exception: the forced-compaction nudge (see [Agents](./agents.md#memory-compaction)) is also a `role: "user"` chunk, but carries `hidden: true` since it's `BaseAgent` talking to the model, not the actual user — check that flag before showing a `"user"` chunk to an end user.
 2. **`role: "assistant"`** — the model's reply, streamed incrementally: zero or more `done: false` chunks each carrying the next slice of `content` as it arrives from the provider, followed by one final `done: true` chunk. `content` is delta text, not the accumulated total — append it yourself if you need the running message.
    - If the model requested tool calls this turn, each one is emitted individually, mid-stream, as soon as *that* call finishes generating — `toolCalls: [thatOneCall]` on a `done: false` chunk. See "Incremental tool-call chunks" below.
 3. **`role: "tool"`** — one chunk per dispatched tool call, `done: true`, with `content` set to the tool's result text and `toolCallId`/`toolName` identifying which call it answers. Emitted after the tool has actually finished executing (there is no streaming/progress signal for tool execution itself through this callback — see below for how to get that separately).
@@ -49,6 +51,21 @@ Detection differs by provider, since not every API has an explicit "this call is
 | OpenAI / DeepSeek | Inferred: the API streams one call's arguments at a time with no end marker, so a call is treated as finished the instant the *next* call's first delta (carrying a fresh `id`) arrives, or the stream ends (for the last/only call). |
 | Ollama | Ollama sends each tool call already fully formed (not built up token by token), so it's emitted the moment it's seen — deduplicated in case the same call reappears in a later chunk. |
 
+## Usage chunks
+
+Every `assistant`-role chunk carries `usage` — not just the ones with a fresh number to report. Where a value isn't known yet, it's an explicit `0`, never an absent/undefined field, so consuming code can always read `chunk.usage.outputTokens` etc. without separately guarding for `usage` or its fields being missing. This is the same `streamCallback` passed into `run()`, forwarded straight through to the provider, so external code (e.g. something watching an external credit/cost budget, deciding when to call `immediateStop()`) sees it too — no separate channel.
+
+Every number here is real, straight from whatever the provider reports — nothing in this SDK estimates or guesses a token count. What differs is *when* each field stops being a placeholder `0` and starts reflecting the API's real number:
+
+| Provider | Mid-stream `usage` | Final `usage` (on `done: true`) |
+|---|---|---|
+| Anthropic | `inputMissTokens`/`inputCacheTokens` are real from the very first chunk (before any content) — that's simply when its protocol first has them, not something computed early on purpose. `outputTokens` updates on every delta. | Same numbers, just the last update. |
+| OpenAI | All three fields are `0` on every chunk before the last one (`include_usage` is honored correctly, but only delivers on completion — there's nothing to report early). | Real, billed numbers for all three fields. If the call is cut off before this chunk arrives, `inputMissTokens`/`inputCacheTokens`/`outputTokens` come back `0` on `ChatResponse` too — the API never computed them, so there's nothing to report. |
+| DeepSeek | Same as OpenAI — `0` on every chunk before the last one. | Real, billed numbers, including its own `prompt_cache_hit_tokens`/`prompt_cache_miss_tokens` for the cache split. Same `0`-on-abort behavior as OpenAI if the call is cut off first. |
+| Ollama | All three fields are `0` on every chunk before the last one — no live signal exists for this provider at all. | Real. |
+
+Practical implication: for Anthropic and DeepSeek you get a live, reactable signal before a turn finishes (immediately for Anthropic's input side, throughout for either provider's output side); for OpenAI there's no live signal at all today — you only find out after the turn finishes, or get a hard `0` if it was cut off first.
+
 ## Minimal consumer
 
 ```ts
@@ -110,7 +127,7 @@ Once a turn finishes, its accumulated reasoning is available two ways on both `C
 
 ## What's *not* in the stream today
 
-- **No live per-tool progress channel through `StreamCallback`.** For domains that emit their own progress (browser navigation, long-running bash processes), subscribe to that domain's `*Interaction` `EventEmitter` directly instead (e.g. `BrowserInteraction`, `BashInteraction`) — see [Tools](./tools.md).
+- **No live per-tool progress channel through `StreamCallback`.** There's no built-in way to observe a long-running tool call (browser navigation, a bash process) mid-flight — `execute()` runs to completion and its return value is the only thing surfaced.
 - **Tool calls still only appear once, fully assembled** — see above. `thinkEffort`/thinking chunks don't change that.
 
 If your use case depends on either of these, check the project's issue tracker / recent changes before assuming they're unavailable — this area of the SDK is actively evolving.

package/docs/tools.md

@@ -16,21 +16,54 @@ new MCPTool({
     default?: any;
   }[];
   stopIterationAfterUsingThisTool?: boolean;   // default false
-  customClass?: MCPCustomClass;                // optional stateful helper, see below
   execute: (
     envID: string,
     inputs: Record<string, any>,
-    mcp?: MCP,
-    customClass?: typeof customClass
+    toolCallId?: string,
+    mcp?: MCP
   ) => Promise<any>;
 });
 ```
 
 - `inputs` doubles as both the JSON-schema-like description sent to the model (translated per-provider by `src/core/tool-schema-translator.ts`) and the shape `execute` should expect on `inputs`.
 - `stopIterationAfterUsingThisTool: true` ends the agent's run loop immediately after this tool fires, even if the model didn't naturally stop calling tools — useful for a "final answer" or "hand off" style tool.
-- `execute`'s return value is normalized via `normalizeToolOutput()` into either `{ type: "text", output: {...} }` or `{ type: "image", output: { image, mimeType, ... } }`. Return `{ type: "image", output: {...} }` explicitly to hand back an image (it gets routed through the agent's image provider — see [Providers](./providers.md#two-provider-roles-main--image)); anything else is treated as a plain JSON text result.
+- `execute`'s return value is normalized via `normalizeToolOutput()` into `{ output: {...} }` — return a plain object (or anything else) and it becomes `output` as-is, same convention as always.
 - `envID` identifies which "environment" (in-process client id, IPC/WS session) the call came from — most tools ignore it unless they need per-environment isolated state (see `MCPStorage`/`MCPFSStorage` in [MCP Architecture](./mcp-architecture.md)).
-- `customClass` (a subclass of `MCPCustomClass`) is a way to give a tool its own persistent helper object with access to the owning `MCP` instance (`getMCP()`) and an `EventEmitter` (`getEvents()`) for progress/status events — see how the built-in `*Interaction` classes are used (e.g. `FsInteraction`, `BrowserInteraction`) if you want to expose live progress to a UI. **Experimental:** `customClass` is still settling — its name and shape may change (or it may be removed) in a future release, so avoid depending on it for anything beyond the built-in `*Interaction` pattern until it stabilizes.
+- `toolCallId` is this specific call's id, needed only by a tool that calls `mcp.executeProvider(...)` (see below) — everything else can ignore it.
+
+### `mcp.executeProvider(...)` — calling a provider from inside a tool
+
+`MCP` (what `execute`'s 4th argument gives you, same as it always has) has one method beyond `getStorage()`/`getRNG()`:
+
+```ts
+abstract class MCP {
+  getStorage(): MCPStorage;
+  getRNG(): MCPRNG;
+  executeProvider(toolCallId: string, type: ProviderType, input: Record<string, any>): Promise<{ output: Record<string, any> }>;
+}
+```
+
+`type` picks a role (see [Provider roles](./providers.md#provider-roles)); `input` must be `Message`-shaped — `{ content: string }` for plain text, or `{ parts: MessageContentPart[] }` when real multimodal content (an image, say) needs to reach the provider. `BaseAgent#executeProvider` spreads `input` straight onto the outgoing `role: "user"` message (no `safetyPolicies` system message here, unlike the main run loop — see [Guardrail system prompt](./agents.md#guardrail-system-prompt)) and stays completely type-agnostic itself — it never branches on `type`, so a role that needs `parts` (`image-describer`) and one that only needs `content` (a plain text-generation role) go through the exact same code path. Shaping `input` correctly is the calling tool's job: **use `parts`, not a `data:` URI folded into `content`, for images/video** — every built-in provider (Anthropic/OpenAI/DeepSeek/Ollama) only builds a real multimodal content block from `Message.parts`; a base64 string inside `content` is sent to the model as literal text, not an image. This works identically no matter where the tool is registered — directly on an `MCPClient`, or on a sandboxed/remote `MCPServer` (see [MCP Architecture](./mcp-architecture.md#custom-mcp-subclasses)) — because the actual provider call always happens on the trusted side (wherever `BaseAgent` lives): a client-registered tool reaches it in-process, a server-registered one has the request proxied there over the existing connection and back. No provider instance, or the credentials it holds, is ever visible to the tool — only this request/response.
+
+**Off by default.** `executeProvider` throws immediately unless the owning `BaseAgent` was constructed with `features: { executeProviderFromMCPTool: true }` (see [`executeProvider` feature flag](./agents.md#executeprovider-feature-flag)) — a tool reaching a provider directly, with no guardrail prompt ahead of it, is a bigger trust surface than one that only returns data to the model, so it needs an explicit opt-in from whoever builds the agent.
+
+`read-image`/`browser-screenshot` (below) are the reference example — the instruction as a `text` part, the base64 image as an `image` part:
+
+```ts
+const { output } = await mcp.executeProvider(toolCallId, "image-describer", {
+  parts: [
+    { type: "text", text: instruction },
+    { type: "image", image, mimeType },
+  ],
+});
+return { description: output.content }; // output.content: string
+```
+
+If `executeProvider` rejects (no provider configured for that role, etc.), just let it throw — `BaseAgent`'s existing `execute()` error handling turns it into a normal `Error: ...` tool result, no special handling needed.
+
+### Usage is never something a tool reports itself
+
+There's no `usage` field on what `execute` returns. Every `executeProvider` call is recorded by `BaseAgent` the instant the real provider call resolves — keyed by `toolCallId`, moved onto the tool's result `Message.usage` only after `execute()` returns (see [Token accounting](./agents.md#token-accounting)) — specifically so a compromised or rewritten tool can't just under-report what it spent; it never holds that number to begin with.
 
 Tools are registered on an `MCPServer` and invoked through an `MCPClient`; `SimpleAgent` does this wiring for you from a flat `tools` array.
 
@@ -38,7 +71,7 @@ One tool schema the model sees isn't an `MCPTool` at all: `BaseAgent` always app
 
 ## Built-in tool catalog
 
-Import a whole domain's tools at once (e.g. `FsTools()`), or import individual tools by name. Every domain also exports a `*Interaction` `EventEmitter` for subscribing to that domain's live progress events (e.g. `FsInteraction`, `BashInteraction`).
+Import a whole domain's tools at once (e.g. `FsTools()`), or import individual tools by name.
 
 ### Filesystem — `FsTools()`
 
@@ -90,7 +123,7 @@ Drives a real headless browser session.
 | `BrowserNetworkTool` | Read captured network request/response activity. |
 | `BrowserNetworkStatusTool` | Check whether the page currently has in-flight network activity. |
 | `BrowserInjectTool` | Evaluate arbitrary JS in the page and return the result. |
-| `BrowserScreenshotTool` | Capture a screenshot; can be described by the `image` provider instead of passed raw to the main model. |
+| `BrowserScreenshotTool` | Capture a screenshot and return a text description of it (via an `image-describer` provider — see [Provider roles](./providers.md#provider-roles)). |
 
 ### Todo lists — `TodoTools()`
 
@@ -126,7 +159,7 @@ Backs the `PlannerSkill` (see [Skills](./skills.md)), but usable standalone.
 |---|---|
 | `GetCurrentTimeTool` | Get the current date/time (ISO string, unix timestamp, timezone). |
 | `DelayTool` | Wait a given number of milliseconds (max 60000ms) before continuing. |
-| `ReadImageTool` | Read an image file off disk and hand it to the agent as an image tool-output — same handling as `BrowserScreenshotTool` (raw bytes to the model, or described by the image provider first, depending on config). |
+| `ReadImageTool` | Read an image file off disk and return a text description of it — same handling as `BrowserScreenshotTool`. |
 
 ### Multi-agent — `AgentTools(availableAgents, options?)` *(experimental)*
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "orbitx",
-  "version": "0.2.6",
+  "version": "0.2.7",
   "description": "An AI SDK providing agents, swarms, planners, skills, MCP tools, workflows, and multi-provider AI integrations.",
   "repository": {
     "type": "git",