@assistant-ui/mcp-docs-server 0.1.25 → 0.1.27

package/.docs/raw/docs/(docs)/guides/voice.mdx

@@ -0,0 +1,333 @@
+---
+title: Realtime Voice
+description: Bidirectional realtime voice conversations with AI agents.
+---
+
+import { VoiceSample } from "@/components/docs/samples/voice";
+
+assistant-ui supports realtime bidirectional voice via the `RealtimeVoiceAdapter` interface. This enables live voice conversations where the user speaks into their microphone and the AI agent responds with audio, with transcripts appearing in the thread in real time.
+
+<VoiceSample />
+
+## How It Works
+
+Unlike [Speech Synthesis](/docs/guides/speech) (text-to-speech) and [Dictation](/docs/guides/dictation) (speech-to-text), the voice adapter handles **both directions simultaneously** — the user's microphone audio is streamed to the agent, and the agent's audio response is played back, all while transcripts are appended to the message thread.
+
+| Feature | Adapter | Direction |
+|---------|---------|-----------|
+| [Speech Synthesis](/docs/guides/speech) | `SpeechSynthesisAdapter` | Text → Audio (one message at a time) |
+| [Dictation](/docs/guides/dictation) | `DictationAdapter` | Audio → Text (into composer) |
+| **Realtime Voice** | `RealtimeVoiceAdapter` | Audio ↔ Audio (bidirectional, live) |
+
+## Configuration
+
+Pass a `RealtimeVoiceAdapter` implementation to the runtime via `adapters.voice`:
+
+```tsx
+const runtime = useChatRuntime({
+  adapters: {
+    voice: new MyVoiceAdapter({ /* ... */ }),
+  },
+});
+```
+
+When a voice adapter is provided, `capabilities.voice` is automatically set to `true`.
+
+## Hooks
+
+### useVoiceState
+
+Returns the current voice session state, or `undefined` when no session is active.
+
+```tsx
+import { useVoiceState, useVoiceVolume } from "@assistant-ui/react";
+
+const voiceState = useVoiceState();
+// voiceState?.status.type — "starting" | "running" | "ended"
+// voiceState?.isMuted — boolean
+// voiceState?.mode — "listening" | "speaking"
+
+const volume = useVoiceVolume();
+// volume — number (0–1, real-time audio level via separate subscription)
+```
+
+### useVoiceControls
+
+Returns methods to control the voice session.
+
+```tsx
+import { useVoiceControls } from "@assistant-ui/react";
+
+const { connect, disconnect, mute, unmute } = useVoiceControls();
+```
+
+## UI Example
+
+```tsx
+import { useVoiceState, useVoiceControls } from "@assistant-ui/react";
+import { PhoneIcon, PhoneOffIcon, MicIcon, MicOffIcon } from "lucide-react";
+
+function VoiceControls() {
+  const voiceState = useVoiceState();
+  const { connect, disconnect, mute, unmute } = useVoiceControls();
+
+  const isRunning = voiceState?.status.type === "running";
+  const isStarting = voiceState?.status.type === "starting";
+  const isMuted = voiceState?.isMuted ?? false;
+
+  if (!isRunning && !isStarting) {
+    return (
+      <button onClick={() => connect()}>
+        <PhoneIcon /> Connect
+      </button>
+    );
+  }
+
+  return (
+    <>
+      <button onClick={() => (isMuted ? unmute() : mute())} disabled={!isRunning}>
+        {isMuted ? <MicOffIcon /> : <MicIcon />}
+        {isMuted ? "Unmute" : "Mute"}
+      </button>
+      <button onClick={() => disconnect()}>
+        <PhoneOffIcon /> Disconnect
+      </button>
+    </>
+  );
+}
+```
+
+## Custom Adapters
+
+Implement the `RealtimeVoiceAdapter` interface to integrate with any voice provider.
+
+### RealtimeVoiceAdapter Interface
+
+```tsx
+import type { RealtimeVoiceAdapter } from "@assistant-ui/react";
+
+class MyVoiceAdapter implements RealtimeVoiceAdapter {
+  connect(options: {
+    abortSignal?: AbortSignal;
+  }): RealtimeVoiceAdapter.Session {
+    // Establish connection to your voice service
+    return {
+      get status() { /* ... */ },
+      get isMuted() { /* ... */ },
+
+      disconnect: () => { /* ... */ },
+      mute: () => { /* ... */ },
+      unmute: () => { /* ... */ },
+
+      onStatusChange: (callback) => {
+        // Status: { type: "starting" } → { type: "running" } → { type: "ended", reason }
+        return () => {}; // Return unsubscribe
+      },
+
+      onTranscript: (callback) => {
+        // callback({ role: "user" | "assistant", text: "...", isFinal: true })
+        // Transcripts are automatically appended as messages in the thread.
+        return () => {};
+      },
+
+      // Report who is speaking (drives the VoiceOrb speaking animation)
+      onModeChange: (callback) => {
+        // callback("listening") — user's turn
+        // callback("speaking") — agent's turn
+        return () => {};
+      },
+
+      // Report real-time audio level (0–1) for visual feedback
+      onVolumeChange: (callback) => {
+        // callback(0.72) — drives VoiceOrb amplitude and waveform bar heights
+        return () => {};
+      },
+    };
+  }
+}
+```
+
+### Session Lifecycle
+
+The session status follows the same pattern as other adapters:
+
+```
+starting → running → ended
+```
+
+The `ended` status includes a `reason`:
+- `"finished"` — session ended normally
+- `"cancelled"` — session was cancelled by the user
+- `"error"` — session ended due to an error (includes `error` field)
+
+### Mode and Volume
+
+All adapters must implement `onModeChange` and `onVolumeChange`. If your provider doesn't support these, return a no-op unsubscribe:
+
+- **`onModeChange`** — Reports `"listening"` (user's turn) or `"speaking"` (agent's turn). The `VoiceOrb` switches to the active speaking animation.
+- **`onVolumeChange`** — Reports a real-time audio level (`0`–`1`). The `VoiceOrb` modulates its amplitude and glow, and waveform bars scale to match.
+
+When using `createVoiceSession`, these are handled automatically — call `session.emitMode()` and `session.emitVolume()` when your provider delivers data.
+
+### Transcript Handling
+
+Transcripts emitted via `onTranscript` are automatically appended to the message thread:
+
+- **User transcripts** (`role: "user"`, `isFinal: true`) are appended as user messages.
+- **Assistant transcripts** (`role: "assistant"`) are streamed into an assistant message. The message shows a "running" status until `isFinal: true` is received.
+
+## Example: ElevenLabs Conversational AI
+
+[ElevenLabs Conversational AI](https://elevenlabs.io/docs/agents-platform/overview) provides realtime voice agents via WebRTC.
+
+### Install Dependencies
+
+```bash
+npm install @elevenlabs/client
+```
+
+### Adapter
+
+```tsx title="lib/elevenlabs-voice-adapter.ts"
+import type { RealtimeVoiceAdapter, Unsubscribe } from "@assistant-ui/react";
+import { VoiceConversation } from "@elevenlabs/client";
+
+export class ElevenLabsVoiceAdapter implements RealtimeVoiceAdapter {
+  private _agentId: string;
+
+  constructor(options: { agentId: string }) {
+    this._agentId = options.agentId;
+  }
+
+  connect(options: {
+    abortSignal?: AbortSignal;
+  }): RealtimeVoiceAdapter.Session {
+    const statusCallbacks = new Set<(s: RealtimeVoiceAdapter.Status) => void>();
+    const transcriptCallbacks = new Set<(t: RealtimeVoiceAdapter.TranscriptItem) => void>();
+    const modeCallbacks = new Set<(m: RealtimeVoiceAdapter.Mode) => void>();
+    const volumeCallbacks = new Set<(v: number) => void>();
+
+    let currentStatus: RealtimeVoiceAdapter.Status = { type: "starting" };
+    let isMuted = false;
+    let conversation: VoiceConversation | null = null;
+    let disposed = false;
+
+    const updateStatus = (status: RealtimeVoiceAdapter.Status) => {
+      if (disposed) return;
+      currentStatus = status;
+      for (const cb of statusCallbacks) cb(status);
+    };
+
+    const cleanup = () => {
+      disposed = true;
+      conversation = null;
+      statusCallbacks.clear();
+      transcriptCallbacks.clear();
+      modeCallbacks.clear();
+      volumeCallbacks.clear();
+    };
+
+    const session: RealtimeVoiceAdapter.Session = {
+      get status() { return currentStatus; },
+      get isMuted() { return isMuted; },
+      disconnect: () => { conversation?.endSession(); cleanup(); },
+      mute: () => { conversation?.setMicMuted(true); isMuted = true; },
+      unmute: () => { conversation?.setMicMuted(false); isMuted = false; },
+      onStatusChange: (cb): Unsubscribe => {
+        statusCallbacks.add(cb);
+        return () => statusCallbacks.delete(cb);
+      },
+      onTranscript: (cb): Unsubscribe => {
+        transcriptCallbacks.add(cb);
+        return () => transcriptCallbacks.delete(cb);
+      },
+      onModeChange: (cb): Unsubscribe => {
+        modeCallbacks.add(cb);
+        return () => modeCallbacks.delete(cb);
+      },
+      onVolumeChange: (cb): Unsubscribe => {
+        volumeCallbacks.add(cb);
+        return () => volumeCallbacks.delete(cb);
+      },
+    };
+
+    if (options.abortSignal) {
+      options.abortSignal.addEventListener("abort", () => {
+        conversation?.endSession(); cleanup();
+      }, { once: true });
+    }
+
+    const doConnect = async () => {
+      if (disposed) return;
+      try {
+        conversation = await VoiceConversation.startSession({
+          agentId: this._agentId,
+          onConnect: () => updateStatus({ type: "running" }),
+          onDisconnect: () => { updateStatus({ type: "ended", reason: "finished" }); cleanup(); },
+          onError: (msg) => { updateStatus({ type: "ended", reason: "error", error: new Error(msg) }); cleanup(); },
+          onModeChange: ({ mode }) => {
+            if (disposed) return;
+            for (const cb of modeCallbacks) cb(mode === "speaking" ? "speaking" : "listening");
+          },
+          onMessage: (msg) => {
+            if (disposed) return;
+            for (const cb of transcriptCallbacks) {
+              cb({ role: msg.role === "user" ? "user" : "assistant", text: msg.message, isFinal: true });
+            }
+          },
+        });
+      } catch (error) {
+        updateStatus({ type: "ended", reason: "error", error }); cleanup();
+      }
+    };
+
+    doConnect();
+    return session;
+  }
+}
+```
+
+### Usage
+
+```tsx
+import { ElevenLabsVoiceAdapter } from "@/lib/elevenlabs-voice-adapter";
+
+const runtime = useChatRuntime({
+  adapters: {
+    voice: new ElevenLabsVoiceAdapter({
+      agentId: process.env.NEXT_PUBLIC_ELEVENLABS_AGENT_ID!,
+    }),
+  },
+});
+```
+
+## Example: LiveKit
+
+[LiveKit](https://livekit.io/) provides realtime voice via WebRTC rooms with transcription support.
+
+### Install Dependencies
+
+```bash
+npm install livekit-client
+```
+
+### Usage
+
+```tsx
+import { LiveKitVoiceAdapter } from "@/lib/livekit-voice-adapter";
+
+const runtime = useChatRuntime({
+  adapters: {
+    voice: new LiveKitVoiceAdapter({
+      url: process.env.NEXT_PUBLIC_LIVEKIT_URL!,
+      token: async () => {
+        const res = await fetch("/api/livekit-token", { method: "POST" });
+        const { token } = await res.json();
+        return token;
+      },
+    }),
+  },
+});
+```
+
+See the `examples/with-livekit` directory in the repository for a complete implementation including the adapter and token endpoint.

package/.docs/raw/docs/(reference)/api-reference/primitives/composer.mdx

@@ -415,3 +415,131 @@ import { AuiIf } from "@assistant-ui/react";
   {/* rendered if dictation is active */}
 </AuiIf>
 ```
+
+## Mention Primitives (Unstable)
+
+<Callout type="warn">
+  These primitives are under the `Unstable_` prefix and may change without notice.
+</Callout>
+
+Primitives for an @-mention picker in the composer. See the [Mention component guide](/docs/ui/mention) for a pre-built implementation.
+
+### Anatomy
+
+```tsx
+import { ComposerPrimitive } from "@assistant-ui/react";
+
+const Composer = () => (
+  <ComposerPrimitive.Unstable_MentionRoot adapter={mentionAdapter}>
+    <ComposerPrimitive.Root>
+      <ComposerPrimitive.Input />
+      <ComposerPrimitive.Unstable_MentionPopover>
+        <ComposerPrimitive.Unstable_MentionCategories>
+          {(categories) =>
+            categories.map((cat) => (
+              <ComposerPrimitive.Unstable_MentionCategoryItem
+                key={cat.id}
+                categoryId={cat.id}
+              >
+                {cat.label}
+              </ComposerPrimitive.Unstable_MentionCategoryItem>
+            ))
+          }
+        </ComposerPrimitive.Unstable_MentionCategories>
+        <ComposerPrimitive.Unstable_MentionItems>
+          {(items) =>
+            items.map((item) => (
+              <ComposerPrimitive.Unstable_MentionItem
+                key={item.id}
+                item={item}
+              >
+                {item.label}
+              </ComposerPrimitive.Unstable_MentionItem>
+            ))
+          }
+        </ComposerPrimitive.Unstable_MentionItems>
+        <ComposerPrimitive.Unstable_MentionBack>
+          Back
+        </ComposerPrimitive.Unstable_MentionBack>
+      </ComposerPrimitive.Unstable_MentionPopover>
+    </ComposerPrimitive.Root>
+  </ComposerPrimitive.Unstable_MentionRoot>
+);
+```
+
+### Unstable_MentionRoot
+
+Provider that wraps the composer with mention trigger detection, keyboard navigation, and popover state.
+
+| Prop | Type | Default | Description |
+| --- | --- | --- | --- |
+| `adapter` | `Unstable_MentionAdapter` | — | Provides categories, items, and search |
+| `trigger` | `string` | `"@"` | Character(s) that activate the popover |
+| `formatter` | `Unstable_DirectiveFormatter` | Default | Serializer/parser for mention directives |
+
+### Unstable_MentionPopover
+
+Container that only renders when a mention trigger is active. Renders a `<div>` with `role="listbox"`.
+
+### Unstable_MentionCategories
+
+Renders the top-level category list. Accepts a render function `(categories) => ReactNode`. Hidden when a category is selected or when in search mode.
+
+### Unstable_MentionCategoryItem
+
+A button that drills into a category. Renders `role="option"` with automatic `data-highlighted` and `aria-selected` when keyboard-navigated.
+
+| Prop | Type | Description |
+| --- | --- | --- |
+| `categoryId` | `string` | The category to select on click |
+
+### Unstable_MentionItems
+
+Renders the item list for the active category or search results. Accepts a render function `(items) => ReactNode`. Hidden when no category is selected and not in search mode.
+
+### Unstable_MentionItem
+
+A button that inserts a mention into the composer. Renders `role="option"` with automatic `data-highlighted` and `aria-selected` when keyboard-navigated.
+
+| Prop | Type | Description |
+| --- | --- | --- |
+| `item` | `Unstable_MentionItem` | The item to insert on click |
+
+### Unstable_MentionBack
+
+A button that navigates back from items to the category list. Only renders when a category is active.
+
+### unstable_useMentionContext
+
+Hook to access the mention popover state and actions from within `Unstable_MentionRoot`.
+
+```tsx
+const {
+  open,           // boolean — whether popover is visible
+  query,          // string — text after the trigger character
+  isSearchMode,   // boolean — whether showing search results
+  highlightedIndex,
+  categories,
+  items,
+  activeCategoryId,
+  selectCategory,
+  selectItem,
+  goBack,
+  close,
+  handleKeyDown,
+  formatter,
+} = unstable_useMentionContext();
+```
+
+### unstable_useToolMentionAdapter
+
+Hook that creates a `Unstable_MentionAdapter` from registered tools (via `useAssistantTool`).
+
+```tsx
+import { unstable_useToolMentionAdapter } from "@assistant-ui/react";
+
+const adapter = unstable_useToolMentionAdapter({
+  formatLabel: (name) => name.replaceAll("_", " "),
+  categoryLabel: "Tools",
+});
+```

package/.docs/raw/docs/(reference)/api-reference/primitives/message-part.mdx

@@ -37,6 +37,29 @@ Custom data events that can be rendered as UI at their position in the message s
 
 You can use either the explicit format `{ type: "data", name: "workflow", data: {...} }` or the shorthand `data-*` prefixed format `{ type: "data-workflow", data: {...} }`. The prefixed format is automatically converted to a `DataMessagePart` (stripping the `data-` prefix as the `name`). Unknown message part types that don't match any built-in type are silently skipped with a console warning.
 
+#### Streaming Data Parts
+
+Data parts can be sent from the server using `appendData()` on the stream controller:
+
+```ts
+controller.appendData({
+  type: "data",
+  name: "chart",
+  data: { labels: ["Q1", "Q2"], values: [10, 20] },
+});
+```
+
+Register a renderer with `makeAssistantDataUI` to display data parts:
+
+```tsx
+import { makeAssistantDataUI } from "@assistant-ui/react";
+
+const ChartUI = makeAssistantDataUI({
+  name: "chart",
+  render: ({ data }) => <MyChart data={data} />,
+});
+```
+
 ## Anatomy
 
 ```tsx

package/.docs/raw/docs/cloud/ai-sdk-assistant-ui.mdx

@@ -256,6 +256,12 @@ const cloud = new AssistantCloud({
 
 Return `null` from `beforeReport` to skip reporting a specific run. To disable telemetry entirely, pass `telemetry: false`.
 
+### Sub-Agent Model Tracking
+
+When tool calls delegate to a different model (e.g., the main run uses GPT but a tool invokes Gemini), you can track the delegated model's usage. Pass sampling call data through `messageMetadata.samplingCalls` in your API route, and the telemetry reporter will automatically include it in the report.
+
+See the [AI SDK Telemetry guide](/docs/cloud/ai-sdk#sub-agent-model-tracking) for the full setup with `createSamplingCollector` and `wrapSamplingHandler`.
+
 ## Authentication
 
 The example above uses anonymous mode (browser session-based user ID) via the env var. For production apps with user accounts, pass an explicit cloud instance:

package/.docs/raw/docs/cloud/ai-sdk.mdx

@@ -252,7 +252,7 @@ export async function POST(req: Request) {
 ```
 
 <Callout type="info">
-  The standalone hook does not capture `duration_ms`, per-step breakdowns (`steps`), custom `metadata` pass-through, or `"error"` status. These require the full runtime integration available via [`useChatRuntime`](/docs/cloud/ai-sdk-assistant-ui).
+  The standalone hook captures message metadata when it is JSON-serializable, but it does not capture `duration_ms`, per-step breakdowns (`steps`), or `"error"` status. Those require the full runtime integration available via [`useChatRuntime`](/docs/cloud/ai-sdk-assistant-ui).
 </Callout>
 
 ### Customizing Reports
@@ -274,6 +274,86 @@ const cloud = new AssistantCloud({
 
 Return `null` from `beforeReport` to skip reporting a specific run. To disable telemetry entirely, pass `telemetry: false`.
 
+### Sub-Agent Model Tracking
+
+In multi-agent setups where tool calls delegate to a different model (e.g., the main run uses GPT but a tool invokes Gemini), you can track the delegated model's usage by passing sampling call data through `messageMetadata`.
+
+**Step 1: Collect sampling data on the server**
+
+Use `createSamplingCollector` and `wrapSamplingHandler` from `assistant-cloud` to capture LLM calls made during tool execution:
+
+```ts title="app/api/chat/route.ts"
+import { streamText } from "ai";
+import { openai } from "@ai-sdk/openai";
+import {
+  createSamplingCollector,
+  wrapSamplingHandler,
+} from "assistant-cloud";
+
+export async function POST(req: Request) {
+  const { messages } = await req.json();
+
+  // Collect sub-agent sampling calls per tool call
+  const samplingCalls: Record<string, SamplingCallData[]> = {};
+
+  const result = streamText({
+    model: openai("gpt-4o"),
+    messages,
+    tools: {
+      delegate_to_gemini: tool({
+        parameters: z.object({ task: z.string() }),
+        execute: async ({ task }, { toolCallId }) => {
+          const collector = createSamplingCollector();
+          // Your sub-agent logic that calls another model
+          const result = await runSubAgent(task, {
+            onSamplingCall: collector.collect,
+          });
+          samplingCalls[toolCallId] = collector.getCalls();
+          return result;
+        },
+      }),
+    },
+  });
+
+  return result.toUIMessageStreamResponse({
+    messageMetadata: ({ part }) => {
+      if (part.type === "finish") {
+        return {
+          usage: part.totalUsage,
+          samplingCalls, // attach collected sampling data
+        };
+      }
+      if (part.type === "finish-step") {
+        return { modelId: part.response.modelId };
+      }
+      return undefined;
+    },
+  });
+}
+```
+
+**Step 2: That's it.** The telemetry reporter automatically reads `samplingCalls` from message metadata and attaches the data to matching tool calls in the report. The Cloud dashboard will show each delegated model in the model distribution chart with its own token and cost breakdown.
+
+<Callout type="info">
+  For MCP tools that use the sampling protocol, `wrapSamplingHandler` can wrap the MCP client's sampling handler directly to capture all nested LLM calls transparently.
+</Callout>
+
+<Callout type="tip">
+  **On older versions** that don't yet read `samplingCalls` from metadata, use `beforeReport` to inject the data manually:
+
+  ```ts
+  telemetry: {
+    beforeReport: (report) => ({
+      ...report,
+      tool_calls: report.tool_calls?.map((tc) => ({
+        ...tc,
+        sampling_calls: samplingCalls[tc.tool_call_id],
+      })),
+    }),
+  }
+  ```
+</Callout>
+
 ## Authentication
 
 The example above uses anonymous mode (browser session-based user ID) via the env var. For production apps with user accounts, pass an explicit cloud instance: