dsh-agora 0.2.0

package/assets/agora/references/conversational-ai/agent-toolkit-android.md

@@ -0,0 +1,209 @@
+# ConvoAI Agent Toolkit — Android (Kotlin)
+
+The Android agent toolkit (`ConversationalAIAPIImpl`) wraps RTC + RTM to deliver AI transcripts, agent state, interrupts, and metrics. It is sourced from the Conversational AI Demo repo — not from the Agora SDK packages.
+
+Source: `convoaiApi/IConversationalAIAPI.kt` and `ConversationalAIAPIImpl.kt` in the demo repo.
+
+## Table of Contents
+
+- [Setup](#setup)
+- [Initialization](#initialization)
+- [Audio Configuration](#audio-configuration)
+- [Subscribing to Events](#subscribing-to-events)
+- [Sending Messages to the Agent](#sending-messages-to-the-agent)
+- [Interrupting the Agent](#interrupting-the-agent)
+- [Handling Events](#handling-events)
+- [Cleanup](#cleanup)
+
+## Setup
+
+The toolkit takes an existing `RtcEngine` and `RtmClient` — initialize both SDKs first.
+
+```kotlin
+import io.agora.rtc2.RtcEngine
+import io.agora.rtm.RtmClient
+// Copy or import from the demo repo:
+// io.agora.scene.convoai.convoaiApi.*
+```
+
+## Initialization
+
+```kotlin
+// 1. Create your RTC engine and RTM client first (standard SDK init)
+val rtcEngine: RtcEngine = // ... your existing engine
+val rtmClient: RtmClient = // ... your existing RTM client (already logged in)
+
+// 2. Create the toolkit config
+val config = ConversationalAIAPIConfig(
+    rtcEngine = rtcEngine,
+    rtmClient = rtmClient,
+    renderMode = TranscriptRenderMode.Word,   // Word (word-by-word) or Text (full sentence)
+    enableLog = true,
+    enableRenderModeFallback = true           // fall back to Text if server lacks word timestamps
+)
+
+// 3. Create the API instance
+val api = ConversationalAIAPIImpl(config)
+
+// 4. Register your event handler
+api.addHandler(eventHandler)
+```
+
+## Audio Configuration
+
+**Must be called before `rtcEngine.joinChannel()`** to configure optimal AI audio settings.
+
+```kotlin
+// Standard ConvoAI mode
+api.loadAudioSettings()
+// Equivalent to: api.loadAudioSettings(Constants.AUDIO_SCENARIO_AI_CLIENT)
+
+// If using Avatar (requires audio mixing)
+api.loadAudioSettings(Constants.AUDIO_SCENARIO_DEFAULT)
+
+// Then join RTC channel
+rtcEngine.joinChannel(token, channelName, uid, channelMediaOptions)
+```
+
+## Subscribing to Events
+
+Subscribe after logging in to RTM and before the agent starts speaking.
+
+```kotlin
+api.subscribeMessage(channelName) { error ->
+    if (error != null) {
+        Log.e("ConvoAI", "Subscribe failed: ${error.errorMessage}")
+        return@subscribeMessage
+    }
+    Log.d("ConvoAI", "Subscribed — ready to receive agent events")
+}
+
+// When done
+api.unsubscribeMessage(channelName) { error -> }
+```
+
+## Sending Messages to the Agent
+
+```kotlin
+// Text message (default priority: INTERRUPT)
+val textMsg = TextMessage(
+    text = "What is the weather today?",
+    priority = Priority.INTERRUPT,
+    responseInterruptable = true
+)
+api.chat(agentUserId, textMsg) { error ->
+    if (error != null) Log.e("ConvoAI", "Chat failed: ${error.errorMessage}")
+}
+
+// Text with APPEND priority (queue after current response)
+val appendMsg = TextMessage(text = "And tomorrow?", priority = Priority.APPEND)
+api.chat(agentUserId, appendMsg) { }
+
+// Image message (URL-based; keep base64 under 32KB total)
+val imageMsg = ImageMessage(uuid = UUID.randomUUID().toString(), imageUrl = "https://example.com/photo.jpg")
+api.chat(agentUserId, imageMsg) { error -> }
+```
+
+## Interrupting the Agent
+
+```kotlin
+api.interrupt(agentUserId) { error ->
+    if (error != null) Log.e("ConvoAI", "Interrupt failed: ${error.errorMessage}")
+}
+```
+
+## Handling Events
+
+Implement `IConversationalAIAPIEventHandler`:
+
+```kotlin
+val eventHandler = object : IConversationalAIAPIEventHandler {
+
+    // Agent state: IDLE | SILENT | LISTENING | THINKING | SPEAKING | UNKNOWN
+    override fun onAgentStateChanged(agentUserId: String, event: StateChangeEvent) {
+        Log.d("ConvoAI", "Agent $agentUserId state: ${event.state}, turn: ${event.turnId}")
+        runOnUiThread { updateStateIndicator(event.state) }
+    }
+
+    // Transcript update (fires frequently — dedup by turnId if needed)
+    override fun onTranscriptUpdated(agentUserId: String, transcript: Transcript) {
+        // transcript.type: TranscriptType.AGENT or .USER
+        // transcript.status: IN_PROGRESS | END | INTERRUPTED | UNKNOWN
+        // transcript.renderMode: Word or Text
+        runOnUiThread { updateTranscriptUI(transcript) }
+    }
+
+    // Agent interrupted mid-speech
+    override fun onAgentInterrupted(agentUserId: String, event: InterruptEvent) {
+        Log.d("ConvoAI", "Interrupted turn: ${event.turnId}")
+    }
+
+    // Performance metrics (LLM/TTS latency)
+    override fun onAgentMetrics(agentUserId: String, metric: Metric) {
+        Log.d("ConvoAI", "Metric: ${metric.type} ${metric.name} = ${metric.value}ms")
+    }
+
+    // Agent-side error (LLM/TTS failure)
+    override fun onAgentError(agentUserId: String, error: ModuleError) {
+        Log.e("ConvoAI", "Agent error: ${error.type} code=${error.code} ${error.message}")
+    }
+
+    // Message send error (e.g., image too large)
+    override fun onMessageError(agentUserId: String, error: MessageError) {
+        Log.e("ConvoAI", "Message error: ${error.chatMessageType} code=${error.code}")
+    }
+
+    // Message receipt (server acknowledged image/text)
+    override fun onMessageReceiptUpdated(agentUserId: String, receipt: MessageReceipt) {
+        Log.d("ConvoAI", "Receipt: ${receipt.type} turnId=${receipt.turnId}")
+    }
+
+    // Voiceprint registration status (technical preview)
+    override fun onAgentVoiceprintStateChanged(agentUserId: String, event: VoiceprintStateChangeEvent) {
+        Log.d("ConvoAI", "Voiceprint: ${event.status}")
+    }
+
+    // Internal debug messages — useful during development
+    override fun onDebugLog(log: String) {
+        Log.v("ConvoAI", log)
+    }
+}
+```
+
+## Cleanup
+
+```kotlin
+fun cleanup() {
+    api.unsubscribeMessage(channelName) { }
+    api.removeHandler(eventHandler)
+    api.destroy()
+}
+```
+
+Call `destroy()` when the conversation session ends. After this call the instance cannot be reused.
+
+## Key Types Reference
+
+| Type | Purpose |
+|------|---------|
+| `ConversationalAIAPIConfig` | Init config: `rtcEngine`, `rtmClient`, `renderMode`, `enableLog` |
+| `ConversationalAIAPIImpl` | Concrete implementation — create one per session |
+| `IConversationalAIAPIEventHandler` | Interface for receiving all events |
+| `TextMessage` | Text to send: `text`, `priority`, `responseInterruptable` |
+| `ImageMessage` | Image to send: `uuid`, `imageUrl` or `imageBase64` (≤32KB) |
+| `Priority` | `INTERRUPT` / `APPEND` / `IGNORE` |
+| `AgentState` | `IDLE` / `SILENT` / `LISTENING` / `THINKING` / `SPEAKING` |
+| `Transcript` | `turnId`, `text`, `type` (AGENT/USER), `status`, `renderMode` |
+| `TranscriptRenderMode` | `Word` (word-level) or `Text` (full sentence) |
+| `StateChangeEvent` | `state`, `turnId`, `timestamp` |
+| `Metric` | `type` (LLM/TTS), `name`, `value` (ms), `timestamp` |
+| `ConversationalAIAPIError` | Sealed: `RtmError(code, msg)`, `RtcError(code, msg)`, `UnknownError(msg)` |
+
+## Notes
+
+- `onTranscriptUpdated` fires at high frequency. Deduplicate on `turnId` in your UI if needed.
+- All callbacks are dispatched on the main thread — safe for UI updates.
+- The `renderMode = Word` setting falls back to `Text` automatically if the server doesn't provide word timestamps (when `enableRenderModeFallback = true`).
+- Agent state arrives via RTM presence events (REMOTE_STATE_CHANGED); transcripts arrive via RTM channel messages.
+- Image payloads via `imageBase64` must keep the total JSON message under 32KB. Use `imageUrl` for larger images.
+- Audio routing changes re-apply audio parameters automatically via `onAudioRouteChanged`.

package/assets/agora/references/conversational-ai/agent-toolkit-ios.md

@@ -0,0 +1,208 @@
+# ConvoAI Agent Toolkit — iOS (Swift)
+
+The iOS agent toolkit (`ConversationalAIAPIImpl`) wraps RTC + RTM to deliver AI transcripts, agent state, interrupts, and metrics. It is sourced from the Conversational AI Demo repo — not from the Agora SDK packages.
+
+Source: `ConversationalAIAPI/ConversationalAIAPI.swift` and `ConversationalAIAPIImpl.swift` in the demo repo.
+
+## Table of Contents
+
+- [Setup](#setup)
+- [Initialization](#initialization)
+- [Audio Configuration](#audio-configuration)
+- [Subscribing to Events](#subscribing-to-events)
+- [Sending Messages to the Agent](#sending-messages-to-the-agent)
+- [Interrupting the Agent](#interrupting-the-agent)
+- [Handling Events](#handling-events)
+- [Cleanup](#cleanup)
+
+## Setup
+
+The toolkit takes an existing `AgoraRtcEngineKit` and `AgoraRtmClientKit` — initialize both SDKs first.
+
+```swift
+import AgoraRtcKit
+import AgoraRtmKit
+import ConversationalAIAPI // or copy files directly from the demo repo
+```
+
+## Initialization
+
+```swift
+// 1. Create your RTC engine and RTM client first (standard SDK init)
+let rtcEngine: AgoraRtcEngineKit = // ... your existing engine
+let rtmKit: AgoraRtmClientKit = // ... your existing RTM client (already logged in)
+
+// 2. Create the toolkit config
+let config = ConversationalAIAPIConfig(
+    rtcEngine: rtcEngine,
+    rtmEngine: rtmKit,
+    renderMode: .words,        // .words (word-by-word) or .text (full sentence)
+    enableLog: true,
+    enableRenderModeFallback: true // fall back to .text if server lacks word timestamps
+)
+
+// 3. Create the API instance
+let api = ConversationalAIAPIImpl(config: config)
+
+// 4. Register your event handler
+api.addHandler(handler: self) // self implements ConversationalAIAPIEventHandler
+```
+
+## Audio Configuration
+
+**Must be called before `joinChannel`** on every join to configure optimal AI audio settings.
+
+```swift
+// Standard ConvoAI mode
+api.loadAudioSettings()
+// Equivalent to: api.loadAudioSettings(secnario: .aiClient)
+
+// If using Avatar (requires audio mixing)
+api.loadAudioSettings(secnario: .default)
+
+// Then join RTC channel
+rtcEngine.joinChannel(byToken: token, channelId: channelName, info: nil, uid: userId)
+```
+
+## Subscribing to Events
+
+Subscribe after logging in to RTM and before the agent starts speaking.
+
+```swift
+api.subscribeMessage(channelName: channelName) { error in
+    if let error = error {
+        print("Subscribe failed: \(error)")
+        return
+    }
+    print("Subscribed — ready to receive agent events")
+}
+
+// When done
+api.unsubscribeMessage(channelName: channelName) { error in }
+```
+
+## Sending Messages to the Agent
+
+```swift
+// Text message (default priority: INTERRUPT)
+let textMsg = TextMessage(
+    text: "What is the weather today?",
+    priority: .interrupt,
+    responseInterruptable: true
+)
+api.chat(agentUserId: agentUid, message: textMsg) { error in
+    if let error = error { print("Chat failed: \(error)") }
+}
+
+// Text with APPEND priority (queue after current response)
+let appendMsg = TextMessage(text: "And tomorrow?", priority: .append)
+api.chat(agentUserId: agentUid, message: appendMsg) { _ in }
+
+// Image message (URL-based; keep base64 under 32KB total)
+let imageMsg = ImageMessage(uuid: UUID().uuidString, imageUrl: "https://example.com/photo.jpg")
+api.chat(agentUserId: agentUid, message: imageMsg) { error in }
+```
+
+## Interrupting the Agent
+
+```swift
+api.interrupt(agentUserId: agentUid) { error in
+    if let error = error { print("Interrupt failed: \(error)") }
+}
+```
+
+## Handling Events
+
+Implement `ConversationalAIAPIEventHandler`:
+
+```swift
+extension YourViewController: ConversationalAIAPIEventHandler {
+
+    // Agent state: .silent | .listening | .thinking | .speaking | .idle | .unknown
+    func onAgentStateChanged(agentUserId: String, event: StateChangeEvent) {
+        print("Agent \(agentUserId) state: \(event.state), turn: \(event.turnId)")
+        DispatchQueue.main.async {
+            self.updateStateIndicator(event.state)
+        }
+    }
+
+    // Transcript update (fires frequently — dedup by turnId if needed)
+    func onTranscriptUpdated(agentUserId: String, transcript: Transcript) {
+        // transcript.type: .agent or .user
+        // transcript.status: .inProgress | .end | .interrupted | .unknown
+        // transcript.renderMode: .words or .text
+        DispatchQueue.main.async {
+            self.updateTranscriptUI(transcript)
+        }
+    }
+
+    // Agent interrupted mid-speech
+    func onAgentInterrupted(agentUserId: String, event: InterruptEvent) {
+        print("Interrupted turn: \(event.turnId)")
+    }
+
+    // Performance metrics (LLM/TTS latency)
+    func onAgentMetrics(agentUserId: String, metrics: Metric) {
+        print("Metric: \(metrics.type) \(metrics.name) = \(metrics.value)ms")
+    }
+
+    // Agent-side error (LLM/TTS failure)
+    func onAgentError(agentUserId: String, error: ModuleError) {
+        print("Agent error: \(error.type) code=\(error.code) \(error.message)")
+    }
+
+    // Message send error (e.g., image too large)
+    func onMessageError(agentUserId: String, error: MessageError) {
+        print("Message error: \(error.chatMessageType) code=\(error.code)")
+    }
+
+    // Message receipt (server acknowledged image/text)
+    func onMessageReceiptUpdated(agentUserId: String, messageReceipt: MessageReceipt) {
+        print("Receipt: \(messageReceipt.type) turnId=\(messageReceipt.turnId)")
+    }
+
+    // Voiceprint registration status (technical preview)
+    func onAgentVoiceprintStateChanged(agentUserId: String, event: VoiceprintStateChangeEvent) {
+        print("Voiceprint: \(event.status)")
+    }
+
+    func onDebugLog(log: String) {
+        // Internal debug messages — useful during development
+    }
+}
+```
+
+## Cleanup
+
+```swift
+func cleanup() {
+    api.unsubscribeMessage(channelName: channelName) { _ in }
+    api.removeHandler(handler: self)
+    api.destroy()
+}
+```
+
+Call `destroy()` when the conversation session ends. After this call the instance cannot be reused.
+
+## Key Types Reference
+
+| Type | Purpose |
+|------|---------|
+| `ConversationalAIAPIConfig` | Init config: `rtcEngine`, `rtmEngine`, `renderMode`, `enableLog` |
+| `ConversationalAIAPIImpl` | Concrete implementation — create one per session |
+| `ConversationalAIAPIEventHandler` | Protocol for receiving all events |
+| `TextMessage` | Text to send: `text`, `priority`, `responseInterruptable` |
+| `ImageMessage` | Image to send: `uuid`, `imageUrl` or `imageBase64` (≤32KB) |
+| `Priority` | `.interrupt` / `.append` / `.ignore` |
+| `AgentState` | `.silent` / `.listening` / `.thinking` / `.speaking` |
+| `Transcript` | `turnId`, `text`, `type` (agent/user), `status`, `renderMode` |
+| `TranscriptRenderMode` | `.words` (word-level) or `.text` (full sentence) |
+| `StateChangeEvent` | `state`, `turnId`, `timestamp` |
+| `Metric` | `type` (LLM/TTS), `name`, `value` (ms), `timestamp` |
+
+## Notes
+
+- `onTranscriptUpdated` fires at high frequency. Deduplicate on `turnId` in your UI if needed.
+- The `renderMode: .words` setting falls back to `.text` automatically if the server doesn't provide word timestamps (when `enableRenderModeFallback: true`).
+- Agent state arrives via RTM presence events; transcripts arrive via RTM channel messages.
+- Image payloads via `imageBase64` must keep the total JSON message under 32KB. Use `imageUrl` for larger images.

package/assets/agora/references/conversational-ai/agent-toolkit.md

@@ -0,0 +1,201 @@
+---
+name: agora-agent-client-toolkit
+description: |
+  Client-side TypeScript SDK for adding Agora Conversational AI features to applications
+  already using the Agora RTC SDK. Use when the user needs to integrate agora-agent-client-toolkit
+  or agora-agent-client-toolkit-react, receive transcripts, track agent state, send messages
+  to an AI agent, handle agent events, or build a ConvoAI front-end client. Triggers on
+  agora-agent-client-toolkit, AgoraVoiceAI, useConversationalAI, useTranscript, useAgentState,
+  agent transcript, agent state, TRANSCRIPT_UPDATED, AGENT_STATE_CHANGED, ConversationalAIProvider.
+license: MIT
+metadata:
+  author: agora
+  version: '1.0.0'
+---
+
+# Agent Client Toolkit
+
+Client-side SDK for adding Agora Conversational AI features to applications already using the Agora RTC SDK. Runs in the browser — adds transcript rendering, agent state tracking, and RTM-based messaging controls on top of `agora-rtc-sdk-ng`.
+
+**npm:** `agora-agent-client-toolkit` (core) · `agora-agent-client-toolkit-react` (React hooks)
+**Repo:** <https://github.com/AgoraIO-Conversational-AI/agent-client-toolkit-ts>
+
+> This toolkit is a **client add-on** — it does not start agents. Start agents via the ConvoAI REST API first. See [README.md](README.md) for the REST API.
+
+## Installation
+
+```bash
+npm install agora-agent-client-toolkit agora-rtc-sdk-ng agora-rtm
+
+# React
+npm install agora-agent-client-toolkit-react agora-rtc-react agora-rtm
+```
+
+## Initialization
+
+`AgoraVoiceAI.init()` is **async** — always `await` it. Pass the RTC client you already have.
+
+```typescript
+import AgoraRTC from 'agora-rtc-sdk-ng';
+import AgoraRTM from 'agora-rtm';
+import { AgoraVoiceAI } from 'agora-agent-client-toolkit';
+
+// Your existing RTC + RTM setup
+const rtcClient = AgoraRTC.createClient({ mode: 'rtc', codec: 'vp8' });
+const rtmClient = new AgoraRTM.RTM('APP_ID', 'RTM_USER_ID'); // must match the RTM token subject; often String(rtcUid)
+await rtmClient.login({ token: 'RTM_TOKEN' });
+
+// Initialize the toolkit — pass your existing clients
+const ai = await AgoraVoiceAI.init({
+  rtcEngine: rtcClient,
+  rtmConfig: { rtmEngine: rtmClient }, // optional — needed for sendText/interrupt
+});
+
+// Join + publish via RTC directly (toolkit does not wrap join/publish)
+await rtcClient.join('APP_ID', 'CHANNEL', 'RTC_TOKEN', null);
+const micTrack = await AgoraRTC.createMicrophoneAudioTrack();
+await rtcClient.publish([micTrack]);
+
+// Start receiving agent messages
+ai.subscribeMessage('CHANNEL');
+```
+
+## Configuration
+
+| Field                | Type                       | Required | Description                                                         |
+| -------------------- | -------------------------- | -------- | ------------------------------------------------------------------- |
+| `rtcEngine`          | `IAgoraRTCClient`          | Yes      | Your existing Agora RTC client                                      |
+| `rtmConfig`          | `{ rtmEngine: RTMClient }` | No       | Pass your RTM client for sendText/interrupt                         |
+| `renderMode`         | `TranscriptHelperMode`     | No       | `TEXT`, `WORD`, `CHUNK`, `AUTO` (default: `AUTO`) — see table below |
+| `enableLog`          | `boolean`                  | No       | Debug logging (default: `false`)                                    |
+| `enableAgoraMetrics` | `boolean`                  | No       | Load `@agora-js/report` for usage metrics                           |
+
+### Render Modes
+
+| Mode    | Update cadence               | Word timing in metadata                 | PTS required                         | When to use                                                   |
+| ------- | ---------------------------- | --------------------------------------- | ------------------------------------ | ------------------------------------------------------------- |
+| `TEXT`  | Per sentence (`final: true`) | No                                      | No                                   | Lowest overhead; subtitles                                    |
+| `WORD`  | Per word                     | Yes (`words[].start_ms`, `duration_ms`) | **Yes** (before RTC client creation) | Karaoke-style highlight                                       |
+| `CHUNK` | When all parts reassembled   | No                                      | No                                   | Fragmented transport                                          |
+| `AUTO`  | Detected from first message  | Depends on detected mode                | If WORD detected                     | Default; fine unless you need WORD and must pre-configure PTS |
+
+## Events
+
+Register handlers before calling `subscribeMessage()`. All 9 events:
+
+```typescript
+import { AgoraVoiceAIEvents } from 'agora-agent-client-toolkit';
+
+// Transcript — delivers FULL history every time, replace don't append
+ai.on(AgoraVoiceAIEvents.TRANSCRIPT_UPDATED, (transcript) => {
+  renderTranscript(transcript);
+});
+
+// Agent state — requires RTM + enable_rtm: true in agent start config
+ai.on(AgoraVoiceAIEvents.AGENT_STATE_CHANGED, (agentUserId, event) => {
+  // event.state: 'idle' | 'listening' | 'thinking' | 'speaking' | 'silent'
+  updateStatusUI(event.state);
+});
+
+// Agent interrupted (user cut off agent's response)
+ai.on(AgoraVoiceAIEvents.AGENT_INTERRUPTED, (agentUserId, event) => {
+  // event: { turnID: number, timestamp: number }
+});
+
+// Performance metrics — requires enable_metrics: true in agent start config
+ai.on(AgoraVoiceAIEvents.AGENT_METRICS, (agentUserId, metrics) => {
+  // metrics: { type: ModuleType, name: string, value: number, timestamp: number }
+  // ModuleType: 'llm' | 'mllm' | 'tts' | 'context' | 'unknown'
+});
+
+// Agent pipeline error — requires enable_error_message: true in agent start config
+ai.on(AgoraVoiceAIEvents.AGENT_ERROR, (agentUserId, error) => {
+  // error: { type: ModuleType, code: number, message: string, timestamp: number }
+  showErrorToast(error.message);
+});
+
+// Message delivery receipt — requires RTM
+ai.on(AgoraVoiceAIEvents.MESSAGE_RECEIPT_UPDATED, (agentUserId, receipt) => {});
+
+// RTM message delivery failure
+ai.on(AgoraVoiceAIEvents.MESSAGE_ERROR, (agentUserId, error) => {});
+
+// Speech Activity Level registration status — requires RTM
+ai.on(AgoraVoiceAIEvents.MESSAGE_SAL_STATUS, (agentUserId, salStatus) => {});
+
+// Internal debug log
+ai.on(AgoraVoiceAIEvents.DEBUG_LOG, (message) => {});
+```
+
+## Sending Messages & Interrupting
+
+Requires `rtmConfig` — throws `RTMRequiredError` if called without RTM.
+
+```typescript
+import {
+  ChatMessageType,
+  ChatMessagePriority,
+} from 'agora-agent-client-toolkit';
+
+// Send text to the agent
+await ai.sendText(agentUserId, {
+  messageType: ChatMessageType.TEXT,
+  text: 'What is the weather like today?',
+  priority: ChatMessagePriority.INTERRUPTED, // interrupts current agent speech
+  responseInterruptable: true,
+});
+
+// Send image to the agent
+await ai.sendImage(agentUserId, {
+  messageType: ChatMessageType.IMAGE,
+  uuid: crypto.randomUUID(), // caller-generated unique ID for receipt correlation
+  url: 'https://example.com/image.png', // or: base64: '<inline data>'
+});
+
+// Interrupt the agent's current speech
+await ai.interrupt(agentUserId);
+```
+
+### `ChatMessagePriority` values
+
+| Value         | Behavior                                                                                               |
+| ------------- | ------------------------------------------------------------------------------------------------------ |
+| `INTERRUPTED` | Sends the message and immediately interrupts any speech the agent is currently producing               |
+| `APPEND`      | Queues the message to be processed after the agent finishes its current speech turn                    |
+| `IGNORE`      | Drops the message silently if the agent is busy — use for low-priority updates only relevant when idle |
+
+`responseInterruptable: boolean` on `ChatMessageText` — when `true`, the agent's response to this message can itself be interrupted by subsequent user input.
+
+### `ChatMessageImage` fields
+
+| Field         | Required          | Description                                                                      |
+| ------------- | ----------------- | -------------------------------------------------------------------------------- |
+| `messageType` | Yes               | Must be `ChatMessageType.IMAGE`                                                  |
+| `uuid`        | Yes               | Caller-generated unique ID for receipt correlation via `MESSAGE_RECEIPT_UPDATED` |
+| `url`         | One of url/base64 | Publicly accessible image URL                                                    |
+| `base64`      | One of url/base64 | Inline image data                                                                |
+
+## Cleanup
+
+```typescript
+ai.unsubscribe(); // stop receiving channel messages
+ai.destroy(); // remove all event handlers, clear singleton
+
+await rtmClient.logout(); // you manage RTM lifecycle
+```
+
+## Critical Rules
+
+1. **`init()` is async** — always `await AgoraVoiceAI.init()`. Missing the await causes `getInstance()` to throw `NotInitializedError`.
+2. **Register events before `subscribeMessage()`** — events from messages already in the stream will be missed otherwise.
+3. **Transcript replaces, never appends** — `TRANSCRIPT_UPDATED` delivers the complete history every time. Set state to the full array, not `prev.concat(next)`.
+4. **`AgoraVoiceAI` is a singleton** — calling `init()` twice replaces the first instance. Call `destroy()` before re-initializing.
+5. **RTM is optional but required for several features** — `sendText`, `sendImage`, and `interrupt` throw `RTMRequiredError` without `rtmConfig`. `AGENT_STATE_CHANGED`, `MESSAGE_RECEIPT_UPDATED`, `MESSAGE_ERROR`, `MESSAGE_SAL_STATUS` only fire with RTM.
+6. **Agent start config flags are required for some events** — `AGENT_STATE_CHANGED` requires `advanced_features.enable_rtm: true` AND `parameters.data_channel: "rtm"`. `AGENT_METRICS` requires `parameters.enable_metrics: true`. `AGENT_ERROR` requires `parameters.enable_error_message: true`.
+7. **Toolkit does not wrap join/publish** — call `rtcClient.join()` and `rtcClient.publish()` yourself before `subscribeMessage()`.
+8. **WORD mode requires PTS metadata enabled before RTC client creation** — if `renderMode` is `TranscriptHelperMode.WORD`, call `AgoraRTC.setParameter('ENABLE_AUDIO_PTS_METADATA', true)` before calling `AgoraRTC.createClient()`. Setting it after client creation produces no error — word-level timing data simply never arrives. This also applies when using `AUTO` mode if WORD is detected; prefer explicit `WORD` mode when karaoke display is required so the pre-configuration step is obvious.
+9. **RTM identity must match the RTM token subject** — if your RTM token was minted for `String(rtcUid)`, create or log in the RTM client with that same identity. Do not mint an RTM token for one user and log in RTM with another random ID; that can show up as generic conversation-start failures instead of a clear auth error.
+
+## React Hooks
+
+For React integration, see **[agent-client-toolkit-react.md](agent-client-toolkit-react.md)**.

package/assets/agora/references/conversational-ai/agent-ui-kit.md

@@ -0,0 +1,63 @@
+# Agent UI Kit
+
+React component library (`@agora/agent-ui-kit`) — pre-built UI for ConvoAI interfaces.
+
+**Repo:** <https://github.com/AgoraIO-Conversational-AI/agent-ui-kit>
+**npm:** `@agora/agent-ui-kit`
+
+## Voice Components
+
+- `MicButton` — microphone toggle
+- `AgentVisualizer` — agent state indicator (not-joined/listening/talking/thinking)
+- `AudioVisualizer` — audio level bars
+- `LiveWaveform` — real-time waveform display
+- `MicSelector` — microphone device picker
+- `SimpleVisualizer` — simplified audio visualizer
+- `MicButtonWithVisualizer` — combined mic toggle + audio visualization
+
+> **[README — Voice Components](https://github.com/AgoraIO-Conversational-AI/agent-ui-kit#voice-components)**
+
+## Chat Components
+
+- `Conversation` — auto-scroll chat container
+- `Message` — user/assistant message bubbles
+- `ConvoTextStream` — typewriter streaming text effect
+- `Response` — structured agent response display
+
+> **[README — Chat Components](https://github.com/AgoraIO-Conversational-AI/agent-ui-kit#chat-components)**
+
+## Video Components
+
+- `LocalVideoPreview` — local camera with mirror mode
+- `AvatarVideoDisplay` — remote avatar video with connection states
+- `CameraSelector` — camera device picker
+- `Avatar` — avatar video display with connection-aware states
+
+> **[README — Video Components](https://github.com/AgoraIO-Conversational-AI/agent-ui-kit#video-components)**
+
+## Settings Components
+
+- `AgentSettings` — AIVAD toggle, language, prompt, greeting
+- `SettingsDialog` — modal wrapper
+
+> **[README — Settings Components](https://github.com/AgoraIO-Conversational-AI/agent-ui-kit#settings-components)**
+
+## Layout Components
+
+- `VideoGrid` — 2x2 desktop grid
+- `MobileTabs` — tab switcher
+- `VideoGridWithControls` — video grid with integrated media controls
+
+> **[README — Layout Components](https://github.com/AgoraIO-Conversational-AI/agent-ui-kit#layout-components)**
+
+## Hooks
+
+- `useRTMSubscription` — generic RTM message subscription by object type
+
+> **[README — Hooks](https://github.com/AgoraIO-Conversational-AI/agent-ui-kit#hooks)**
+
+## Theming
+
+Tailwind CSS-based theming.
+
+> **[README — Theming](https://github.com/AgoraIO-Conversational-AI/agent-ui-kit#theming)**