@convai/web-sdk 1.0.0-beta.1 → 1.1.0

package/README.md

@@ -1,230 +1,308 @@
 # @convai/web-sdk
 
-JavaScript/TypeScript SDK for Convai AI voice assistants. Build voice-powered AI interactions for web applications with real-time audio/video streaming. Supports both React and Vanilla JavaScript/TypeScript.
-
-## Installation
+`@convai/web-sdk` is a TypeScript-first SDK for building real-time conversational AI experiences with Convai characters on the web. It supports:
+
+- React applications with ready-to-use hooks and widget components
+- Vanilla TypeScript/JavaScript applications with a framework-agnostic widget
+- Direct core client usage for custom UIs and advanced integrations
+- Optional lipsync data pipelines for ARKit and MetaHuman rigs
+
+This document is written as a complete implementation reference, from first setup to production hardening.
+
+## Table of Contents
+
+- [1. Package Entry Points](#1-package-entry-points)
+- [2. Installation and Requirements](#2-installation-and-requirements)
+- [3. Credentials and Environment Setup](#3-credentials-and-environment-setup)
+- [4. Quick Start](#4-quick-start)
+- [5. Build a Chatbot from Scratch](#5-build-a-chatbot-from-scratch)
+- [6. Core Concepts and Lifecycle](#6-core-concepts-and-lifecycle)
+- [7. Configuration Reference (`ConvaiConfig`)](#7-configuration-reference-convaiconfig)
+- [8. Core API Reference (`ConvaiClient`)](#8-core-api-reference-convaiclient)
+- [9. Message Semantics and Turn Completion](#9-message-semantics-and-turn-completion)
+- [10. React API Reference](#10-react-api-reference)
+- [11. Vanilla API Reference](#11-vanilla-api-reference)
+- [12. Audio Integration Best Practices (Vanilla TypeScript)](#12-audio-integration-best-practices-vanilla-typescript)
+- [13. Lipsync Helpers Reference](#13-lipsync-helpers-reference)
+- [14. Error Handling and Reliability Patterns](#14-error-handling-and-reliability-patterns)
+- [15. Troubleshooting](#15-troubleshooting)
+- [16. Production Readiness Checklist](#16-production-readiness-checklist)
+- [17. Examples](#17-examples)
+- [18. License](#18-license)
+
+## 1. Package Entry Points
+
+The SDK is published with multiple entry points for different integration styles.
+
+### `@convai/web-sdk` (default)
+
+Primary exports:
+
+- `useConvaiClient`
+- `ConvaiWidget`
+- `useCharacterInfo`
+- `useLocalCameraTrack`
+- `ConvaiClient`
+- `AudioRenderer` (re-export of LiveKit `RoomAudioRenderer` for React usage)
+- `AudioContext` (re-export of LiveKit `RoomContext`)
+- Core types re-exported from `core/types`:
+  - `AudioSettings`
+  - `ConvaiConfig`
+  - `ChatMessage`
+  - `ConvaiClientState`
+  - `AudioControls`
+  - `VideoControls`
+  - `ScreenShareControls`
+  - `IConvaiClient`
+- All exports from `@convai/web-sdk/lipsync-helpers`
+- Type exports for latency models:
+  - `LatencyMonitor` (type)
+  - `LatencyMeasurement`
+  - `LatencyStats`
+
+### `@convai/web-sdk/react`
+
+React-focused entry point, equivalent to the default React API surface.
+
+### `@convai/web-sdk/vanilla`
+
+Vanilla/browser-focused exports:
+
+- `ConvaiClient`
+- `AudioRenderer` (vanilla audio playback manager)
+- `createConvaiWidget`
+- `destroyConvaiWidget`
+- Types:
+  - `VanillaWidget`
+  - `VanillaWidgetOptions`
+  - `IConvaiClient`
+  - `ConvaiConfig`
+  - `ConvaiClientState`
+  - `ChatMessage`
+
+### `@convai/web-sdk/core`
+
+Framework-agnostic low-level API:
+
+- `ConvaiClient`
+- `AudioManager`
+- `VideoManager`
+- `ScreenShareManager`
+- `MessageHandler`
+- `BlendshapeQueue`
+- `EventEmitter`
+- Type alias: `ConvaiClientType`
+- All core types from `core/types`
+- `TurnStats` type
+
+### `@convai/web-sdk/lipsync-helpers`
+
+Dedicated helpers for blendshape formats and queue creation. Full function list is in [Section 13](#13-lipsync-helpers-reference).
+
+## 2. Installation and Requirements
+
+### Install
 
 ```bash
 npm install @convai/web-sdk
 ```
 
-## Basic Setup
-
-### React
-
-```tsx
-import { useConvaiClient, ConvaiWidget } from "@convai/web-sdk";
-
-function App() {
-  const convaiClient = useConvaiClient({
-    apiKey: "your-api-key",
-    characterId: "your-character-id",
-  });
+or
 
-  return <ConvaiWidget convaiClient={convaiClient} />;
-}
+```bash
+pnpm add @convai/web-sdk
 ```
 
-### Vanilla TypeScript
-
-```typescript
-import { ConvaiClient, createConvaiWidget } from "@convai/web-sdk/vanilla";
-
-// Create client with configuration
-const client = new ConvaiClient({
-  apiKey: "your-api-key",
-  characterId: "your-character-id",
-});
-
-// Create widget - auto-connects on first user click
-const widget = createConvaiWidget(document.body, {
-  convaiClient: client,
-});
+or
 
-// Cleanup when done
-widget.destroy();
+```bash
+yarn add @convai/web-sdk
 ```
 
-## Exports
+### Runtime requirements
 
-### React Exports (`@convai/web-sdk` or `@convai/web-sdk/react`)
+- Modern browser with WebRTC support
+- Secure context (`https://` or `http://localhost`) for microphone/camera/screen access
 
-**Components:**
+### Peer dependencies
 
-- `ConvaiWidget` - Main chat widget component
+If you are using React APIs:
 
-**Hooks:**
+- `react` `^18 || ^19`
+- `react-dom` `^18 || ^19`
 
-- `useConvaiClient(config?)` - Main client hook
-- `useCharacterInfo(characterId, apiKey)` - Fetch character metadata
-- `useLocalCameraTrack()` - Get local camera track
+## 3. Credentials and Environment Setup
 
-**Core Client:**
+### Obtain credentials
 
-- `ConvaiClient` - Core client class
+1. Create/login to your Convai account.
+2. Create or select a character.
+3. Copy:
+   - API key
+   - Character ID
 
-**Types:**
+### Store credentials in environment variables
 
-- `ConvaiConfig` - Configuration interface
-- `ConvaiClientState` - Client state interface
-- `ChatMessage` - Message interface
-- `IConvaiClient` - Client interface
-- `AudioControls` - Audio control interface
-- `VideoControls` - Video control interface
-- `ScreenShareControls` - Screen share control interface
+Do not hardcode credentials in source files.
 
-**Components:**
+```bash
+# .env.local (example)
+VITE_CONVAI_API_KEY=<YOUR_CONVAI_API_KEY>
+VITE_CONVAI_CHARACTER_ID=<YOUR_CONVAI_CHARACTER_ID>
+VITE_CONVAI_API_URL=<OPTIONAL_CONVAI_BASE_URL>
+```
 
-- `AudioRenderer` - Audio playback component
-- `AudioContext` - Audio context provider
+Use these values through your build system (`import.meta.env`, process env injection, or server-provided config).
 
-### Vanilla Exports (`@convai/web-sdk/vanilla`)
+## 4. Quick Start
 
-**Functions:**
+### React
 
-- `createConvaiWidget(container, options)` - Create widget instance
-- `destroyConvaiWidget(widget)` - Destroy widget instance
+```tsx
+import { ConvaiWidget, useConvaiClient } from "@convai/web-sdk";
 
-**Classes:**
+export function App() {
+  const convaiClient = useConvaiClient({
+    apiKey: import.meta.env.VITE_CONVAI_API_KEY,
+    characterId: import.meta.env.VITE_CONVAI_CHARACTER_ID,
+    enableVideo: false,
+    startWithAudioOn: false,
+  });
 
-- `ConvaiClient` - Core client class
-- `AudioRenderer` - Audio playback handler
+  return <ConvaiWidget convaiClient={convaiClient} />;
+}
+```
 
-**Types:**
+### Vanilla TypeScript
 
-- `VanillaWidget` - Widget instance interface
-- `VanillaWidgetOptions` - Widget options interface
-- `IConvaiClient` - Client interface
-- `ConvaiConfig` - Configuration interface
-- `ConvaiClientState` - Client state interface
-- `ChatMessage` - Message interface
+```ts
+import { ConvaiClient, createConvaiWidget } from "@convai/web-sdk/vanilla";
 
-### Core Exports (`@convai/web-sdk/core`)
+const client = new ConvaiClient({
+  apiKey: import.meta.env.VITE_CONVAI_API_KEY,
+  characterId: import.meta.env.VITE_CONVAI_CHARACTER_ID,
+  enableVideo: false,
+});
 
-**Classes:**
+const widget = createConvaiWidget(document.body, {
+  convaiClient: client,
+  defaultVoiceMode: true,
+  onConnect: () => console.log("Connected"),
+  onDisconnect: () => console.log("Disconnected"),
+});
 
-- `ConvaiClient` - Main client class
-- `AudioManager` - Audio management
-- `VideoManager` - Video management
-- `ScreenShareManager` - Screen share management
-- `MessageHandler` - Message handling
-- `EventEmitter` - Event emitter base class
+window.addEventListener("beforeunload", () => {
+  widget.destroy();
+  void client.disconnect().catch(() => undefined);
+});
+```
 
-**Types:**
+## 5. Build a Chatbot from Scratch
 
-- All types from React/Vanilla exports
-- `ConvaiClientType` - Type alias for ConvaiClient
+This section shows an end-to-end approach you can use in production.
 
-## Props and Configuration
+### A) React from scratch (custom connection flow)
 
-### ConvaiWidget Props (React)
+#### Step 1: Create the client
 
 ```tsx
-interface ConvaiWidgetProps {
-  /** Convai client instance (required) */
-  convaiClient: IConvaiClient & {
-    activity?: string;
-    isAudioMuted: boolean;
-    isVideoEnabled: boolean;
-    isScreenShareActive: boolean;
-  };
-  /** Show video toggle button in settings (default: true) */
-  showVideo?: boolean;
-  /** Show screen share toggle button in settings (default: true) */
-  showScreenShare?: boolean;
-}
+import { useConvaiClient } from "@convai/web-sdk";
+
+const convaiClient = useConvaiClient({
+  apiKey: import.meta.env.VITE_CONVAI_API_KEY,
+  characterId: import.meta.env.VITE_CONVAI_CHARACTER_ID,
+  endUserId: "<UNIQUE_END_USER_ID>",
+  enableVideo: true,
+  startWithVideoOn: false,
+  startWithAudioOn: false,
+  ttsEnabled: true,
+  enableLipsync: true,
+  blendshapeConfig: {
+    format: "arkit",
+    frames_buffer_duration: 0.5,
+  },
+});
 ```
 
-### createConvaiWidget Options (Vanilla)
+#### Step 2: Connect from a user gesture with error handling
 
-```typescript
-interface VanillaWidgetOptions {
-  /** Convai client instance (required) */
-  convaiClient: IConvaiClient & {
-    activity?: string;
-    chatMessages: ChatMessage[];
-  };
-  /** Show video toggle button in settings (default: true) */
-  showVideo?: boolean;
-  /** Show screen share toggle button in settings (default: true) */
-  showScreenShare?: boolean;
+```tsx
+async function handleConnect() {
+  try {
+    await convaiClient.connect();
+  } catch (error) {
+    console.error("Connection failed:", error);
+  }
 }
 ```
 
-### ConvaiConfig
-
-```typescript
-interface ConvaiConfig {
-  /** Your Convai API key from convai.com dashboard (required) */
-  apiKey: string;
-  /** The Character ID to connect to (required) */
-  characterId: string;
-  /**
-   * End user identifier for speaker management (optional).
-   * When provided: enables long-term memory and analytics
-   * When not provided: anonymous mode, no persistent memory
-   */
-  endUserId?: string;
-  /** Custom Convai API URL (optional, defaults to production endpoint) */
-  url?: string;
-  /**
-   * Enable video capability (default: false).
-   * If true, connection_type will be "video" (supports audio, video, and screenshare).
-   * If false, connection_type will be "audio" (audio only).
-   */
-  enableVideo?: boolean;
-  /**
-   * Start with video camera on when connecting (default: false).
-   * Only works if enableVideo is true.
-   */
-  startWithVideoOn?: boolean;
-  /**
-   * Start with microphone on when connecting (default: false).
-   * If false, microphone stays off until user enables it.
-   */
-  startWithAudioOn?: boolean;
-  /** Enable text-to-speech audio generation (default: true) */
-  ttsEnabled?: boolean;
+#### Step 3: Wait for readiness before sending text
+
+```tsx
+function sendMessage(text: string) {
+  if (!convaiClient.state.isConnected || !convaiClient.isBotReady) return;
+  convaiClient.sendUserTextMessage(text);
 }
 ```
 
-## Features
+#### Step 4: Render the widget or your own UI
 
-### Video Enabled Chat
+```tsx
+import { ConvaiWidget } from "@convai/web-sdk";
 
-To enable video capabilities, set `enableVideo: true` in your configuration. This enables audio, video, and screen sharing.
+<ConvaiWidget
+  convaiClient={convaiClient}
+  showVideo={true}
+  showScreenShare={true}
+  defaultVoiceMode={true}
+/>;
+```
 
-**React:**
+#### Step 5: Subscribe to lifecycle events
 
 ```tsx
-import { useConvaiClient, ConvaiWidget } from "@convai/web-sdk";
+useEffect(() => {
+  const unsubError = convaiClient.on("error", (error) => {
+    console.error("Convai error:", error);
+  });
 
-function App() {
-  const convaiClient = useConvaiClient({
-    apiKey: "your-api-key",
-    characterId: "your-character-id",
-    enableVideo: true,
-    startWithVideoOn: false, // Camera off by default
+  const unsubState = convaiClient.on("stateChange", (state) => {
+    console.log("State:", state.agentState);
   });
 
-  return (
-    <ConvaiWidget
-      convaiClient={convaiClient}
-      showVideo={true}
-      showScreenShare={true}
-    />
-  );
-}
+  const unsubMessages = convaiClient.on("messagesChange", (messages) => {
+    console.log("Messages:", messages.length);
+  });
+
+  return () => {
+    unsubError();
+    unsubState();
+    unsubMessages();
+  };
+}, [convaiClient]);
+```
+
+#### Step 6: Clean up on unmount
+
+```tsx
+useEffect(() => {
+  return () => {
+    void convaiClient.disconnect().catch(() => undefined);
+  };
+}, [convaiClient]);
 ```
 
-**Vanilla:**
+### B) Vanilla TypeScript from scratch (widget + custom hooks)
 
-```typescript
+#### Step 1: Initialize client and widget
+
+```ts
 import { ConvaiClient, createConvaiWidget } from "@convai/web-sdk/vanilla";
 
 const client = new ConvaiClient({
-  apiKey: "your-api-key",
-  characterId: "your-character-id",
+  apiKey: "<YOUR_CONVAI_API_KEY>",
+  characterId: "<YOUR_CHARACTER_ID>",
+  endUserId: "<UNIQUE_END_USER_ID>",
   enableVideo: true,
   startWithVideoOn: false,
 });
@@ -233,624 +311,702 @@ const widget = createConvaiWidget(document.body, {
   convaiClient: client,
   showVideo: true,
   showScreenShare: true,
+  defaultVoiceMode: true,
+  onConnect: () => console.log("Connected"),
+  onDisconnect: () => console.log("Disconnected"),
+  onMessage: (message) => console.log("Message:", message),
 });
 ```
 
-**Manual Video Controls:**
+#### Step 2: Add explicit error listeners
 
-```typescript
-// Enable video camera
-await convaiClient.videoControls.enableVideo();
+```ts
+const unsubError = client.on("error", (error) => {
+  console.error("SDK error:", error);
+});
+```
 
-// Disable video camera
-await convaiClient.videoControls.disableVideo();
+#### Step 3: Add guarded send utility
 
-// Toggle video
-await convaiClient.videoControls.toggleVideo();
+```ts
+function safeSend(text: string) {
+  if (!text.trim()) return;
+  if (!client.state.isConnected) return;
+  if (!client.isBotReady) return;
+  client.sendUserTextMessage(text);
+}
+```
 
-// Check video state
-const isVideoEnabled = convaiClient.videoControls.isVideoEnabled;
+#### Step 4: Cleanup
 
-// Set video quality
-await convaiClient.videoControls.setVideoQuality("high"); // 'low' | 'medium' | 'high'
+```ts
+function destroy() {
+  unsubError();
+  widget.destroy();
+  void client.disconnect().catch(() => undefined);
+}
+```
 
-// Get available video devices
-const devices = await convaiClient.videoControls.getVideoDevices();
+### C) Custom UI (framework-agnostic)
 
-// Set specific video device
-await convaiClient.videoControls.setVideoDevice(deviceId);
-```
+If you are not using the built-in widget:
 
-**Screen Sharing:**
+- Use `ConvaiClient` from `@convai/web-sdk/core`
+- Use `AudioRenderer` from `@convai/web-sdk/vanilla` for remote audio playback
+- Render your own UI based on `stateChange`, `messagesChange`, and control manager events
 
-```typescript
-// Enable screen share
-await convaiClient.screenShareControls.enableScreenShare();
+```ts
+import { ConvaiClient } from "@convai/web-sdk/core";
+import { AudioRenderer } from "@convai/web-sdk/vanilla";
 
-// Enable screen share with audio
-await convaiClient.screenShareControls.enableScreenShareWithAudio();
+const client = new ConvaiClient({
+  apiKey: "<YOUR_CONVAI_API_KEY>",
+  characterId: "<YOUR_CHARACTER_ID>",
+});
 
-// Disable screen share
-await convaiClient.screenShareControls.disableScreenShare();
+await client.connect();
+const audioRenderer = new AudioRenderer(client.room);
 
-// Toggle screen share
-await convaiClient.screenShareControls.toggleScreenShare();
+// ... your custom UI logic
 
-// Check screen share state
-const isActive = convaiClient.screenShareControls.isScreenShareActive;
+audioRenderer.destroy();
+await client.disconnect();
 ```
 
-**Video State Monitoring:**
+## 6. Core Concepts and Lifecycle
 
-```typescript
-// React
-const { isVideoEnabled } = convaiClient;
+### Connection lifecycle
 
-// Core API (event-based)
-convaiClient.videoControls.on("videoStateChange", (state) => {
-  console.log("Video enabled:", state.isVideoEnabled);
-  console.log("Video hidden:", state.isVideoHidden);
-});
-```
+1. `connect()` starts room and transport setup.
+2. `state.isConnected` becomes true when room connection is established.
+3. `botReady` event indicates the character is ready for interaction.
+4. Messages stream through data events into `chatMessages`.
+5. Audio/video/screen-share are managed through dedicated control managers.
+6. `disconnect()` tears down the session.
 
-### Interruption
+### Activity lifecycle
 
-Interrupt the character's current response to allow the user to speak immediately.
+- `state.isThinking`: model is generating response
+- `state.isSpeaking`: model audio is currently speaking
+- `state.agentState`: combined high-level state (`disconnected | connected | listening | thinking | speaking`)
 
-**React:**
+### Widget lifecycle
 
-```tsx
-function ChatInterface() {
-  const convaiClient = useConvaiClient({
-    /* config */
-  });
+Both React and vanilla widgets:
 
-  const handleInterrupt = () => {
-    // Interrupt the bot's current response
-    convaiClient.sendInterruptMessage();
-  };
+- auto-connect on first user interaction
+- expose optional callbacks/events
+- need explicit cleanup on app teardown
 
-  return <button onClick={handleInterrupt}>Interrupt</button>;
-}
-```
+## 7. Configuration Reference (`ConvaiConfig`)
 
-**Vanilla:**
+| Field                                     | Type               | Required | Default              | Description                                                                         |
+| ----------------------------------------- | ------------------ | -------- | -------------------- | ----------------------------------------------------------------------------------- |
+| `apiKey`                                  | `string`           | Yes      | -                    | Convai API key.                                                                     |
+| `characterId`                             | `string`           | Yes      | -                    | Target character identifier.                                                        |
+| `endUserId`                               | `string`           | No       | `undefined`          | Stable end-user identity for memory/analytics continuity.                           |
+| `url`                                     | `string`           | No       | SDK internal default | Convai base URL. Set explicitly if your deployment requires a specific environment. |
+| `enableVideo`                             | `boolean`          | No       | `false`              | Enables video-capable connection type.                                              |
+| `startWithVideoOn`                        | `boolean`          | No       | `false`              | Auto-enable camera after connect.                                                   |
+| `startWithAudioOn`                        | `boolean`          | No       | `false`              | Auto-enable microphone after connect.                                               |
+| `ttsEnabled`                              | `boolean`          | No       | `true`               | Enables model text-to-speech output.                                                |
+| `enableLipsync`                           | `boolean`          | No       | `false`              | Requests blendshape payloads for facial animation.                                  |
+| `blendshapeConfig.format`                 | `"arkit" \| "mha"` | No       | `"mha"`              | Blendshape output format.                                                           |
+| `blendshapeConfig.frames_buffer_duration` | `number`           | No       | server-defined       | Buffering hint for audio/blendshape synchronization.                                |
+| `actionConfig`                            | object             | No       | `undefined`          | Action and scene-context metadata (actions, characters, objects, attention object). |
 
-```typescript
-const interruptButton = document.getElementById("interrupt-btn");
+## 8. Core API Reference (`ConvaiClient`)
 
-interruptButton.addEventListener("click", () => {
-  client.sendInterruptMessage();
-});
+Import:
+
+```ts
+import { ConvaiClient } from "@convai/web-sdk/core";
 ```
 
-**Voice Mode Interruption Pattern:**
+### Constructor
+
+```ts
+new ConvaiClient(config?: ConvaiConfig)
+```
+
+### Properties
+
+| Property                | Type                         | Description                                              |
+| ----------------------- | ---------------------------- | -------------------------------------------------------- |
+| `state`                 | `ConvaiClientState`          | Real-time connection/activity state.                     |
+| `connectionType`        | `"audio" \| "video" \| null` | Active transport mode.                                   |
+| `apiKey`                | `string \| null`             | Active API key.                                          |
+| `characterId`           | `string \| null`             | Active character ID.                                     |
+| `speakerId`             | `string \| null`             | Resolved speaker identity.                               |
+| `room`                  | `Room`                       | Internal LiveKit room instance.                          |
+| `chatMessages`          | `ChatMessage[]`              | Conversation message store.                              |
+| `userTranscription`     | `string`                     | Current non-final voice transcription text.              |
+| `characterSessionId`    | `string \| null`             | Server conversation session identifier.                  |
+| `isBotReady`            | `boolean`                    | Character readiness flag.                                |
+| `audioControls`         | `AudioControls`              | Microphone controls.                                     |
+| `videoControls`         | `VideoControls`              | Camera controls.                                         |
+| `screenShareControls`   | `ScreenShareControls`        | Screen sharing controls.                                 |
+| `latencyMonitor`        | `LatencyMonitor`             | Measurement manager used by the client for turn latency. |
+| `blendshapeQueue`       | `BlendshapeQueue`            | Buffer queue for lipsync frames.                         |
+| `conversationSessionId` | `number`                     | Incremental turn session ID used by conversation events. |
+
+### Methods
+
+| Method                 | Signature                                                           | Description                                                |
+| ---------------------- | ------------------------------------------------------------------- | ---------------------------------------------------------- |
+| `connect`              | `(config?: ConvaiConfig) => Promise<void>`                          | Connect using passed config or stored config.              |
+| `disconnect`           | `() => Promise<void>`                                               | Disconnect and release session resources.                  |
+| `reconnect`            | `() => Promise<void>`                                               | Disconnect then connect with stored config.                |
+| `resetSession`         | `() => void`                                                        | Reset character session and clear conversation history.    |
+| `sendUserTextMessage`  | `(text: string) => void`                                            | Send text message to character.                            |
+| `sendTriggerMessage`   | `(triggerName?: string, triggerMessage?: string) => void`           | Send trigger/action message.                               |
+| `sendInterruptMessage` | `() => void`                                                        | Interrupt current bot response.                            |
+| `updateTemplateKeys`   | `(templateKeys: Record<string, string>) => void`                    | Update runtime template variables.                         |
+| `updateDynamicInfo`    | `(dynamicInfo: { text: string }) => void`                           | Update dynamic context text.                               |
+| `toggleTts`            | `(enabled: boolean) => void`                                        | Enable/disable TTS for subsequent responses.               |
+| `on`                   | `(event: string, callback: (...args: any[]) => void) => () => void` | Subscribe to an event and receive an unsubscribe function. |
+| `off`                  | `(event: string, callback: (...args: any[]) => void) => void`       | Remove a specific listener.                                |
+
+### Common event names and payloads
+
+| Event                     | Payload                                 | Notes                                                                                                           |
+| ------------------------- | --------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
+| `stateChange`             | `ConvaiClientState`                     | Any state transition.                                                                                           |
+| `message`                 | `ChatMessage`                           | Last message whenever `messagesChange` updates.                                                                 |
+| `messagesChange`          | `ChatMessage[]`                         | Full message array update.                                                                                      |
+| `userTranscriptionChange` | `string`                                | Live user speech text updates.                                                                                  |
+| `speakingChange`          | `boolean`                               | Bot speaking started/stopped.                                                                                   |
+| `botReady`                | `void`                                  | Bot can now receive interaction.                                                                                |
+| `connect`                 | `void`                                  | Client connected.                                                                                               |
+| `disconnect`              | `void`                                  | Client disconnected.                                                                                            |
+| `error`                   | `unknown`                               | Error surfaced by client.                                                                                       |
+| `conversationStart`       | `{ sessionId, userMessage, timestamp }` | Conversation turn started.                                                                                      |
+| `turnEnd`                 | `{ sessionId, duration, timestamp }`    | Server signaled end of turn (bot stopped speaking). Same semantics as `BlendshapeQueue.hasReceivedEndSignal()`. |
+| `blendshapes`             | `unknown`                               | Incoming blendshape chunk payload.                                                                              |
+| `blendshapeStatsReceived` | `unknown`                               | End-of-turn blendshape stats marker.                                                                            |
+| `latencyMeasurement`      | `LatencyMeasurement`                    | Latency sample from monitor.                                                                                    |
+
+### Control manager APIs
+
+#### `audioControls`
+
+Properties:
+
+- `isAudioEnabled`
+- `isAudioMuted`
+- `audioLevel`
+
+Methods:
+
+- `enableAudio()`
+- `disableAudio()`
+- `muteAudio()`
+- `unmuteAudio()`
+- `toggleAudio()`
+- `setAudioDevice(deviceId)`
+- `getAudioDevices()`
+- `startAudioLevelMonitoring()`
+- `stopAudioLevelMonitoring()`
+- `on("audioStateChange", callback)`
+- `off("audioStateChange", callback)`
+
+#### `videoControls`
+
+Properties:
+
+- `isVideoEnabled`
+- `isVideoHidden`
+
+Methods:
+
+- `enableVideo()`
+- `disableVideo()`
+- `hideVideo()`
+- `showVideo()`
+- `toggleVideo()`
+- `setVideoDevice(deviceId)`
+- `getVideoDevices()`
+- `setVideoQuality("low" | "medium" | "high")`
+- `on("videoStateChange", callback)`
+- `off("videoStateChange", callback)`
+
+#### `screenShareControls`
+
+Properties:
+
+- `isScreenShareEnabled`
+- `isScreenShareActive`
+
+Methods:
+
+- `enableScreenShare()`
+- `disableScreenShare()`
+- `toggleScreenShare()`
+- `enableScreenShareWithAudio()`
+- `getScreenShareTracks()`
+- `on("screenShareStateChange", callback)`
+- `off("screenShareStateChange", callback)`
+
+### `latencyMonitor` API (via `client.latencyMonitor`)
+
+`latencyMonitor` is available on every client instance for instrumentation and diagnostics.
+
+Methods:
+
+- `enable()`
+- `disable()`
+- `startMeasurement(type, userMessage?)`
+- `endMeasurement()`
+- `cancelMeasurement()`
+- `getMeasurements()`
+- `getLatestMeasurement()`
+- `getStats()`
+- `clear()`
+- `getPendingMeasurement()`
+- `on("measurement", callback)`
+- `on("measurementsChange", callback)`
+- `on("enabledChange", callback)`
+
+Properties:
+
+- `enabled`
+- `hasPendingMeasurement`
+
+### Advanced core classes (`@convai/web-sdk/core`)
+
+These are exported for advanced and custom pipeline use-cases.
 
-When implementing voice mode, interrupt the bot when the user starts speaking:
+#### `BlendshapeQueue`
 
-```typescript
-// When user enters voice mode
-const enterVoiceMode = async () => {
-  // Interrupt any ongoing bot response
-  convaiClient.sendInterruptMessage();
+Buffer for lipsync frames. Use `isConversationEnded()` for definitive end-of-conversation: it returns true only when the server has sent `blendshape-turn-stats` and either all expected frames have been consumed or the queue is empty (handles dropped frames). Use `hasReceivedEndSignal()` when you only need to know that the server signaled end (e.g. to keep playing remaining frames).
 
-  // Unmute microphone
-  await convaiClient.audioControls.unmuteAudio();
-};
+Methods:
 
-// When user exits voice mode
-const exitVoiceMode = async () => {
-  // Interrupt any ongoing bot response
-  convaiClient.sendInterruptMessage();
+- `addChunk(blendshapes)`
+- `getFrames()`
+- `getFrame(index)`
+- `getFrameWithAlpha(index)`
+- `consumeFrames(count)`
+- `hasFrames()`
+- `isConversationActive()`
+- `isConversationEnded()` — true when server signaled end and playback is complete (all frames consumed or queue empty)
+- `hasReceivedEndSignal()` — true when server sent `blendshape-turn-stats` (does not check frame consumption)
+- `startConversation()`
+- `startBotSpeaking()`
+- `stopBotSpeaking()`
+- `isBotSpeaking()`
+- `endConversation(stats?)`
+- `interrupt()`
+- `getTurnStats()`
+- `getFramesConsumed()`
+- `getTimeLeftMs()`
+- `isAllFramesConsumed()`
+- `reset()`
+- `getFrameAtTime(elapsedTime)`
+- `getDebugInfo()`
 
-  // Mute microphone
-  await convaiClient.audioControls.muteAudio();
-};
-```
+Properties:
 
-### User Microphone Mute/Unmute
+- `length`
 
-Control the user's microphone input.
+#### `MessageHandler`
 
-**React:**
+Methods:
 
-```tsx
-function AudioControls() {
-  const convaiClient = useConvaiClient({
-    /* config */
-  });
+- `getBlendshapeQueue()`
+- `setLatencyMonitor(monitor)`
+- `getChatMessages()`
+- `getUserTranscription()`
+- `getIsBotResponding()`
+- `getIsSpeaking()`
+- `setRoom(room)`
+- `reset()`
+- inherited event APIs from `EventEmitter`:
+  - `on(event, callback)`
+  - `off(event, callback)`
 
-  const handleMute = async () => {
-    await convaiClient.audioControls.muteAudio();
-  };
+#### `EventEmitter`
 
-  const handleUnmute = async () => {
-    await convaiClient.audioControls.unmuteAudio();
-  };
-
-  const handleToggle = async () => {
-    await convaiClient.audioControls.toggleAudio();
-  };
+Methods:
 
-  return (
-    <div>
-      <button onClick={handleMute}>Mute</button>
-      <button onClick={handleUnmute}>Unmute</button>
-      <button onClick={handleToggle}>Toggle</button>
-      <p>Muted: {convaiClient.audioControls.isAudioMuted ? "Yes" : "No"}</p>
-    </div>
-  );
-}
-```
+- `on(event, callback)`
+- `off(event, callback)`
+- `emit(event, ...args)`
+- `removeAllListeners()`
+- `listenerCount(event)`
 
-**Vanilla:**
+## 9. Message Semantics and Turn Completion
 
-```typescript
-// Mute microphone
-await client.audioControls.muteAudio();
+### `ChatMessage` model
 
-// Unmute microphone
-await client.audioControls.unmuteAudio();
+`ChatMessage` includes:
 
-// Toggle mute state
-await client.audioControls.toggleAudio();
+- `id`
+- `type`
+- `content`
+- `timestamp`
+- `isFinal?`
 
-// Check mute state
-const isMuted = client.audioControls.isAudioMuted;
+Supported message `type` values include:
 
-// Enable audio (request permissions if needed)
-await client.audioControls.enableAudio();
+- `user`
+- `convai`
+- `emotion`
+- `behavior-tree`
+- `action`
+- `user-transcription`
+- `bot-llm-text`
+- `bot-emotion`
+- `user-llm-text`
+- `interrupt-bot`
 
-// Disable audio
-await client.audioControls.disableAudio();
-```
+### Important: current `isFinal` behavior
 
-**Audio Device Management:**
+In the current implementation, `isFinal` is used as an accumulation flag:
 
-```typescript
-// Get available audio devices
-const devices = await convaiClient.audioControls.getAudioDevices();
+- `isFinal: true` means the message is still in a mutable/streaming state
+- `isFinal: false` means the message has been finalized
 
-// Set specific audio device
-await convaiClient.audioControls.setAudioDevice(deviceId);
+This naming is counterintuitive. Treat `isFinal` as an internal streaming marker rather than a turn-completion signal.
 
-// Monitor audio level
-convaiClient.audioControls.startAudioLevelMonitoring();
+### Recommended way to detect response completion
 
-convaiClient.audioControls.on("audioLevelChange", (level) => {
-  console.log("Audio level:", level);
-  // level is a number between 0 and 1
-});
+Use events instead of `isFinal`:
 
-convaiClient.audioControls.stopAudioLevelMonitoring();
-```
+- `turnEnd` for the server turn-end signal (bot stopped speaking; same as `hasReceivedEndSignal()`)
+- `blendshapeStatsReceived` as additional completion marker when lipsync/animation output is enabled
 
-**Audio State Monitoring:**
+When driving lipsync from `BlendshapeQueue`, use `blendshapeQueue.isConversationEnded()` for definitive end-of-conversation. It returns true only when the server has signaled end and playback is complete (all expected frames consumed or queue empty). Call `blendshapeQueue.reset()` and your `onConversationEnded` when it becomes true. Use `hasReceivedEndSignal()` only when you need the raw server signal (e.g. to decide whether to keep playing remaining frames).
 
-```typescript
-// React
-const { isAudioMuted } = convaiClient;
+Example:
 
-// Core API (event-based)
-convaiClient.audioControls.on("audioStateChange", (state) => {
-  console.log("Audio enabled:", state.isAudioEnabled);
-  console.log("Audio muted:", state.isAudioMuted);
-  console.log("Audio level:", state.audioLevel);
-});
-```
+```ts
+type TurnCompletionOptions = {
+  expectBlendshapes: boolean;
+  onComplete: () => void;
+};
 
-### Character TTS Mute/Unmute
+function subscribeTurnCompletion(client: any, options: TurnCompletionOptions) {
+  let spokenDone = false;
+  let animationDone = !options.expectBlendshapes;
 
-Control whether the character's responses are spoken aloud (text-to-speech).
+  const invokeOnCompleteIfReady = () => {
+    if (spokenDone && animationDone) {
+      options.onComplete();
+    }
+  };
 
-**React:**
+  const unsubTurnEnd = client.on("turnEnd", () => {
+    spokenDone = true;
+    invokeOnCompleteIfReady();
+  });
 
-```tsx
-function TTSControls() {
-  const convaiClient = useConvaiClient({
-    /* config */
+  const unsubBlendshapeStats = client.on("blendshapeStatsReceived", () => {
+    animationDone = true;
+    invokeOnCompleteIfReady();
   });
 
-  const handleToggleTTS = (enabled: boolean) => {
-    convaiClient.toggleTts(enabled);
+  return () => {
+    unsubTurnEnd();
+    unsubBlendshapeStats();
   };
-
-  return (
-    <div>
-      <button onClick={() => handleToggleTTS(true)}>Enable TTS</button>
-      <button onClick={() => handleToggleTTS(false)}>Disable TTS</button>
-    </div>
-  );
 }
 ```
 
-**Vanilla:**
-
-```typescript
-// Enable text-to-speech (character will speak responses)
-client.toggleTts(true);
+When to use both signals: You only need to wait for both `turnEnd` and `blendshapeStatsReceived` when you use lipsync. Set `expectBlendshapes: false` when you do not use facial animation; then `animationDone` is effectively always true and completion runs as soon as `turnEnd` fires. Set `expectBlendshapes: true` when you drive lipsync from the queue; speech and blendshape data are separate pipelines and can finish in either order, so waiting for both ensures "turn complete" means both speech and animation are done before you run `onComplete`.
 
-// Disable text-to-speech (character will only send text, no audio)
-client.toggleTts(false);
-```
+## 10. React API Reference
 
-**Initial TTS Configuration:**
+### `useConvaiClient(config?)`
 
-```typescript
-// Set TTS state during connection
-const client = new ConvaiClient({
-  apiKey: "your-api-key",
-  characterId: "your-character-id",
-  ttsEnabled: true, // Enable TTS by default
-});
-
-// Or disable initially
-const client = new ConvaiClient({
-  apiKey: "your-api-key",
-  characterId: "your-character-id",
-  ttsEnabled: false, // Disable TTS
-});
-```
-
-### Voice Mode Implementation
-
-Voice mode allows users to speak instead of typing. The widget automatically handles voice mode, but you can implement it manually.
-
-**React - Manual Voice Mode:**
+Import:
 
 ```tsx
 import { useConvaiClient } from "@convai/web-sdk";
-import { useState, useEffect } from "react";
-
-function CustomChatInterface() {
-  const convaiClient = useConvaiClient({
-    /* config */
-  });
-  const [isVoiceMode, setIsVoiceMode] = useState(false);
-
-  const enterVoiceMode = async () => {
-    // Interrupt any ongoing bot response
-    convaiClient.sendInterruptMessage();
-
-    // Unmute microphone
-    await convaiClient.audioControls.unmuteAudio();
+```
 
-    setIsVoiceMode(true);
-  };
+Returns full `IConvaiClient` plus React-friendly reactive fields:
 
-  const exitVoiceMode = async () => {
-    // Interrupt any ongoing bot response
-    convaiClient.sendInterruptMessage();
+- `activity`
+- `chatMessages`
+- `isAudioMuted`
+- `isVideoEnabled`
+- `isScreenShareActive`
 
-    // Mute microphone
-    await convaiClient.audioControls.muteAudio();
+### `ConvaiWidget`
 
-    setIsVoiceMode(false);
-  };
+Import:
 
-  // Monitor user transcription for voice input
-  useEffect(() => {
-    const transcription = convaiClient.userTranscription;
-    if (transcription && isVoiceMode) {
-      // Display real-time transcription
-      console.log("User is saying:", transcription);
-    }
-  }, [convaiClient.userTranscription, isVoiceMode]);
-
-  return (
-    <div>
-      {isVoiceMode ? (
-        <div>
-          <p>Listening: {convaiClient.userTranscription}</p>
-          <button onClick={exitVoiceMode}>Stop Voice Mode</button>
-        </div>
-      ) : (
-        <button onClick={enterVoiceMode}>Start Voice Mode</button>
-      )}
-    </div>
-  );
-}
+```tsx
+import { ConvaiWidget } from "@convai/web-sdk";
 ```
 
-**Vanilla - Manual Voice Mode:**
+Props:
 
-```typescript
-let isVoiceMode = false;
+| Prop               | Type                                                                                                                  | Default  | Description                                                        |
+| ------------------ | --------------------------------------------------------------------------------------------------------------------- | -------- | ------------------------------------------------------------------ |
+| `convaiClient`     | `IConvaiClient & { activity?: string; isAudioMuted: boolean; isVideoEnabled: boolean; isScreenShareActive: boolean }` | required | Client instance returned by `useConvaiClient`.                     |
+| `showVideo`        | `boolean`                                                                                                             | `true`   | Shows video toggle in settings if connection type is video.        |
+| `showScreenShare`  | `boolean`                                                                                                             | `true`   | Shows screen-share toggle in settings if connection type is video. |
+| `defaultVoiceMode` | `boolean`                                                                                                             | `true`   | Opens in voice mode on first widget session.                       |
 
-const enterVoiceMode = async () => {
-  // Interrupt any ongoing bot response
-  client.sendInterruptMessage();
+### `useCharacterInfo(characterId?, apiKey?)`
 
-  // Unmute microphone
-  await client.audioControls.unmuteAudio();
+Returns:
 
-  isVoiceMode = true;
-  updateUI();
-};
+- `name`
+- `image`
+- `isLoading`
+- `error`
 
-const exitVoiceMode = async () => {
-  // Interrupt any ongoing bot response
-  client.sendInterruptMessage();
+### `useLocalCameraTrack()`
 
-  // Mute microphone
-  await client.audioControls.muteAudio();
+Returns a LiveKit `TrackReferenceOrPlaceholder` for local camera rendering in custom React video UIs.
 
-  isVoiceMode = false;
-  updateUI();
-};
+### React audio utility exports
 
-// Monitor user transcription
-client.on("userTranscriptionChange", (transcription) => {
-  if (isVoiceMode && transcription) {
-    // Display real-time transcription
-    document.getElementById("transcription").textContent = transcription;
-  }
-});
+- `AudioRenderer` from LiveKit React components
+- `AudioContext` from LiveKit React components
 
-function updateUI() {
-  const voiceButton = document.getElementById("voice-btn");
-  const transcriptionDiv = document.getElementById("transcription");
+## 11. Vanilla API Reference
 
-  if (isVoiceMode) {
-    voiceButton.textContent = "Stop Voice Mode";
-    transcriptionDiv.style.display = "block";
-  } else {
-    voiceButton.textContent = "Start Voice Mode";
-    transcriptionDiv.style.display = "none";
-  }
-}
-```
+### `createConvaiWidget(container, options)`
 
-**Voice Mode with State Monitoring:**
-
-```typescript
-// Monitor agent state to handle voice mode transitions
-convaiClient.on("stateChange", (state) => {
-  if (isVoiceMode) {
-    switch (state.agentState) {
-      case "listening":
-        // User can speak
-        console.log("Bot is listening");
-        break;
-      case "thinking":
-        // Bot is processing
-        console.log("Bot is thinking");
-        break;
-      case "speaking":
-        // Bot is responding
-        console.log("Bot is speaking");
-        // Optionally interrupt if user wants to speak
-        break;
-    }
-  }
-});
+```ts
+import { createConvaiWidget } from "@convai/web-sdk/vanilla";
 ```
 
-### Connection Management
+Creates and mounts a complete floating chat widget.
 
-**Connect:**
+#### `VanillaWidgetOptions`
 
-```typescript
-// React - config passed to hook
-const convaiClient = useConvaiClient({
-  apiKey: "your-api-key",
-  characterId: "your-character-id",
-});
+| Field              | Type                             | Required | Default     | Description                                        |
+| ------------------ | -------------------------------- | -------- | ----------- | -------------------------------------------------- |
+| `convaiClient`     | `IConvaiClient`                  | No\*     | -           | Existing client instance.                          |
+| `apiKey`           | `string`                         | No\*     | -           | Used only when `convaiClient` is not provided.     |
+| `characterId`      | `string`                         | No\*     | -           | Used only when `convaiClient` is not provided.     |
+| `enableVideo`      | `boolean`                        | No       | `false`     | Used for auto-created client only.                 |
+| `startWithVideoOn` | `boolean`                        | No       | `false`     | Used for auto-created client only.                 |
+| `enableLipsync`    | `boolean`                        | No       | `false`     | Used for auto-created client only.                 |
+| `blendshapeConfig` | object                           | No       | `undefined` | Used for auto-created client only.                 |
+| `showVideo`        | `boolean`                        | No       | `true`      | Show video toggle in settings.                     |
+| `showScreenShare`  | `boolean`                        | No       | `true`      | Show screen-share toggle in settings.              |
+| `defaultVoiceMode` | `boolean`                        | No       | `true`      | Start in voice mode when opened.                   |
+| `onConnect`        | `() => void`                     | No       | `undefined` | Called when widget client connects.                |
+| `onDisconnect`     | `() => void`                     | No       | `undefined` | Called when widget client disconnects.             |
+| `onMessage`        | `(message: ChatMessage) => void` | No       | `undefined` | Called on each message change with latest message. |
 
-// Or connect manually
-await convaiClient.connect({
-  apiKey: "your-api-key",
-  characterId: "your-character-id",
-});
+\* You must provide either `convaiClient` OR both `apiKey` and `characterId`.
 
-// Vanilla
-const client = new ConvaiClient();
-await client.connect({
-  apiKey: "your-api-key",
-  characterId: "your-character-id",
-});
-```
+#### Return type: `VanillaWidget`
 
-**Disconnect:**
+- `element`: root widget element
+- `client`: resolved client instance
+- `destroy()`: unmount and cleanup
+- `update?`: optional future extension field
 
-```typescript
-await convaiClient.disconnect();
-```
+### `destroyConvaiWidget(widget)`
 
-**Reconnect:**
+Convenience wrapper that calls `widget.destroy()`.
 
-```typescript
-await convaiClient.reconnect();
-```
+### `AudioRenderer` (vanilla)
 
-**Reset Session:**
+`AudioRenderer` listens to LiveKit room track subscriptions and auto-attaches remote audio tracks to hidden `audio` elements for playback. Use one renderer instance per active room session and destroy it during cleanup.
 
-```typescript
-// Clear conversation history and start new session
-convaiClient.resetSession();
-```
+## 12. Audio Integration Best Practices (Vanilla TypeScript)
 
-**Connection State:**
+This section provides the recommended integration for stable audio playback.
 
-```typescript
-// React
-const { state } = convaiClient;
-console.log("Connected:", state.isConnected);
-console.log("Connecting:", state.isConnecting);
-console.log("Agent state:", state.agentState); // 'disconnected' | 'connected' | 'listening' | 'thinking' | 'speaking'
+### Recommended reference implementation
 
-// Core API (event-based)
-convaiClient.on("stateChange", (state) => {
-  console.log("State changed:", state);
-});
+```ts
+import { ConvaiClient } from "@convai/web-sdk/core";
+import { AudioRenderer } from "@convai/web-sdk/vanilla";
+
+class ConvaiAudioSession {
+  private client: ConvaiClient;
+  private audioRenderer: AudioRenderer | null = null;
+  private audioContext: AudioContext | null = null;
+
+  constructor() {
+    this.client = new ConvaiClient({
+      apiKey: "<YOUR_CONVAI_API_KEY>",
+      characterId: "<YOUR_CHARACTER_ID>",
+      ttsEnabled: true,
+    });
+  }
 
-convaiClient.on("connect", () => {
-  console.log("Connected");
-});
+  async connectFromUserGesture(): Promise<void> {
+    await this.client.connect();
 
-convaiClient.on("disconnect", () => {
-  console.log("Disconnected");
-});
-```
+    // Required for remote audio playback wiring.
+    this.audioRenderer = new AudioRenderer(this.client.room);
+
+    // Optional: if your app performs WebAudio analysis/effects.
+    if (!this.audioContext) {
+      this.audioContext = new AudioContext();
+    }
+    if (this.audioContext.state === "suspended") {
+      await this.audioContext.resume();
+    }
+  }
 
-### Messaging
+  async disconnect(): Promise<void> {
+    if (this.audioRenderer) {
+      this.audioRenderer.destroy();
+      this.audioRenderer = null;
+    }
 
-**Send Text Message:**
+    await this.client.disconnect();
 
-```typescript
-convaiClient.sendUserTextMessage("Hello, how are you?");
+    if (this.audioContext && this.audioContext.state !== "closed") {
+      await this.audioContext.close();
+      this.audioContext = null;
+    }
+  }
+}
 ```
 
-**Send Trigger Message:**
+### AudioContext guidance
 
-```typescript
-// Trigger specific character action
-convaiClient.sendTriggerMessage("greet", "User entered the room");
+- Create/resume `AudioContext` only after user interaction in browsers that enforce autoplay policy.
+- If you are not processing audio with WebAudio, you do not need a custom `AudioContext`; `AudioRenderer` is enough for playback.
+- Always close your custom `AudioContext` in teardown.
 
-// Trigger without message
-convaiClient.sendTriggerMessage("wave");
-```
+### Lifecycle and cleanup order
 
-**Update Context:**
+Recommended shutdown order:
 
-```typescript
-// Update template keys (e.g., user name, location)
-convaiClient.updateTemplateKeys({
-  user_name: "John",
-  location: "New York",
-});
+1. Stop UI input loops/listeners
+2. Destroy `AudioRenderer`
+3. Disconnect `ConvaiClient`
+4. Close custom `AudioContext` (if created)
 
-// Update dynamic information
-convaiClient.updateDynamicInfo({
-  text: "User is currently browsing the products page",
-});
-```
+### Common failure modes and fixes
 
-**Message History:**
+| Symptom                 | Likely cause                              | Recommended action                                                                    |
+| ----------------------- | ----------------------------------------- | ------------------------------------------------------------------------------------- |
+| No AI audio output      | `AudioRenderer` not created               | Instantiate `new AudioRenderer(client.room)` immediately after successful connect.    |
+| No AI audio output      | Browser autoplay restriction              | Trigger connect/playback from a user click, and resume `AudioContext` if suspended.   |
+| No AI audio output      | TTS disabled                              | Ensure `ttsEnabled` is true for sessions that need speech output.                     |
+| Intermittent playback   | Multiple renderers or stale room instance | Use one renderer per session and always destroy old renderer before reconnecting.     |
+| Works once, then silent | Incomplete cleanup on previous session    | Destroy renderer and disconnect client on teardown; avoid reusing invalid room state. |
+| Random muted behavior   | App-side muting of remote tracks          | Verify no custom code is muting remote publications or media elements.                |
 
-```typescript
-// React
-const { chatMessages } = convaiClient;
+## 13. Error Handling and Reliability Patterns
 
-// Core API (event-based)
-convaiClient.on("message", (message: ChatMessage) => {
-  console.log("New message:", message.content);
-  console.log("Message type:", message.type);
-});
+### Pattern 1: Centralized SDK error handling
 
-convaiClient.on("messagesChange", (messages: ChatMessage[]) => {
-  console.log("All messages:", messages);
+```ts
+const unsubError = client.on("error", (error) => {
+  console.error("Convai SDK error:", error);
+  // Optional: route to telemetry/monitoring
 });
 ```
 
-**Message Types:**
-
-```typescript
-type ChatMessageType =
-  | "user" // User's sent message
-  | "convai" // Character's response
-  | "user-transcription" // Real-time speech-to-text from user
-  | "bot-llm-text" // Character's LLM-generated text
-  | "emotion" // Character's emotional state
-  | "behavior-tree" // Behavior tree response
-  | "action" // Action execution
-  | "bot-emotion" // Bot emotional response
-  | "user-llm-text" // User text processed by LLM
-  | "interrupt-bot"; // Interrupt message
+### Pattern 2: Retry connect with exponential backoff
+
+```ts
+async function connectWithRetry(
+  client: any,
+  attempts = 3,
+  initialDelayMs = 500,
+): Promise<void> {
+  let delay = initialDelayMs;
+
+  for (let i = 1; i <= attempts; i++) {
+    try {
+      await client.connect();
+      return;
+    } catch (error) {
+      if (i === attempts) throw error;
+      await new Promise((resolve) => setTimeout(resolve, delay));
+      delay *= 2;
+    }
+  }
+}
 ```
 
-### State Monitoring
+### Pattern 3: Safe send guard
 
-**Agent State:**
+```ts
+function safeSendText(client: any, text: string) {
+  if (!text.trim()) return;
+  if (!client.state.isConnected) return;
+  if (!client.isBotReady) return;
+  client.sendUserTextMessage(text);
+}
+```
 
-```typescript
-// React
-const { state } = convaiClient;
+### Pattern 4: Protect media control calls
 
-// Check specific states
-if (state.isListening) {
-  console.log("Bot is listening");
+```ts
+async function safeToggleMic(client: any) {
+  try {
+    await client.audioControls.toggleAudio();
+  } catch (error) {
+    console.error("Failed to toggle microphone:", error);
+  }
 }
+```
 
-if (state.isThinking) {
-  console.log("Bot is thinking");
-}
+### Pattern 5: Always unsubscribe listeners
 
-if (state.isSpeaking) {
-  console.log("Bot is speaking");
-}
+```ts
+const unsubscribers = [
+  client.on("stateChange", () => {}),
+  client.on("messagesChange", () => {}),
+];
 
-// Combined state
-console.log(state.agentState); // 'disconnected' | 'connected' | 'listening' | 'thinking' | 'speaking'
+function cleanupListeners() {
+  for (const unsub of unsubscribers) unsub();
+}
 ```
 
-**User Transcription:**
+## 14. Troubleshooting
 
-```typescript
-// React
-const { userTranscription } = convaiClient;
+### Connection issues
 
-// Core API (event-based)
-convaiClient.on("userTranscriptionChange", (transcription: string) => {
-  console.log("User is saying:", transcription);
-});
-```
+- Verify API key and character ID are valid.
+- Ensure requests are allowed from your browser origin.
+- Set `url` explicitly if your environment does not use the SDK default endpoint.
+- Listen to `error` and inspect failed network calls in browser devtools.
 
-**Bot Ready State:**
+### `connect()` succeeds but bot never responds
 
-```typescript
-// React
-const { isBotReady } = convaiClient;
+- Wait for `botReady` before sending messages.
+- Confirm `ttsEnabled` and message flow are configured as expected.
+- Verify `messagesChange` receives content.
 
-// Core API (event-based)
-convaiClient.on("botReady", () => {
-  console.log("Bot is ready to receive messages");
-});
-```
+### Audio does not play
 
-## Getting Convai Credentials
+- Ensure an `AudioRenderer` is active for the connected room (vanilla custom UI).
+- Ensure playback starts from a user gesture path to satisfy autoplay policies.
+- Confirm no custom muting code is muting remote tracks.
 
-1. Visit [convai.com](https://convai.com) and create an account
-2. Navigate to your dashboard
-3. Create a new character or use an existing one
-4. Copy your **API Key** from the dashboard
-5. Copy your **Character ID** from the character details
+### Microphone does not capture user voice
 
-## Import Paths
+- Ensure app is served over secure context.
+- Verify browser microphone permission.
+- Handle permission errors from `audioControls.enableAudio()/unmuteAudio()`.
 
-```typescript
-// Default: React version (backward compatible)
-import { useConvaiClient, ConvaiWidget } from "@convai/web-sdk";
+### Video or screen share controls fail
 
-// Explicit React import
-import { useConvaiClient, ConvaiWidget } from "@convai/web-sdk/react";
+- Use `enableVideo: true` in config when you need video capabilities.
+- Screen share can be blocked by browser policy or user denial.
+- Wrap calls in `try/catch` and provide fallback UX.
 
-// Vanilla JS/TS
-import { ConvaiClient, createConvaiWidget } from "@convai/web-sdk/vanilla";
+### Lipsync appears out of sync or shape
 
-// Core only (no UI, framework agnostic)
-import { ConvaiClient } from "@convai/web-sdk/core";
-```
+- Validate blendshape format (`arkit` vs `mha`) matches your rig expectations.
+- Tune `frames_buffer_duration` so you atleast have some duration of blendshapes before the audio starts playing.
+- Align lipsync start and stop with the queue: start playback when the bot starts speaking (`isBotSpeaking()` true) and treat the turn as finished when `blendshapeQueue.isConversationEnded()` is true before resetting.
+- Drive blendshape application from a single loop (e.g. `requestAnimationFrame`) and advance frame index at 60fps so mouth movement stays in sync with audio.
 
-## TypeScript Support
-
-All exports are fully typed:
-
-```typescript
-import type {
-  ConvaiClient,
-  ConvaiConfig,
-  ConvaiClientState,
-  ChatMessage,
-  AudioControls,
-  VideoControls,
-  ScreenShareControls,
-  IConvaiClient,
-} from "@convai/web-sdk";
-```
+## 16. Examples
 
-## Support
+Repository examples:
 
-- [Convai Forum](https://forum.convai.com)
-- [API Reference](./API_REFERENCE.md)
-- [Convai Website](https://convai.com)
+- `examples/react-three-fiber`
+- `examples/three-vanilla`
+- `examples/README.md` for example-level setup notes