@spatialwalk/avatarkit 1.0.0-beta.2 → 1.0.0-beta.21

package/README.md

@@ -1,25 +1,26 @@
 # SPAvatarKit SDK
 
-基于 3D Gaussian Splatting 的实时虚拟人物头像渲染 SDK，支持音频驱动的动画渲染和高质量 3D 渲染。
+Real-time virtual avatar rendering SDK based on 3D Gaussian Splatting, supporting audio-driven animation rendering and high-quality 3D rendering.
 
-## 🚀 特性
+## 🚀 Features
 
-- **3D Gaussian Splatting 渲染** - 基于最新的点云渲染技术，提供高质量的 3D 虚拟人物
-- **音频驱动的实时动画渲染** - 用户提供音频数据，SDK 负责接收动画数据并渲染
-- **WebGPU/WebGL 双渲染后端** - 自动选择最佳渲染后端，确保兼容性
-- **WASM 高性能计算** - 使用 C++ 编译的 WebAssembly 模块进行几何计算
-- **TypeScript 支持** - 完整的类型定义和智能提示
-- **模块化架构** - 清晰的组件分离，易于集成和扩展
+- **3D Gaussian Splatting Rendering** - Based on the latest point cloud rendering technology, providing high-quality 3D virtual avatars
+- **Audio-Driven Real-Time Animation Rendering** - Users provide audio data, SDK handles receiving animation data and rendering
+- **Multi-Character Support** - Support multiple avatar instances simultaneously, each with independent state and rendering
+- **WebGPU/WebGL Dual Rendering Backend** - Automatically selects the best rendering backend for compatibility
+- **WASM High-Performance Computing** - Uses C++ compiled WebAssembly modules for geometric calculations
+- **TypeScript Support** - Complete type definitions and IntelliSense
+- **Modular Architecture** - Clear component separation, easy to integrate and extend
 
-## 📦 安装
+## 📦 Installation
 
 ```bash
 npm install @spatialwalk/avatarkit
 ```
 
-## 🎯 快速开始
+## 🎯 Quick Start
 
-### 基础使用
+### Basic Usage
 
 ```typescript
 import { 
@@ -30,190 +31,469 @@ import {
   Environment 
 } from '@spatialwalk/avatarkit'
 
-// 1. 初始化 SDK
+// 1. Initialize SDK
+import { DrivingServiceMode } from '@spatialwalk/avatarkit'
+
 const configuration: Configuration = {
   environment: Environment.test,
+  drivingServiceMode: DrivingServiceMode.sdk, // Optional, 'sdk' is default
+  // - DrivingServiceMode.sdk: SDK mode - SDK handles WebSocket communication
+  // - DrivingServiceMode.host: Host mode - Host app provides audio and animation data
 }
 
 await AvatarKit.initialize('your-app-id', configuration)
 
-// 设置 sessionToken（如果需要，单独调用）
+// Set sessionToken (if needed, call separately)
 // AvatarKit.setSessionToken('your-session-token')
 
-// 2. 加载角色
-const avatarManager = new AvatarManager()
+// 2. Load character
+const avatarManager = AvatarManager.shared
 const avatar = await avatarManager.load('character-id', (progress) => {
   console.log(`Loading progress: ${progress.progress}%`)
 })
 
-// 3. 创建视图（自动创建 Canvas 和 AvatarController）
+// 3. Create view (automatically creates Canvas and AvatarController)
+// The playback mode is determined by drivingServiceMode in AvatarKit configuration
+// - DrivingServiceMode.sdk: SDK mode - SDK handles WebSocket communication
+// - DrivingServiceMode.host: Host mode - Host app provides audio and animation data
 const container = document.getElementById('avatar-container')
 const avatarView = new AvatarView(avatar, container)
 
-// 4. 启动实时通信
+// 4. Start real-time communication (SDK mode only)
 await avatarView.avatarController.start()
 
-// 5. 发送音频数据
-// 如果音频是 Uint8Array，可以使用 slice().buffer 转换为 ArrayBuffer
-const audioUint8 = new Uint8Array(1024) // 示例：音频数据
-const audioData = audioUint8.slice().buffer // 简化的转换方式，适用于 ArrayBuffer 和 SharedArrayBuffer
-avatarView.avatarController.send(audioData, false) // 发送音频数据，积累到一定量后会自动开始播放
-avatarView.avatarController.send(audioData, true) // end=true 表示立即返回动画数据，不再积累
+// 5. Send audio data (SDK mode)
+// ⚠️ Important: Audio must be 16kHz mono PCM16 format
+// If audio is Uint8Array, you can use slice().buffer to convert to ArrayBuffer
+const audioUint8 = new Uint8Array(1024) // Example: 16kHz PCM16 audio data (512 samples = 1024 bytes)
+const audioData = audioUint8.slice().buffer // Simplified conversion, works for ArrayBuffer and SharedArrayBuffer
+avatarView.avatarController.send(audioData, false) // Send audio data, will automatically start playing after accumulating enough data
+avatarView.avatarController.send(audioData, true) // end=true marks the end of current conversation round
 ```
 
-### 完整示例
+### Host Mode Example
+
+```typescript
+import { AvatarPlaybackMode } from '@spatialwalk/avatarkit'
+
+// 1-3. Same as SDK mode (initialize SDK, load character)
+
+// 3. Create view with Host mode
+const container = document.getElementById('avatar-container')
+const avatarView = new AvatarView(avatar, container)
+
+// 4. Host Mode Workflow:
+// ⚠️ IMPORTANT: In Host mode, you MUST send audio data FIRST to get a conversationId,
+//    then use that conversationId to send animation data.
+//    Animation data with mismatched conversationId will be discarded.
+
+// Option A: Playback existing audio and animation data (replay mode)
+const initialAudioChunks = [{ data: audioData1, isLast: false }, { data: audioData2, isLast: false }]
+const initialKeyframes = animationData1 // Animation keyframes from your service
+// Step 1: Send audio first to get conversationId
+const conversationId = await avatarView.avatarController.playback(initialAudioChunks, initialKeyframes)
+
+// Option B: Stream new audio and animation data (start a new session directly)
+// Step 1: Send audio data first to get conversationId (automatically generates conversationId if starting new session)
+const currentConversationId = avatarView.avatarController.yieldAudioData(audioData3, false)
+// Step 2: Use the conversationId to send animation data (mismatched conversationId will be discarded)
+avatarView.avatarController.yieldFramesData(animationData2, currentConversationId || conversationId)
+// Note: To start playback, you need to call playback() with the accumulated data, or ensure enough audio data is sent
+```
+
+### Complete Examples
+
+Check the example code in the GitHub repository for complete usage flows for both modes.
+
+**Example Project:** [AvatarKit-Web-Demo](https://github.com/spatialwalk/AvatarKit-Web-Demo)
+
+This repository contains complete examples for Vanilla JS, Vue 3, and React, demonstrating:
+- SDK mode: Real-time audio input with automatic animation data reception
+- Host mode: Custom data sources with manual audio/animation data management
+
+## 🏗️ Architecture Overview
+
+### Three-Layer Architecture
 
-查看 GitHub 仓库中的示例代码了解完整的使用流程。
+The SDK uses a three-layer architecture for clear separation of concerns:
 
-**示例项目：** [Avatarkit-web-demo](https://github.com/spatialwalk/Avatarkit-web-demo)
+1. **Rendering Layer (AvatarView)** - Responsible for 3D rendering only
+2. **Playback Layer (AvatarController)** - Manages audio/animation synchronization and playback
+3. **Network Layer** - Handles WebSocket communication (only in SDK mode, internal implementation)
 
-该仓库包含 Vanilla JS、Vue 3 和 React 的完整示例，展示了如何在不同框架中集成和使用 SPAvatarKit SDK。
+### Core Components
 
-## 🏗️ 架构概览
+- **AvatarKit** - SDK initialization and management
+- **AvatarManager** - Character resource loading and management
+- **AvatarView** - 3D rendering view (rendering layer)
+- **AvatarController** - Audio/animation playback controller (playback layer)
 
-### 核心组件
+### Playback Modes
 
-- **AvatarKit** - SDK 初始化和管理
-- **AvatarManager** - 角色资源加载和管理
-- **AvatarView** - 3D 渲染视图（内部包含 AvatarController）
-- **AvatarController** - 实时通信和数据处理
-- **AvatarCoreAdapter** - WASM 模块适配器
+The SDK supports two playback modes, configured in `AvatarKit.initialize()`:
 
-### 数据流
+#### 1. SDK Mode (Default)
+- Configured via `drivingServiceMode: DrivingServiceMode.sdk` in `AvatarKit.initialize()`
+- SDK handles WebSocket communication automatically
+- Send audio data via `AvatarController.send()`
+- SDK receives animation data from backend and synchronizes playback
+- Best for: Real-time audio input scenarios
+
+#### 2. Host Mode
+- Configured via `drivingServiceMode: DrivingServiceMode.host` in `AvatarKit.initialize()`
+- Host application manages its own network/data fetching
+- Host application provides both audio and animation data
+- SDK only handles synchronized playback
+- Best for: Custom data sources, pre-recorded content, or custom network implementations
+
+**Note:** The playback mode is determined by `drivingServiceMode` in `AvatarKit.initialize()` configuration.
+
+### Fallback Mechanism
+
+The SDK includes a fallback mechanism to ensure audio playback continues even when animation data is unavailable:
+
+- **SDK Mode Connection Failure**: If WebSocket connection fails to establish within 15 seconds, the SDK automatically enters fallback mode. In this mode, audio data can still be sent and will play normally, even though no animation data will be received from the server. This ensures that audio playback is not interrupted even when the service connection fails.
+- **SDK Mode Server Error**: If the server returns an error after connection is established, the SDK automatically enters audio-only mode for that session and continues playing audio independently.
+- **Host Mode**: If empty animation data is provided (empty array or undefined), the SDK automatically enters audio-only mode.
+- Once in audio-only mode, any subsequent animation data for that session will be ignored, and only audio will continue playing.
+- The fallback mode is interruptible, just like normal playback mode.
+- Connection state callbacks (`onConnectionState`) will notify you when connection fails or times out, allowing you to handle the fallback state appropriately.
+
+### Data Flow
+
+#### SDK Mode Flow
 
 ```
-用户音频输入（16kHz mono PCM） → AvatarController → WebSocket → 后台处理
-                                              ↓
-后台返回动画数据（FLAME 关键帧） → AvatarController → AnimationPlayer
-                                              ↓
-FLAME 参数 → AvatarCore.computeFrameFlatFromParams() → Splat 数据
-                                              ↓
-Splat 数据 → RenderSystem → WebGPU/WebGL → Canvas 渲染
+User audio input (16kHz mono PCM16) 
+    ↓
+AvatarController.send() 
+    ↓
+WebSocket → Backend processing
+    ↓
+Backend returns animation data (FLAME keyframes) 
+    ↓
+AvatarController → AnimationPlayer
+    ↓
+FLAME parameters → AvatarCore.computeFrameFlatFromParams() → Splat data
+    ↓
+AvatarController (playback loop) → AvatarView.renderRealtimeFrame()
+    ↓
+RenderSystem → WebGPU/WebGL → Canvas rendering
 ```
 
-**注意：** 用户需要自己提供音频数据（16kHz mono PCM），SDK 负责接收动画数据并渲染。
+#### Host Mode Flow
+
+```
+External data source (audio + animation)
+    ↓
+Step 1: Send audio data FIRST to get conversationId
+    ↓
+AvatarController.playback(initialAudio, initialKeyframes) // Returns conversationId
+    OR
+AvatarController.yieldAudioData(audioChunk) // Returns conversationId
+    ↓
+Step 2: Use conversationId to send animation data
+    ↓
+AvatarController.yieldFramesData(keyframes, conversationId) // Requires conversationId
+    ↓
+AvatarController → AnimationPlayer (synchronized playback)
+    ↓
+FLAME parameters → AvatarCore.computeFrameFlatFromParams() → Splat data
+    ↓
+AvatarController (playback loop) → AvatarView.renderRealtimeFrame()
+    ↓
+RenderSystem → WebGPU/WebGL → Canvas rendering
+```
+
+**Note:** 
+- In SDK mode, users provide audio data, SDK handles network communication and animation data reception
+- In Host mode, users provide both audio and animation data, SDK handles synchronized playback only
+
+### Audio Format Requirements
+
+**⚠️ Important:** The SDK requires audio data to be in **16kHz mono PCM16** format:
+
+- **Sample Rate**: 16kHz (16000 Hz) - This is a backend requirement
+- **Channels**: Mono (single channel)
+- **Format**: PCM16 (16-bit signed integer, little-endian)
+- **Byte Order**: Little-endian
+
+**Audio Data Format:**
+- Each sample is 2 bytes (16-bit)
+- Audio data should be provided as `ArrayBuffer` or `Uint8Array`
+- For example: 1 second of audio = 16000 samples × 2 bytes = 32000 bytes
 
-## 📚 API 参考
+**Resampling:**
+- If your audio source is at a different sample rate (e.g., 24kHz, 48kHz), you must resample it to 16kHz before sending to the SDK
+- For high-quality resampling, we recommend using Web Audio API's `OfflineAudioContext` with anti-aliasing filtering
+- See example projects for resampling implementation
+
+## 📚 API Reference
 
 ### AvatarKit
 
-SDK 的核心管理类，负责初始化和全局配置。
+The core management class of the SDK, responsible for initialization and global configuration.
 
 ```typescript
-// 初始化 SDK
+// Initialize SDK
 await AvatarKit.initialize(appId: string, configuration: Configuration)
 
-// 检查初始化状态
+// Check initialization status
 const isInitialized = AvatarKit.isInitialized
 
-// 清理资源（不再使用时必须调用）
+// Get initialized app ID
+const appId = AvatarKit.appId
+
+// Get configuration
+const config = AvatarKit.configuration
+
+// Set sessionToken (if needed, call separately)
+AvatarKit.setSessionToken('your-session-token')
+
+// Set userId (optional, for telemetry)
+AvatarKit.setUserId('user-id')
+
+// Get sessionToken
+const sessionToken = AvatarKit.sessionToken
+
+// Get userId
+const userId = AvatarKit.userId
+
+// Get SDK version
+const version = AvatarKit.version
+
+// Cleanup resources (must be called when no longer in use)
 AvatarKit.cleanup()
 ```
 
 ### AvatarManager
 
-角色资源管理器，负责下载、缓存和加载角色数据。
+Character resource manager, responsible for downloading, caching, and loading character data. Use the singleton instance via `AvatarManager.shared`.
 
 ```typescript
-const manager = new AvatarManager()
+// Get singleton instance
+const manager = AvatarManager.shared
 
-// 加载角色
+// Load character
 const avatar = await manager.load(
   characterId: string, 
   onProgress?: (progress: LoadProgressInfo) => void
 )
 
-// 清理缓存
+// Clear cache
 manager.clearCache()
 ```
 
 ### AvatarView
 
-3D 渲染视图，内部自动创建和管理 AvatarController。
+3D rendering view (rendering layer), responsible for 3D rendering only. Internally automatically creates and manages `AvatarController`.
 
-**⚠️ 重要限制：** 目前 SDK 只支持同时存在一个 AvatarView 实例。如果需要切换角色，必须先调用 `dispose()` 方法清理当前的 AvatarView，然后再创建新的实例。
+**Playback Mode Configuration:**
+- The playback mode is fixed when creating `AvatarView` and persists throughout its lifecycle
+- Cannot be changed after creation
 
 ```typescript
-// 创建视图（Canvas 会自动添加到容器中）
-const avatarView = new AvatarView(avatar: Avatar, container?: HTMLElement)
+import { AvatarPlaybackMode } from '@spatialwalk/avatarkit'
 
-// 获取 Canvas 元素
-const canvas = avatarView.getCanvas()
+// Create view (Canvas is automatically added to container)
+// Create view (playback mode is determined by drivingServiceMode in AvatarKit configuration)
+const container = document.getElementById('avatar-container')
+const avatarView = new AvatarView(avatar, container)
 
-// 设置背景
-avatarView.setBackgroundImage('path/to/image.jpg')
-avatarView.setBackgroundOpaque(true)
+// Get playback mode
+const mode = avatarView.playbackMode // 'network' | 'external'
 
-// 更新相机配置
-avatarView.updateCameraConfig(cameraConfig: CameraConfig)
+// Wait for first frame to render
+await avatarView.ready // Promise that resolves when the first frame is rendered
 
-// 清理资源（切换角色前必须调用）
+// Cleanup resources (must be called before switching characters)
 avatarView.dispose()
 ```
 
-**切换角色示例：**
+**Character Switching Example:**
 
 ```typescript
-// 切换角色前，必须先清理旧的 AvatarView
+// To switch characters, simply dispose the old view and create a new one
 if (currentAvatarView) {
   currentAvatarView.dispose()
-  currentAvatarView = null
 }
 
-// 加载新角色
+// Load new character
 const newAvatar = await avatarManager.load('new-character-id')
 
-// 创建新的 AvatarView
+// Create new AvatarView
 currentAvatarView = new AvatarView(newAvatar, container)
-await currentAvatarView.avatarController.start()
+
+// SDK mode: start connection
+if (currentAvatarView.playbackMode === AvatarPlaybackMode.network) {
+  await currentAvatarView.controller.start()
+}
 ```
 
 ### AvatarController
 
-实时通信控制器，处理 WebSocket 连接和动画数据。
+Audio/animation playback controller (playback layer), manages synchronized playback of audio and animation. Automatically handles WebSocket communication in SDK mode.
+
+**Two Usage Patterns:**
+
+#### SDK Mode Methods
 
 ```typescript
-// 启动连接
+// Start WebSocket service
 await avatarView.avatarController.start()
 
-// 发送音频数据
-avatarView.avatarController.send(audioData: ArrayBuffer, end: boolean)
-// audioData: 音频数据（ArrayBuffer 格式）
-// end: false（默认）- 正常发送音频数据，服务端会积累音频数据，积累到一定量后会自动返回动画数据并开始同步播放动画和音频
-// end: true - 立即返回动画数据，不再积累，用于结束当前对话或需要立即响应的场景
+// Send audio data
+const conversationId = avatarView.avatarController.send(audioData: ArrayBuffer, end: boolean)
+// Returns: conversationId - Conversation ID for this conversation session (used to distinguish each conversation round)
+// audioData: Audio data (ArrayBuffer format, must be 16kHz mono PCM16)
+//   - Sample rate: 16kHz (16000 Hz) - backend requirement
+//   - Format: PCM16 (16-bit signed integer, little-endian)
+//   - Channels: Mono (single channel)
+//   - Example: 1 second = 16000 samples × 2 bytes = 32000 bytes
+// end: false (default) - Continue sending audio data for current conversation
+// end: true - Mark the end of current conversation round. After end=true, sending new audio data will interrupt any ongoing playback from the previous conversation round
+
+// Close WebSocket service
+avatarView.avatarController.close()
+```
+
+#### Host Mode Methods
+
+```typescript
+// Playback existing audio and animation data (starts a new conversation)
+const conversationId = await avatarView.avatarController.playback(
+  initialAudioChunks?: Array<{ data: Uint8Array, isLast: boolean }>,  // Existing audio chunks (16kHz mono PCM16)
+  initialKeyframes?: any[]  // Existing animation keyframes (obtained from your service)
+)
+// Returns: conversationId - New conversation ID for this conversation session
+
+// Stream audio chunks (can be called directly to start a new session, or after playback() to add more data)
+const conversationId = avatarView.avatarController.yieldAudioData(
+  data: Uint8Array,               // Audio chunk data
+  isLast: boolean = false         // Whether this is the last chunk
+)
+// Returns: conversationId - Conversation ID for this audio session
+// Note: If no conversationId exists, a new one will be automatically generated
+
+// Stream animation keyframes (requires conversationId from audio data)
+avatarView.avatarController.yieldFramesData(
+  keyframes: any[],                // Animation keyframes (obtained from your service)
+  conversationId: string                    // Conversation ID (required). Use getCurrentConversationId() or yieldAudioData() to get conversationId.
+)
+```
+
+**⚠️ Important: Conversation ID (conversationId) Management**
+
+**SDK Mode:**
+- `send()` returns a conversationId to distinguish each conversation round
+- `end=true` marks the end of a conversation round. After `end=true`, sending new audio data will interrupt any ongoing playback from the previous conversation round
+
+**Host Mode:**
+For each conversation session, you **must**:
+1. **First send audio data** to get a conversationId (used to distinguish each conversation round):
+   - `playback()` returns a conversationId when playback existing audio and animation data (replay mode)
+   - `yieldAudioData()` returns a conversationId for streaming new audio data
+2. **Then use that conversationId** to send animation data:
+   - `yieldFramesData()` requires a valid conversationId parameter
+   - Animation data with mismatched conversationId will be **discarded**
+   - Use `getCurrentConversationId()` to retrieve the current active conversationId
+
+**Example Flow (Host Mode):**
+```typescript
+// Option A: Playback existing complete data (replay mode)
+const conversationId = await avatarView.avatarController.playback(initialAudioChunks, initialKeyframes)
+
+// Option B: Start streaming new data directly
+// Step 1: Send audio data first to get conversationId (automatically generates if starting new session)
+const conversationId = avatarView.avatarController.yieldAudioData(audioChunk, false)
+// Step 2: Use the conversationId to send animation data
+avatarView.avatarController.yieldFramesData(keyframes, conversationId)
+// Note: To start playback with Option B, call playback() with accumulated data or ensure enough audio is sent
+```
+
+**Why conversationId is required:**
+- Ensures audio and animation data belong to the same conversation session
+- Prevents data from different sessions from being mixed
+- Automatically discards mismatched animation data for data integrity
+
+#### Common Methods (Both Modes)
+
+```typescript
+// Pause playback (can be resumed later)
+avatarView.avatarController.pause()
+
+// Resume playback (from paused state)
+await avatarView.avatarController.resume()
 
-// 打断对话
+// Interrupt current playback (stops and clears data)
 avatarView.avatarController.interrupt()
 
-// 关闭连接
-avatarView.avatarController.close()
+// Clear all data and resources
+avatarView.avatarController.clear()
+
+// Get current conversation ID (for Host mode)
+const conversationId = avatarView.avatarController.getCurrentConversationId()
+// Returns: Current conversationId for the active audio session, or null if no active session
 
-// 设置事件回调
-avatarView.avatarController.onConnectionState = (state: ConnectionState) => {}
+// Set event callbacks
+avatarView.avatarController.onConnectionState = (state: ConnectionState) => {} // SDK mode only
 avatarView.avatarController.onAvatarState = (state: AvatarState) => {}
 avatarView.avatarController.onError = (error: Error) => {}
-
-// 注意：不支持 sendText() 方法，调用会抛出错误
 ```
 
-## 🔧 配置
+**Important Notes:**
+- `start()` and `close()` are only available in SDK mode
+- `playback()`, `yieldAudioData()`, and `yieldFramesData()` are only available in Host mode
+- `pause()`, `resume()`, `interrupt()`, `clear()`, and `getCurrentConversationId()` are available in both modes
+- The playback mode is determined when creating `AvatarView` and cannot be changed
+- **Conversation ID**: In Host mode, always send audio data first to obtain a conversationId, then use that conversationId when sending animation data. Animation data with mismatched conversationId will be discarded. Use `getCurrentConversationId()` to retrieve the current active conversationId.
+
+## 🔧 Configuration
 
 ### Configuration
 
 ```typescript
 interface Configuration {
   environment: Environment
+  drivingServiceMode?: DrivingServiceMode  // Optional, default is 'sdk' (SDK mode)
 }
 ```
 
-**说明：**
-- `environment`: 指定环境（cn/us/test），SDK 会根据环境自动使用对应的 API 地址和 WebSocket 地址
-- `sessionToken`: 通过 `AvatarKit.setSessionToken()` 单独设置，而不是在 Configuration 中
+**Description:**
+- `environment`: Specifies the environment (cn/us/test), SDK will automatically use the corresponding API address and WebSocket address based on the environment
+- `drivingServiceMode`: Specifies the driving service mode
+  - `DrivingServiceMode.sdk` (default): SDK mode - SDK handles WebSocket communication automatically
+  - `DrivingServiceMode.host`: Host mode - Host application provides audio and animation data
+- `sessionToken`: Set separately via `AvatarKit.setSessionToken()`, not in Configuration
 
+```typescript
 enum Environment {
-  cn = 'cn',    // 中国区
-  us = 'us',    // 美国区
-  test = 'test' // 测试环境
+  cn = 'cn',    // China region
+  us = 'us',    // US region
+  test = 'test' // Test environment
+}
+```
+
+### AvatarView Constructor
+
+```typescript
+constructor(avatar: Avatar, container: HTMLElement)
+```
+
+**Parameters:**
+- `avatar`: Avatar 实例
+- `container`: Canvas 容器元素（必选）
+  - Canvas 自动使用容器的完整尺寸（宽度和高度）
+  - Canvas 宽高比适应容器尺寸 - 设置容器尺寸以控制宽高比
+  - Canvas 会自动添加到容器中
+
+**Note:** 播放模式由 `AvatarKit.initialize()` 配置中的 `drivingServiceMode` 决定，而不是在构造函数参数中
+  - SDK automatically handles resize events via ResizeObserver
+
+```typescript
+enum AvatarPlaybackMode {
+  network = 'network',   // SDK mode: SDK handles WebSocket communication
+  external = 'external'  // Host mode: Host provides data, SDK handles playback
 }
 ```
 
@@ -221,17 +501,17 @@ enum Environment {
 
 ```typescript
 interface CameraConfig {
-  position: [number, number, number]  // 相机位置
-  target: [number, number, number]    // 相机目标
-  fov: number                         // 视野角度
-  near: number                        // 近裁剪面
-  far: number                         // 远裁剪面
-  up?: [number, number, number]       // 上方向
-  aspect?: number                     // 宽高比
+  position: [number, number, number]  // Camera position
+  target: [number, number, number]    // Camera target
+  fov: number                         // Field of view angle
+  near: number                        // Near clipping plane
+  far: number                         // Far clipping plane
+  up?: [number, number, number]       // Up direction
+  aspect?: number                     // Aspect ratio
 }
 ```
 
-## 📊 状态管理
+## 📊 State Management
 
 ### ConnectionState
 
@@ -248,77 +528,27 @@ enum ConnectionState {
 
 ```typescript
 enum AvatarState {
-  idle = 'idle',      // 空闲状态，呈现呼吸态
-  active = 'active',  // 活跃中，等待可播放内容
-  playing = 'playing' // 播放中
+  idle = 'idle',      // Idle state, showing breathing animation
+  active = 'active',  // Active, waiting for playable content
+  playing = 'playing', // Playing
+  paused = 'paused'   // Paused (can be resumed)
 }
 ```
 
-## 🎨 渲染系统
-
-SDK 支持两种渲染后端：
-
-- **WebGPU** - 现代浏览器的高性能渲染
-- **WebGL** - 兼容性更好的传统渲染
-
-渲染系统会自动选择最佳的后端，无需手动配置。
-
-## 🔍 调试和监控
-
-### 日志系统
-
-SDK 内置了完整的日志系统，支持不同级别的日志输出：
-
-```typescript
-import { logger } from '@spatialwalk/avatarkit'
-
-// 设置日志级别
-logger.setLevel('verbose') // 'basic' | 'verbose'
-
-// 手动日志输出
-logger.log('Info message')
-logger.warn('Warning message')
-logger.error('Error message')
-```
-
-### 性能监控
-
-SDK 提供了性能监控接口，可以监控渲染性能：
+## 🎨 Rendering System
 
-```typescript
-// 获取渲染性能统计
-const stats = avatarView.getPerformanceStats()
-
-if (stats) {
-  console.log(`渲染耗时: ${stats.renderTime.toFixed(2)}ms`)
-  console.log(`排序耗时: ${stats.sortTime.toFixed(2)}ms`)
-  console.log(`渲染后端: ${stats.backend}`)
-  
-  // 计算帧率
-  const fps = 1000 / stats.renderTime
-  console.log(`帧率: ${fps.toFixed(2)} FPS`)
-}
+The SDK supports two rendering backends:
 
-// 定期监控性能
-setInterval(() => {
-  const stats = avatarView.getPerformanceStats()
-  if (stats) {
-    // 发送到监控服务或显示在 UI 上
-    console.log('Performance:', stats)
-  }
-}, 1000)
-```
+- **WebGPU** - High-performance rendering for modern browsers
+- **WebGL** - Better compatibility traditional rendering
 
-**性能统计说明**：
-- `renderTime`: 总渲染耗时（毫秒），包含排序和 GPU 渲染
-- `sortTime`: 排序耗时（毫秒），使用 Radix Sort 算法对点云进行深度排序
-- `backend`: 当前使用的渲染后端（`'webgpu'` | `'webgl'` | `null`）
+The rendering system automatically selects the best backend, no manual configuration needed.
 
-## 🚨 错误处理
+## 🚨 Error Handling
 
 ### SPAvatarError
 
-SDK 使用自定义错误类型，提供更详细的错误信息：
+The SDK uses custom error types, providing more detailed error information:
 
 ```typescript
 import { SPAvatarError } from '@spatialwalk/avatarkit'
@@ -334,70 +564,147 @@ try {
 }
 ```
 
-### 错误回调
+### Error Callbacks
 
 ```typescript
 avatarView.avatarController.onError = (error: Error) => {
   console.error('AvatarController error:', error)
-  // 处理错误，比如重连、用户提示等
+  // Handle error, such as reconnection, user notification, etc.
 }
 ```
 
-## 🔄 资源管理
+## 🔄 Resource Management
 
-### 生命周期管理
+### Lifecycle Management
+
+#### SDK Mode Lifecycle
 
 ```typescript
-// 初始化
+// Initialize
+const container = document.getElementById('avatar-container')
 const avatarView = new AvatarView(avatar, container)
 await avatarView.avatarController.start()
 
-// 使用
+// Use
 avatarView.avatarController.send(audioData, false)
 
-// 清理（切换角色前必须调用）
-avatarView.dispose() // 自动清理所有资源
+// Cleanup
+avatarView.avatarController.close()
+avatarView.dispose() // Automatically cleans up all resources
+```
+
+#### Host Mode Lifecycle
+
+```typescript
+// Initialize
+const container = document.getElementById('avatar-container')
+const avatarView = new AvatarView(avatar, container)
+
+// Use
+const initialAudioChunks = [{ data: audioData1, isLast: false }]
+// Step 1: Send audio first to get conversationId
+const conversationId = await avatarView.avatarController.playback(initialAudioChunks, initialKeyframes)
+// Step 2: Stream additional audio (returns conversationId)
+const currentConversationId = avatarView.avatarController.yieldAudioData(audioChunk, false)
+// Step 3: Use conversationId to send animation data (mismatched conversationId will be discarded)
+avatarView.avatarController.yieldFramesData(keyframes, currentConversationId || conversationId)
+
+// Cleanup
+avatarView.avatarController.clear() // Clear all data and resources
+avatarView.dispose() // Automatically cleans up all resources
 ```
 
-**⚠️ 重要提示：** 
-- SDK 目前只支持同时存在一个 AvatarView 实例
-- 切换角色时，必须先调用 `dispose()` 清理旧的 AvatarView，然后再创建新的实例
-- 未正确清理可能导致资源泄漏和渲染错误
+**⚠️ Important Notes:** 
+- When disposing AvatarView instances, must call `dispose()` to properly clean up resources
+- Not properly cleaning up may cause resource leaks and rendering errors
+- In SDK mode, call `close()` before `dispose()` to properly close WebSocket connections
+- In Host mode, call `clear()` before `dispose()` to clear all playback data
+
+### Memory Optimization
+
+- SDK automatically manages WASM memory allocation
+- Supports dynamic loading/unloading of character and animation resources
+- Provides memory usage monitoring interface
+
+### Audio Data Sending
+
+#### SDK Mode
+
+The `send()` method receives audio data in `ArrayBuffer` format:
 
-### 内存优化
+**Audio Format Requirements:**
+- **Sample Rate**: 16kHz (16000 Hz) - **Backend requirement, must be exactly 16kHz**
+- **Format**: PCM16 (16-bit signed integer, little-endian)
+- **Channels**: Mono (single channel)
+- **Data Size**: Each sample is 2 bytes, so 1 second of audio = 16000 samples × 2 bytes = 32000 bytes
 
-- SDK 自动管理 WASM 内存分配
-- 支持角色和动画资源的动态加载/卸载
-- 提供内存使用监控接口
+**Usage:**
+- `audioData`: Audio data (ArrayBuffer format, must be 16kHz mono PCM16)
+- `end=false` (default) - Continue sending audio data for current conversation
+- `end=true` - Mark the end of current conversation round. After `end=true`, sending new audio data will interrupt any ongoing playback from the previous conversation round
+- **Important**: No need to wait for `end=true` to start playing, it will automatically start playing after accumulating enough audio data
 
-### 音频数据发送
+#### Host Mode
+
+The `playback()` method is used to playback existing audio and animation data (replay mode), generating a new conversationId and interrupting any existing conversation. 
+
+**Two ways to start a session in Host mode:**
+1. **Use `playback()`** - For replaying existing complete audio and animation data
+2. **Use `yieldAudioData()` directly** - For streaming new audio data (automatically generates conversationId if needed)
+
+Then use `yieldAudioData()` to stream additional audio:
+
+**Audio Format Requirements:**
+- Same as SDK mode: 16kHz mono PCM16 format
+- Audio data should be provided as `Uint8Array` in chunks with `isLast` flag
+
+**Usage:**
+```typescript
+// Playback existing audio and animation data (starts a new conversation)
+// Note: Audio and animation data should be obtained from your backend service
+const initialAudioChunks = [
+  { data: audioData1, isLast: false },
+  { data: audioData2, isLast: false }
+]
+const conversationId = await avatarController.playback(initialAudioChunks, initialKeyframes)
+// Returns: conversationId - New conversation ID for this conversation session
+
+// Stream additional audio chunks
+const conversationId = avatarController.yieldAudioData(audioChunk, isLast)
+// Returns: conversationId - Conversation ID for this audio session
+```
 
-`send()` 方法接收 `ArrayBuffer` 格式的音频数据：
+**⚠️ Conversation ID Workflow:**
+1. **Start a session** → Choose one of two ways:
+   - **Option A**: Use `playback(initialAudioChunks, initialKeyframes)` to replay existing complete data
+   - **Option B**: Use `yieldAudioData(audioChunk)` directly to start streaming (automatically generates conversationId)
+2. **Get conversationId** → Both methods return a conversationId
+3. **Send animation with conversationId** → Use the conversationId from step 1 in `yieldFramesData()`
+4. **Data matching** → Only animation data with matching conversationId will be accepted
 
-**使用说明：**
-- `audioData`: 音频数据（ArrayBuffer 格式）
-- `end=false`（默认）- 正常发送音频数据，服务端会积累音频数据，积累到一定量后会自动返回动画数据并开始同步播放动画和音频
-- `end=true` - 立即返回动画数据，不再积累，用于结束当前对话或需要立即响应的场景
-- **重要**：不需要等待 `end=true` 才开始播放，积累到一定音频数据后就会自动开始播放
+**Resampling (Both Modes):**
+- If your audio source is at a different sample rate (e.g., 24kHz, 48kHz), you **must** resample it to 16kHz before sending
+- For high-quality resampling, use Web Audio API's `OfflineAudioContext` with anti-aliasing filtering
+- See example projects (`vanilla`, `react`, `vue`) for complete resampling implementation
 
-## 🌐 浏览器兼容性
+## 🌐 Browser Compatibility
 
-- **Chrome/Edge** 90+ (推荐 WebGPU)
+- **Chrome/Edge** 90+ (WebGPU recommended)
 - **Firefox** 90+ (WebGL)
 - **Safari** 14+ (WebGL)
-- **移动端** iOS 14+, Android 8+
+- **Mobile** iOS 14+, Android 8+
 
-## 📝 许可证
+## 📝 License
 
 MIT License
 
-## 🤝 贡献
+## 🤝 Contributing
 
-欢迎提交 Issue 和 Pull Request！
+Issues and Pull Requests are welcome!
 
-## 📞 支持
+## 📞 Support
 
-如有问题，请联系：
-- 邮箱：support@spavatar.com
-- 文档：https://docs.spavatar.com
-- GitHub：https://github.com/spavatar/sdk
+For questions, please contact:
+- Email: support@spavatar.com
+- Documentation: https://docs.spatialreal.ai
+- GitHub: https://github.com/spavatar/sdk