@spatialwalk/avatarkit 1.0.0-beta.2 → 1.0.0-beta.20

package/README.md

@@ -1,25 +1,26 @@
 # SPAvatarKit SDK
 
-基于 3D Gaussian Splatting 的实时虚拟人物头像渲染 SDK，支持音频驱动的动画渲染和高质量 3D 渲染。
+Real-time virtual avatar rendering SDK based on 3D Gaussian Splatting, supporting audio-driven animation rendering and high-quality 3D rendering.
 
-## 🚀 特性
+## 🚀 Features
 
-- **3D Gaussian Splatting 渲染** - 基于最新的点云渲染技术，提供高质量的 3D 虚拟人物
-- **音频驱动的实时动画渲染** - 用户提供音频数据，SDK 负责接收动画数据并渲染
-- **WebGPU/WebGL 双渲染后端** - 自动选择最佳渲染后端，确保兼容性
-- **WASM 高性能计算** - 使用 C++ 编译的 WebAssembly 模块进行几何计算
-- **TypeScript 支持** - 完整的类型定义和智能提示
-- **模块化架构** - 清晰的组件分离，易于集成和扩展
+- **3D Gaussian Splatting Rendering** - Based on the latest point cloud rendering technology, providing high-quality 3D virtual avatars
+- **Audio-Driven Real-Time Animation Rendering** - Users provide audio data, SDK handles receiving animation data and rendering
+- **Multi-Character Support** - Support multiple avatar instances simultaneously, each with independent state and rendering
+- **WebGPU/WebGL Dual Rendering Backend** - Automatically selects the best rendering backend for compatibility
+- **WASM High-Performance Computing** - Uses C++ compiled WebAssembly modules for geometric calculations
+- **TypeScript Support** - Complete type definitions and IntelliSense
+- **Modular Architecture** - Clear component separation, easy to integrate and extend
 
-## 📦 安装
+## 📦 Installation
 
 ```bash
 npm install @spatialwalk/avatarkit
 ```
 
-## 🎯 快速开始
+## 🎯 Quick Start
 
-### 基础使用
+### Basic Usage
 
 ```typescript
 import { 
@@ -30,190 +31,467 @@ import {
   Environment 
 } from '@spatialwalk/avatarkit'
 
-// 1. 初始化 SDK
+// 1. Initialize SDK
+import { DrivingServiceMode } from '@spatialwalk/avatarkit'
+
 const configuration: Configuration = {
   environment: Environment.test,
+  drivingServiceMode: DrivingServiceMode.sdk, // Optional, 'sdk' is default
+  // - DrivingServiceMode.sdk: SDK mode - SDK handles WebSocket communication
+  // - DrivingServiceMode.host: Host mode - Host app provides audio and animation data
 }
 
 await AvatarKit.initialize('your-app-id', configuration)
 
-// 设置 sessionToken（如果需要，单独调用）
+// Set sessionToken (if needed, call separately)
 // AvatarKit.setSessionToken('your-session-token')
 
-// 2. 加载角色
-const avatarManager = new AvatarManager()
+// 2. Load character
+const avatarManager = AvatarManager.shared
 const avatar = await avatarManager.load('character-id', (progress) => {
   console.log(`Loading progress: ${progress.progress}%`)
 })
 
-// 3. 创建视图（自动创建 Canvas 和 AvatarController）
+// 3. Create view (automatically creates Canvas and AvatarController)
+// The playback mode is determined by drivingServiceMode in AvatarKit configuration
+// - DrivingServiceMode.sdk: SDK mode - SDK handles WebSocket communication
+// - DrivingServiceMode.host: Host mode - Host app provides audio and animation data
 const container = document.getElementById('avatar-container')
 const avatarView = new AvatarView(avatar, container)
 
-// 4. 启动实时通信
+// 4. Start real-time communication (SDK mode only)
 await avatarView.avatarController.start()
 
-// 5. 发送音频数据
-// 如果音频是 Uint8Array，可以使用 slice().buffer 转换为 ArrayBuffer
-const audioUint8 = new Uint8Array(1024) // 示例：音频数据
-const audioData = audioUint8.slice().buffer // 简化的转换方式，适用于 ArrayBuffer 和 SharedArrayBuffer
-avatarView.avatarController.send(audioData, false) // 发送音频数据，积累到一定量后会自动开始播放
-avatarView.avatarController.send(audioData, true) // end=true 表示立即返回动画数据，不再积累
+// 5. Send audio data (SDK mode)
+// ⚠️ Important: Audio must be 16kHz mono PCM16 format
+// If audio is Uint8Array, you can use slice().buffer to convert to ArrayBuffer
+const audioUint8 = new Uint8Array(1024) // Example: 16kHz PCM16 audio data (512 samples = 1024 bytes)
+const audioData = audioUint8.slice().buffer // Simplified conversion, works for ArrayBuffer and SharedArrayBuffer
+avatarView.avatarController.send(audioData, false) // Send audio data, will automatically start playing after accumulating enough data
+avatarView.avatarController.send(audioData, true) // end=true marks the end of current conversation round
 ```
 
-### 完整示例
+### Host Mode Example
+
+```typescript
+import { AvatarPlaybackMode } from '@spatialwalk/avatarkit'
+
+// 1-3. Same as SDK mode (initialize SDK, load character)
+
+// 3. Create view with Host mode
+const container = document.getElementById('avatar-container')
+const avatarView = new AvatarView(avatar, container)
+
+// 4. Host Mode Workflow:
+// ⚠️ IMPORTANT: In Host mode, you MUST send audio data FIRST to get a conversationId,
+//    then use that conversationId to send animation data.
+//    Animation data with mismatched conversationId will be discarded.
+
+// Option A: Playback existing audio and animation data (replay mode)
+const initialAudioChunks = [{ data: audioData1, isLast: false }, { data: audioData2, isLast: false }]
+const initialKeyframes = animationData1 // Animation keyframes from your service
+// Step 1: Send audio first to get conversationId
+const conversationId = await avatarView.avatarController.playback(initialAudioChunks, initialKeyframes)
+
+// Option B: Stream new audio and animation data (start a new session directly)
+// Step 1: Send audio data first to get conversationId (automatically generates conversationId if starting new session)
+const currentConversationId = avatarView.avatarController.yieldAudioData(audioData3, false)
+// Step 2: Use the conversationId to send animation data (mismatched conversationId will be discarded)
+avatarView.avatarController.yieldFramesData(animationData2, currentConversationId || conversationId)
+// Note: To start playback, you need to call playback() with the accumulated data, or ensure enough audio data is sent
+```
+
+### Complete Examples
+
+Check the example code in the GitHub repository for complete usage flows for both modes.
+
+**Example Project:** [AvatarKit-Web-Demo](https://github.com/spatialwalk/AvatarKit-Web-Demo)
+
+This repository contains complete examples for Vanilla JS, Vue 3, and React, demonstrating:
+- SDK mode: Real-time audio input with automatic animation data reception
+- Host mode: Custom data sources with manual audio/animation data management
+
+## 🏗️ Architecture Overview
+
+### Three-Layer Architecture
 
-查看 GitHub 仓库中的示例代码了解完整的使用流程。
+The SDK uses a three-layer architecture for clear separation of concerns:
 
-**示例项目：** [Avatarkit-web-demo](https://github.com/spatialwalk/Avatarkit-web-demo)
+1. **Rendering Layer (AvatarView)** - Responsible for 3D rendering only
+2. **Playback Layer (AvatarController)** - Manages audio/animation synchronization and playback
+3. **Network Layer** - Handles WebSocket communication (only in SDK mode, internal implementation)
 
-该仓库包含 Vanilla JS、Vue 3 和 React 的完整示例，展示了如何在不同框架中集成和使用 SPAvatarKit SDK。
+### Core Components
 
-## 🏗️ 架构概览
+- **AvatarKit** - SDK initialization and management
+- **AvatarManager** - Character resource loading and management
+- **AvatarView** - 3D rendering view (rendering layer)
+- **AvatarController** - Audio/animation playback controller (playback layer)
 
-### 核心组件
+### Playback Modes
 
-- **AvatarKit** - SDK 初始化和管理
-- **AvatarManager** - 角色资源加载和管理
-- **AvatarView** - 3D 渲染视图（内部包含 AvatarController）
-- **AvatarController** - 实时通信和数据处理
-- **AvatarCoreAdapter** - WASM 模块适配器
+The SDK supports two playback modes, configured in `AvatarKit.initialize()`:
 
-### 数据流
+#### 1. SDK Mode (Default)
+- Configured via `drivingServiceMode: DrivingServiceMode.sdk` in `AvatarKit.initialize()`
+- SDK handles WebSocket communication automatically
+- Send audio data via `AvatarController.send()`
+- SDK receives animation data from backend and synchronizes playback
+- Best for: Real-time audio input scenarios
+
+#### 2. Host Mode
+- Configured via `drivingServiceMode: DrivingServiceMode.host` in `AvatarKit.initialize()`
+- Host application manages its own network/data fetching
+- Host application provides both audio and animation data
+- SDK only handles synchronized playback
+- Best for: Custom data sources, pre-recorded content, or custom network implementations
+
+**Note:** The playback mode is determined by `drivingServiceMode` in `AvatarKit.initialize()` configuration.
+
+### Fallback Mechanism
+
+The SDK includes a fallback mechanism to ensure audio playback continues even when animation data is unavailable:
+
+- **SDK Mode**: If the server returns an error or fails to provide animation data, the SDK automatically enters audio-only mode and continues playing audio independently
+- **Host Mode**: If empty animation data is provided (empty array or undefined), the SDK automatically enters audio-only mode
+- Once in audio-only mode, any subsequent animation data for that session will be ignored, and only audio will continue playing
+- The fallback mode is interruptible, just like normal playback mode
+
+### Data Flow
+
+#### SDK Mode Flow
 
 ```
-用户音频输入（16kHz mono PCM） → AvatarController → WebSocket → 后台处理
-                                              ↓
-后台返回动画数据（FLAME 关键帧） → AvatarController → AnimationPlayer
-                                              ↓
-FLAME 参数 → AvatarCore.computeFrameFlatFromParams() → Splat 数据
-                                              ↓
-Splat 数据 → RenderSystem → WebGPU/WebGL → Canvas 渲染
+User audio input (16kHz mono PCM16) 
+    ↓
+AvatarController.send() 
+    ↓
+WebSocket → Backend processing
+    ↓
+Backend returns animation data (FLAME keyframes) 
+    ↓
+AvatarController → AnimationPlayer
+    ↓
+FLAME parameters → AvatarCore.computeFrameFlatFromParams() → Splat data
+    ↓
+AvatarController (playback loop) → AvatarView.renderRealtimeFrame()
+    ↓
+RenderSystem → WebGPU/WebGL → Canvas rendering
 ```
 
-**注意：** 用户需要自己提供音频数据（16kHz mono PCM），SDK 负责接收动画数据并渲染。
+#### Host Mode Flow
+
+```
+External data source (audio + animation)
+    ↓
+Step 1: Send audio data FIRST to get conversationId
+    ↓
+AvatarController.playback(initialAudio, initialKeyframes) // Returns conversationId
+    OR
+AvatarController.yieldAudioData(audioChunk) // Returns conversationId
+    ↓
+Step 2: Use conversationId to send animation data
+    ↓
+AvatarController.yieldFramesData(keyframes, conversationId) // Requires conversationId
+    ↓
+AvatarController → AnimationPlayer (synchronized playback)
+    ↓
+FLAME parameters → AvatarCore.computeFrameFlatFromParams() → Splat data
+    ↓
+AvatarController (playback loop) → AvatarView.renderRealtimeFrame()
+    ↓
+RenderSystem → WebGPU/WebGL → Canvas rendering
+```
+
+**Note:** 
+- In SDK mode, users provide audio data, SDK handles network communication and animation data reception
+- In Host mode, users provide both audio and animation data, SDK handles synchronized playback only
+
+### Audio Format Requirements
+
+**⚠️ Important:** The SDK requires audio data to be in **16kHz mono PCM16** format:
+
+- **Sample Rate**: 16kHz (16000 Hz) - This is a backend requirement
+- **Channels**: Mono (single channel)
+- **Format**: PCM16 (16-bit signed integer, little-endian)
+- **Byte Order**: Little-endian
+
+**Audio Data Format:**
+- Each sample is 2 bytes (16-bit)
+- Audio data should be provided as `ArrayBuffer` or `Uint8Array`
+- For example: 1 second of audio = 16000 samples × 2 bytes = 32000 bytes
 
-## 📚 API 参考
+**Resampling:**
+- If your audio source is at a different sample rate (e.g., 24kHz, 48kHz), you must resample it to 16kHz before sending to the SDK
+- For high-quality resampling, we recommend using Web Audio API's `OfflineAudioContext` with anti-aliasing filtering
+- See example projects for resampling implementation
+
+## 📚 API Reference
 
 ### AvatarKit
 
-SDK 的核心管理类，负责初始化和全局配置。
+The core management class of the SDK, responsible for initialization and global configuration.
 
 ```typescript
-// 初始化 SDK
+// Initialize SDK
 await AvatarKit.initialize(appId: string, configuration: Configuration)
 
-// 检查初始化状态
+// Check initialization status
 const isInitialized = AvatarKit.isInitialized
 
-// 清理资源（不再使用时必须调用）
+// Get initialized app ID
+const appId = AvatarKit.appId
+
+// Get configuration
+const config = AvatarKit.configuration
+
+// Set sessionToken (if needed, call separately)
+AvatarKit.setSessionToken('your-session-token')
+
+// Set userId (optional, for telemetry)
+AvatarKit.setUserId('user-id')
+
+// Get sessionToken
+const sessionToken = AvatarKit.sessionToken
+
+// Get userId
+const userId = AvatarKit.userId
+
+// Get SDK version
+const version = AvatarKit.version
+
+// Cleanup resources (must be called when no longer in use)
 AvatarKit.cleanup()
 ```
 
 ### AvatarManager
 
-角色资源管理器，负责下载、缓存和加载角色数据。
+Character resource manager, responsible for downloading, caching, and loading character data. Use the singleton instance via `AvatarManager.shared`.
 
 ```typescript
-const manager = new AvatarManager()
+// Get singleton instance
+const manager = AvatarManager.shared
 
-// 加载角色
+// Load character
 const avatar = await manager.load(
   characterId: string, 
   onProgress?: (progress: LoadProgressInfo) => void
 )
 
-// 清理缓存
+// Clear cache
 manager.clearCache()
 ```
 
 ### AvatarView
 
-3D 渲染视图，内部自动创建和管理 AvatarController。
+3D rendering view (rendering layer), responsible for 3D rendering only. Internally automatically creates and manages `AvatarController`.
 
-**⚠️ 重要限制：** 目前 SDK 只支持同时存在一个 AvatarView 实例。如果需要切换角色，必须先调用 `dispose()` 方法清理当前的 AvatarView，然后再创建新的实例。
+**Playback Mode Configuration:**
+- The playback mode is fixed when creating `AvatarView` and persists throughout its lifecycle
+- Cannot be changed after creation
 
 ```typescript
-// 创建视图（Canvas 会自动添加到容器中）
-const avatarView = new AvatarView(avatar: Avatar, container?: HTMLElement)
+import { AvatarPlaybackMode } from '@spatialwalk/avatarkit'
 
-// 获取 Canvas 元素
-const canvas = avatarView.getCanvas()
+// Create view (Canvas is automatically added to container)
+// Create view (playback mode is determined by drivingServiceMode in AvatarKit configuration)
+const container = document.getElementById('avatar-container')
+const avatarView = new AvatarView(avatar, container)
 
-// 设置背景
-avatarView.setBackgroundImage('path/to/image.jpg')
-avatarView.setBackgroundOpaque(true)
+// Get playback mode
+const mode = avatarView.playbackMode // 'network' | 'external'
 
-// 更新相机配置
-avatarView.updateCameraConfig(cameraConfig: CameraConfig)
+// Wait for first frame to render
+await avatarView.ready // Promise that resolves when the first frame is rendered
 
-// 清理资源（切换角色前必须调用）
+// Cleanup resources (must be called before switching characters)
 avatarView.dispose()
 ```
 
-**切换角色示例：**
+**Character Switching Example:**
 
 ```typescript
-// 切换角色前，必须先清理旧的 AvatarView
+// To switch characters, simply dispose the old view and create a new one
 if (currentAvatarView) {
   currentAvatarView.dispose()
-  currentAvatarView = null
 }
 
-// 加载新角色
+// Load new character
 const newAvatar = await avatarManager.load('new-character-id')
 
-// 创建新的 AvatarView
+// Create new AvatarView
 currentAvatarView = new AvatarView(newAvatar, container)
-await currentAvatarView.avatarController.start()
+
+// SDK mode: start connection
+if (currentAvatarView.playbackMode === AvatarPlaybackMode.network) {
+  await currentAvatarView.controller.start()
+}
 ```
 
 ### AvatarController
 
-实时通信控制器，处理 WebSocket 连接和动画数据。
+Audio/animation playback controller (playback layer), manages synchronized playback of audio and animation. Automatically handles WebSocket communication in SDK mode.
+
+**Two Usage Patterns:**
+
+#### SDK Mode Methods
 
 ```typescript
-// 启动连接
+// Start WebSocket service
 await avatarView.avatarController.start()
 
-// 发送音频数据
-avatarView.avatarController.send(audioData: ArrayBuffer, end: boolean)
-// audioData: 音频数据（ArrayBuffer 格式）
-// end: false（默认）- 正常发送音频数据，服务端会积累音频数据，积累到一定量后会自动返回动画数据并开始同步播放动画和音频
-// end: true - 立即返回动画数据，不再积累，用于结束当前对话或需要立即响应的场景
+// Send audio data
+const conversationId = avatarView.avatarController.send(audioData: ArrayBuffer, end: boolean)
+// Returns: conversationId - Conversation ID for this conversation session (used to distinguish each conversation round)
+// audioData: Audio data (ArrayBuffer format, must be 16kHz mono PCM16)
+//   - Sample rate: 16kHz (16000 Hz) - backend requirement
+//   - Format: PCM16 (16-bit signed integer, little-endian)
+//   - Channels: Mono (single channel)
+//   - Example: 1 second = 16000 samples × 2 bytes = 32000 bytes
+// end: false (default) - Continue sending audio data for current conversation
+// end: true - Mark the end of current conversation round. After end=true, sending new audio data will interrupt any ongoing playback from the previous conversation round
+
+// Close WebSocket service
+avatarView.avatarController.close()
+```
+
+#### Host Mode Methods
+
+```typescript
+// Playback existing audio and animation data (starts a new conversation)
+const conversationId = await avatarView.avatarController.playback(
+  initialAudioChunks?: Array<{ data: Uint8Array, isLast: boolean }>,  // Existing audio chunks (16kHz mono PCM16)
+  initialKeyframes?: any[]  // Existing animation keyframes (obtained from your service)
+)
+// Returns: conversationId - New conversation ID for this conversation session
+
+// Stream audio chunks (can be called directly to start a new session, or after playback() to add more data)
+const conversationId = avatarView.avatarController.yieldAudioData(
+  data: Uint8Array,               // Audio chunk data
+  isLast: boolean = false         // Whether this is the last chunk
+)
+// Returns: conversationId - Conversation ID for this audio session
+// Note: If no conversationId exists, a new one will be automatically generated
+
+// Stream animation keyframes (requires conversationId from audio data)
+avatarView.avatarController.yieldFramesData(
+  keyframes: any[],                // Animation keyframes (obtained from your service)
+  conversationId: string                    // Conversation ID (required). Use getCurrentConversationId() or yieldAudioData() to get conversationId.
+)
+```
+
+**⚠️ Important: Conversation ID (conversationId) Management**
+
+**SDK Mode:**
+- `send()` returns a conversationId to distinguish each conversation round
+- `end=true` marks the end of a conversation round. After `end=true`, sending new audio data will interrupt any ongoing playback from the previous conversation round
+
+**Host Mode:**
+For each conversation session, you **must**:
+1. **First send audio data** to get a conversationId (used to distinguish each conversation round):
+   - `playback()` returns a conversationId when playback existing audio and animation data (replay mode)
+   - `yieldAudioData()` returns a conversationId for streaming new audio data
+2. **Then use that conversationId** to send animation data:
+   - `yieldFramesData()` requires a valid conversationId parameter
+   - Animation data with mismatched conversationId will be **discarded**
+   - Use `getCurrentConversationId()` to retrieve the current active conversationId
+
+**Example Flow (Host Mode):**
+```typescript
+// Option A: Playback existing complete data (replay mode)
+const conversationId = await avatarView.avatarController.playback(initialAudioChunks, initialKeyframes)
+
+// Option B: Start streaming new data directly
+// Step 1: Send audio data first to get conversationId (automatically generates if starting new session)
+const conversationId = avatarView.avatarController.yieldAudioData(audioChunk, false)
+// Step 2: Use the conversationId to send animation data
+avatarView.avatarController.yieldFramesData(keyframes, conversationId)
+// Note: To start playback with Option B, call playback() with accumulated data or ensure enough audio is sent
+```
+
+**Why conversationId is required:**
+- Ensures audio and animation data belong to the same conversation session
+- Prevents data from different sessions from being mixed
+- Automatically discards mismatched animation data for data integrity
+
+#### Common Methods (Both Modes)
+
+```typescript
+// Pause playback (can be resumed later)
+avatarView.avatarController.pause()
+
+// Resume playback (from paused state)
+await avatarView.avatarController.resume()
 
-// 打断对话
+// Interrupt current playback (stops and clears data)
 avatarView.avatarController.interrupt()
 
-// 关闭连接
-avatarView.avatarController.close()
+// Clear all data and resources
+avatarView.avatarController.clear()
+
+// Get current conversation ID (for Host mode)
+const conversationId = avatarView.avatarController.getCurrentConversationId()
+// Returns: Current conversationId for the active audio session, or null if no active session
 
-// 设置事件回调
-avatarView.avatarController.onConnectionState = (state: ConnectionState) => {}
+// Set event callbacks
+avatarView.avatarController.onConnectionState = (state: ConnectionState) => {} // SDK mode only
 avatarView.avatarController.onAvatarState = (state: AvatarState) => {}
 avatarView.avatarController.onError = (error: Error) => {}
-
-// 注意：不支持 sendText() 方法，调用会抛出错误
 ```
 
-## 🔧 配置
+**Important Notes:**
+- `start()` and `close()` are only available in SDK mode
+- `playback()`, `yieldAudioData()`, and `yieldFramesData()` are only available in Host mode
+- `pause()`, `resume()`, `interrupt()`, `clear()`, and `getCurrentConversationId()` are available in both modes
+- The playback mode is determined when creating `AvatarView` and cannot be changed
+- **Conversation ID**: In Host mode, always send audio data first to obtain a conversationId, then use that conversationId when sending animation data. Animation data with mismatched conversationId will be discarded. Use `getCurrentConversationId()` to retrieve the current active conversationId.
+
+## 🔧 Configuration
 
 ### Configuration
 
 ```typescript
 interface Configuration {
   environment: Environment
+  drivingServiceMode?: DrivingServiceMode  // Optional, default is 'sdk' (SDK mode)
 }
 ```
 
-**说明：**
-- `environment`: 指定环境（cn/us/test），SDK 会根据环境自动使用对应的 API 地址和 WebSocket 地址
-- `sessionToken`: 通过 `AvatarKit.setSessionToken()` 单独设置，而不是在 Configuration 中
+**Description:**
+- `environment`: Specifies the environment (cn/us/test), SDK will automatically use the corresponding API address and WebSocket address based on the environment
+- `drivingServiceMode`: Specifies the driving service mode
+  - `DrivingServiceMode.sdk` (default): SDK mode - SDK handles WebSocket communication automatically
+  - `DrivingServiceMode.host`: Host mode - Host application provides audio and animation data
+- `sessionToken`: Set separately via `AvatarKit.setSessionToken()`, not in Configuration
 
+```typescript
 enum Environment {
-  cn = 'cn',    // 中国区
-  us = 'us',    // 美国区
-  test = 'test' // 测试环境
+  cn = 'cn',    // China region
+  us = 'us',    // US region
+  test = 'test' // Test environment
+}
+```
+
+### AvatarView Constructor
+
+```typescript
+constructor(avatar: Avatar, container: HTMLElement)
+```
+
+**Parameters:**
+- `avatar`: Avatar 实例
+- `container`: Canvas 容器元素（必选）
+  - Canvas 自动使用容器的完整尺寸（宽度和高度）
+  - Canvas 宽高比适应容器尺寸 - 设置容器尺寸以控制宽高比
+  - Canvas 会自动添加到容器中
+
+**Note:** 播放模式由 `AvatarKit.initialize()` 配置中的 `drivingServiceMode` 决定，而不是在构造函数参数中
+  - SDK automatically handles resize events via ResizeObserver
+
+```typescript
+enum AvatarPlaybackMode {
+  network = 'network',   // SDK mode: SDK handles WebSocket communication
+  external = 'external'  // Host mode: Host provides data, SDK handles playback
 }
 ```
 
@@ -221,17 +499,17 @@ enum Environment {
 
 ```typescript
 interface CameraConfig {
-  position: [number, number, number]  // 相机位置
-  target: [number, number, number]    // 相机目标
-  fov: number                         // 视野角度
-  near: number                        // 近裁剪面
-  far: number                         // 远裁剪面
-  up?: [number, number, number]       // 上方向
-  aspect?: number                     // 宽高比
+  position: [number, number, number]  // Camera position
+  target: [number, number, number]    // Camera target
+  fov: number                         // Field of view angle
+  near: number                        // Near clipping plane
+  far: number                         // Far clipping plane
+  up?: [number, number, number]       // Up direction
+  aspect?: number                     // Aspect ratio
 }
 ```
 
-## 📊 状态管理
+## 📊 State Management
 
 ### ConnectionState
 
@@ -248,77 +526,27 @@ enum ConnectionState {
 
 ```typescript
 enum AvatarState {
-  idle = 'idle',      // 空闲状态，呈现呼吸态
-  active = 'active',  // 活跃中，等待可播放内容
-  playing = 'playing' // 播放中
+  idle = 'idle',      // Idle state, showing breathing animation
+  active = 'active',  // Active, waiting for playable content
+  playing = 'playing', // Playing
+  paused = 'paused'   // Paused (can be resumed)
 }
 ```
 
-## 🎨 渲染系统
-
-SDK 支持两种渲染后端：
-
-- **WebGPU** - 现代浏览器的高性能渲染
-- **WebGL** - 兼容性更好的传统渲染
-
-渲染系统会自动选择最佳的后端，无需手动配置。
-
-## 🔍 调试和监控
-
-### 日志系统
-
-SDK 内置了完整的日志系统，支持不同级别的日志输出：
-
-```typescript
-import { logger } from '@spatialwalk/avatarkit'
-
-// 设置日志级别
-logger.setLevel('verbose') // 'basic' | 'verbose'
-
-// 手动日志输出
-logger.log('Info message')
-logger.warn('Warning message')
-logger.error('Error message')
-```
-
-### 性能监控
-
-SDK 提供了性能监控接口，可以监控渲染性能：
+## 🎨 Rendering System
 
-```typescript
-// 获取渲染性能统计
-const stats = avatarView.getPerformanceStats()
-
-if (stats) {
-  console.log(`渲染耗时: ${stats.renderTime.toFixed(2)}ms`)
-  console.log(`排序耗时: ${stats.sortTime.toFixed(2)}ms`)
-  console.log(`渲染后端: ${stats.backend}`)
-  
-  // 计算帧率
-  const fps = 1000 / stats.renderTime
-  console.log(`帧率: ${fps.toFixed(2)} FPS`)
-}
+The SDK supports two rendering backends:
 
-// 定期监控性能
-setInterval(() => {
-  const stats = avatarView.getPerformanceStats()
-  if (stats) {
-    // 发送到监控服务或显示在 UI 上
-    console.log('Performance:', stats)
-  }
-}, 1000)
-```
+- **WebGPU** - High-performance rendering for modern browsers
+- **WebGL** - Better compatibility traditional rendering
 
-**性能统计说明**：
-- `renderTime`: 总渲染耗时（毫秒），包含排序和 GPU 渲染
-- `sortTime`: 排序耗时（毫秒），使用 Radix Sort 算法对点云进行深度排序
-- `backend`: 当前使用的渲染后端（`'webgpu'` | `'webgl'` | `null`）
+The rendering system automatically selects the best backend, no manual configuration needed.
 
-## 🚨 错误处理
+## 🚨 Error Handling
 
 ### SPAvatarError
 
-SDK 使用自定义错误类型，提供更详细的错误信息：
+The SDK uses custom error types, providing more detailed error information:
 
 ```typescript
 import { SPAvatarError } from '@spatialwalk/avatarkit'
@@ -334,70 +562,147 @@ try {
 }
 ```
 
-### 错误回调
+### Error Callbacks
 
 ```typescript
 avatarView.avatarController.onError = (error: Error) => {
   console.error('AvatarController error:', error)
-  // 处理错误，比如重连、用户提示等
+  // Handle error, such as reconnection, user notification, etc.
 }
 ```
 
-## 🔄 资源管理
+## 🔄 Resource Management
 
-### 生命周期管理
+### Lifecycle Management
+
+#### SDK Mode Lifecycle
 
 ```typescript
-// 初始化
+// Initialize
+const container = document.getElementById('avatar-container')
 const avatarView = new AvatarView(avatar, container)
 await avatarView.avatarController.start()
 
-// 使用
+// Use
 avatarView.avatarController.send(audioData, false)
 
-// 清理（切换角色前必须调用）
-avatarView.dispose() // 自动清理所有资源
+// Cleanup
+avatarView.avatarController.close()
+avatarView.dispose() // Automatically cleans up all resources
+```
+
+#### Host Mode Lifecycle
+
+```typescript
+// Initialize
+const container = document.getElementById('avatar-container')
+const avatarView = new AvatarView(avatar, container)
+
+// Use
+const initialAudioChunks = [{ data: audioData1, isLast: false }]
+// Step 1: Send audio first to get conversationId
+const conversationId = await avatarView.avatarController.playback(initialAudioChunks, initialKeyframes)
+// Step 2: Stream additional audio (returns conversationId)
+const currentConversationId = avatarView.avatarController.yieldAudioData(audioChunk, false)
+// Step 3: Use conversationId to send animation data (mismatched conversationId will be discarded)
+avatarView.avatarController.yieldFramesData(keyframes, currentConversationId || conversationId)
+
+// Cleanup
+avatarView.avatarController.clear() // Clear all data and resources
+avatarView.dispose() // Automatically cleans up all resources
 ```
 
-**⚠️ 重要提示：** 
-- SDK 目前只支持同时存在一个 AvatarView 实例
-- 切换角色时，必须先调用 `dispose()` 清理旧的 AvatarView，然后再创建新的实例
-- 未正确清理可能导致资源泄漏和渲染错误
+**⚠️ Important Notes:** 
+- When disposing AvatarView instances, must call `dispose()` to properly clean up resources
+- Not properly cleaning up may cause resource leaks and rendering errors
+- In SDK mode, call `close()` before `dispose()` to properly close WebSocket connections
+- In Host mode, call `clear()` before `dispose()` to clear all playback data
+
+### Memory Optimization
+
+- SDK automatically manages WASM memory allocation
+- Supports dynamic loading/unloading of character and animation resources
+- Provides memory usage monitoring interface
+
+### Audio Data Sending
+
+#### SDK Mode
+
+The `send()` method receives audio data in `ArrayBuffer` format:
 
-### 内存优化
+**Audio Format Requirements:**
+- **Sample Rate**: 16kHz (16000 Hz) - **Backend requirement, must be exactly 16kHz**
+- **Format**: PCM16 (16-bit signed integer, little-endian)
+- **Channels**: Mono (single channel)
+- **Data Size**: Each sample is 2 bytes, so 1 second of audio = 16000 samples × 2 bytes = 32000 bytes
 
-- SDK 自动管理 WASM 内存分配
-- 支持角色和动画资源的动态加载/卸载
-- 提供内存使用监控接口
+**Usage:**
+- `audioData`: Audio data (ArrayBuffer format, must be 16kHz mono PCM16)
+- `end=false` (default) - Continue sending audio data for current conversation
+- `end=true` - Mark the end of current conversation round. After `end=true`, sending new audio data will interrupt any ongoing playback from the previous conversation round
+- **Important**: No need to wait for `end=true` to start playing, it will automatically start playing after accumulating enough audio data
 
-### 音频数据发送
+#### Host Mode
+
+The `playback()` method is used to playback existing audio and animation data (replay mode), generating a new conversationId and interrupting any existing conversation. 
+
+**Two ways to start a session in Host mode:**
+1. **Use `playback()`** - For replaying existing complete audio and animation data
+2. **Use `yieldAudioData()` directly** - For streaming new audio data (automatically generates conversationId if needed)
+
+Then use `yieldAudioData()` to stream additional audio:
+
+**Audio Format Requirements:**
+- Same as SDK mode: 16kHz mono PCM16 format
+- Audio data should be provided as `Uint8Array` in chunks with `isLast` flag
+
+**Usage:**
+```typescript
+// Playback existing audio and animation data (starts a new conversation)
+// Note: Audio and animation data should be obtained from your backend service
+const initialAudioChunks = [
+  { data: audioData1, isLast: false },
+  { data: audioData2, isLast: false }
+]
+const conversationId = await avatarController.playback(initialAudioChunks, initialKeyframes)
+// Returns: conversationId - New conversation ID for this conversation session
+
+// Stream additional audio chunks
+const conversationId = avatarController.yieldAudioData(audioChunk, isLast)
+// Returns: conversationId - Conversation ID for this audio session
+```
 
-`send()` 方法接收 `ArrayBuffer` 格式的音频数据：
+**⚠️ Conversation ID Workflow:**
+1. **Start a session** → Choose one of two ways:
+   - **Option A**: Use `playback(initialAudioChunks, initialKeyframes)` to replay existing complete data
+   - **Option B**: Use `yieldAudioData(audioChunk)` directly to start streaming (automatically generates conversationId)
+2. **Get conversationId** → Both methods return a conversationId
+3. **Send animation with conversationId** → Use the conversationId from step 1 in `yieldFramesData()`
+4. **Data matching** → Only animation data with matching conversationId will be accepted
 
-**使用说明：**
-- `audioData`: 音频数据（ArrayBuffer 格式）
-- `end=false`（默认）- 正常发送音频数据，服务端会积累音频数据，积累到一定量后会自动返回动画数据并开始同步播放动画和音频
-- `end=true` - 立即返回动画数据，不再积累，用于结束当前对话或需要立即响应的场景
-- **重要**：不需要等待 `end=true` 才开始播放，积累到一定音频数据后就会自动开始播放
+**Resampling (Both Modes):**
+- If your audio source is at a different sample rate (e.g., 24kHz, 48kHz), you **must** resample it to 16kHz before sending
+- For high-quality resampling, use Web Audio API's `OfflineAudioContext` with anti-aliasing filtering
+- See example projects (`vanilla`, `react`, `vue`) for complete resampling implementation
 
-## 🌐 浏览器兼容性
+## 🌐 Browser Compatibility
 
-- **Chrome/Edge** 90+ (推荐 WebGPU)
+- **Chrome/Edge** 90+ (WebGPU recommended)
 - **Firefox** 90+ (WebGL)
 - **Safari** 14+ (WebGL)
-- **移动端** iOS 14+, Android 8+
+- **Mobile** iOS 14+, Android 8+
 
-## 📝 许可证
+## 📝 License
 
 MIT License
 
-## 🤝 贡献
+## 🤝 Contributing
 
-欢迎提交 Issue 和 Pull Request！
+Issues and Pull Requests are welcome!
 
-## 📞 支持
+## 📞 Support
 
-如有问题，请联系：
-- 邮箱：support@spavatar.com
-- 文档：https://docs.spavatar.com
-- GitHub：https://github.com/spavatar/sdk
+For questions, please contact:
+- Email: support@spavatar.com
+- Documentation: https://docs.spatialreal.ai
+- GitHub: https://github.com/spavatar/sdk