@spatialwalk/avatarkit 1.0.0-beta.1 → 1.0.0-beta.100

package/README.md

@@ -1,196 +1,809 @@
-# SPAvatarKit SDK
+# AvatarKit SDK
 
-基于 3D Gaussian Splatting 的实时虚拟人物头像渲染 SDK，支持音频驱动的动画渲染和高质量 3D 渲染。
+Real-time virtual avatar rendering SDK for Web, supporting audio-driven animation and high-quality 3D rendering.
 
-## 🚀 特性
+## 🚀 Features
 
-- **3D Gaussian Splatting 渲染** - 基于最新的点云渲染技术，提供高质量的 3D 虚拟人物
-- **音频驱动的实时动画渲染** - 用户提供音频数据，SDK 负责接收动画数据并渲染
-- **WebGPU/WebGL 双渲染后端** - 自动选择最佳渲染后端，确保兼容性
-- **WASM 高性能计算** - 使用 C++ 编译的 WebAssembly 模块进行几何计算
-- **TypeScript 支持** - 完整的类型定义和智能提示
-- **模块化架构** - 清晰的组件分离，易于集成和扩展
+- **High-Quality 3D Rendering** - GPU-accelerated avatar rendering with automatic backend selection
+- **Audio-Driven Real-Time Animation** - Send audio data, SDK handles animation and rendering
+- **Multi-Avatar Support** - Support multiple avatar instances simultaneously, each with independent state and rendering
+- **TypeScript Support** - Complete type definitions and IntelliSense
+- **Modular Architecture** - Clear component separation, easy to integrate and extend
 
-## 📦 安装
+## 📦 Installation
 
 ```bash
 npm install @spatialwalk/avatarkit
 ```
 
-## 🎯 快速开始
+## 🚧 Release Gate (Hard Rule)
 
-### 基础使用
+Release must pass gates before publish. Do not publish by manual ad-hoc commands.
+
+Required gate checks:
+
+```bash
+pnpm typecheck
+pnpm test
+pnpm build
+./tools/check_perf_baseline_release_gate.sh
+```
+
+If iteration includes bugfixes, `docs/bugfix-history.md` must have completed rows (test mapping + red/green evidence).
+
+Hotfix bypass is allowed only for emergency and must be recorded:
+
+```bash
+HOTFIX_BYPASS=1 ./tools/check_perf_baseline_release_gate.sh
+```
+
+## 🧪 Benchmark Demo (Web SDK)
+
+Use the dedicated benchmark demo (independent from `vanilla/`) for perf/render baseline runs:
+
+```bash
+pnpm demo:benchmark
+```
+
+## 🚀 Demo Repository
+
+<div align="center">
+
+### 📌 **Quick Start: Check Out Our Demo Repository**
+
+We provide complete example code and best practices to help you quickly integrate the SDK.
+
+**The demo repository includes:**
+- ✅ Complete integration examples
+- ✅ Usage examples for both SDK mode and Host mode  
+- ✅ Audio processing examples (PCM16, WAV, MP3, etc.)
+- ✅ Vite configuration examples
+- ✅ Next.js configuration examples
+- ✅ Best practices for common scenarios
+
+**[👉 View Demo Repository](https://github.com/spatialwalk/avatarkit-demo)** | *If not yet created, please contact the team*
+
+</div>
+
+---
+
+## 🔧 Vite Configuration (Recommended)
+
+If you are using Vite as your build tool, we strongly recommend using our Vite plugin to automatically handle WASM file configuration. The plugin automatically handles all necessary configurations, so you don't need to set them up manually.
+
+### Using the Plugin
+
+Add the plugin to `vite.config.ts`:
+
+```typescript
+import { defineConfig } from 'vite'
+import { avatarkitVitePlugin } from '@spatialwalk/avatarkit/vite'
+
+export default defineConfig({
+  plugins: [
+    avatarkitVitePlugin(), // Just add this line
+  ],
+})
+```
+
+### Plugin Features
+
+The plugin automatically handles:
+
+- ✅ **Development Server**: Automatically sets the correct MIME type (`application/wasm`) for WASM files
+- ✅ **Build Time**: Automatically copies WASM files to `dist/assets/` directory
+- ✅ **Cloudflare Pages**: Automatically generates `_headers` file to ensure WASM files use the correct MIME type
+- ✅ **Vite Configuration**: Automatically configures `optimizeDeps`, `assetsInclude`, `assetsInlineLimit`, and other options
+
+### Manual Configuration (Without Plugin)
+
+If you don't use the Vite plugin, you need to manually configure the following:
+
+```typescript
+// vite.config.ts
+export default defineConfig({
+  optimizeDeps: {
+    exclude: ['@spatialwalk/avatarkit'],
+  },
+  assetsInclude: ['**/*.wasm'],
+  build: {
+    assetsInlineLimit: 0,
+    rollupOptions: {
+      output: {
+        assetFileNames: (assetInfo) => {
+          if (assetInfo.name?.endsWith('.wasm')) {
+            return 'assets/[name][extname]'
+          }
+          return 'assets/[name]-[hash][extname]'
+        },
+      },
+    },
+  },
+  // Development server needs to manually configure middleware to set WASM MIME type
+  configureServer(server) {
+    server.middlewares.use((req, res, next) => {
+      if (req.url?.endsWith('.wasm')) {
+        res.setHeader('Content-Type', 'application/wasm')
+      }
+      next()
+    })
+  },
+})
+```
+
+## 🔧 Next.js Configuration
+
+For Next.js projects, use the `withAvatarkit` wrapper to automatically handle WASM file configuration with webpack.
+
+### Using the Plugin
+
+Wrap your Next.js config in `next.config.mjs`:
+
+```javascript
+import { withAvatarkit } from '@spatialwalk/avatarkit/next'
+
+export default withAvatarkit({
+  // ...your existing Next.js config
+})
+```
+
+### Plugin Features
+
+The plugin automatically handles:
+
+- ✅ **Path Fix**: Patches asset path resolution so WASM files are correctly loaded at `/_next/static/chunks/`
+- ✅ **WASM Copying**: Copies `.wasm` files into `static/chunks/` via a custom webpack plugin (client build only)
+- ✅ **Content-Type Headers**: Adds `application/wasm` response header for `/_next/static/chunks/*.wasm`
+- ✅ **Config Chaining**: Preserves your existing `webpack` and `headers` configurations
+
+## 🔐 Authentication
+
+All environments require an **App ID** and **Session Token** for authentication.
+
+### App ID
+
+The App ID is used to identify your application. You can obtain your App ID by:
+
+1. **For Testing**: Use the default test App ID provided in demo repositories (paired with test Session Token, only works with publicly available test avatars like Rohan, Dr.Kellan, Priya, Josh, etc.)
+2. **For Production**: Visit the [Developer Platform](https://dash.spatialreal.ai) to create your own App and avatars. You will receive your own App ID after creating an App.
+
+### Session Token
+
+The Session Token is required for authentication and must be obtained from your SDK provider.
+
+**⚠️ Important Notes:**
+- The Session Token must be valid and not expired
+- In production applications, you **must** manually inject a valid Session Token obtained from your SDK provider
+- The default Session Token provided in demo repositories is **only for demonstration purposes** and can only be used with test avatars
+- If you want to create your own avatars and test them, please visit the [Developer Platform](https://dash.spatialreal.ai) to create your own App and generate Session Tokens
+
+**How to Set Session Token:**
+
+```typescript
+// Initialize SDK with App ID
+await AvatarSDK.initialize('your-app-id', configuration)
+
+// Set Session Token (can be called before or after initialization)
+// If called before initialization, the token will be automatically set when you initialize the SDK
+AvatarSDK.setSessionToken('your-session-token')
+
+// Get current Session Token
+const sessionToken = AvatarSDK.sessionToken
+```
+
+**Token Management:**
+- The Session Token can be set at any time using `AvatarSDK.setSessionToken(token)`
+- If you set the token before initializing the SDK, it will be automatically applied during initialization
+- If you set the token after initialization, it will be applied immediately
+- Handle token refresh logic in your application as needed (e.g., when token expires)
+
+**For Production Integration:**
+- Obtain a valid Session Token from your SDK provider
+- Store the token securely (never expose it in client-side code if possible)
+- Implement token refresh logic to handle token expiration
+- Use `AvatarSDK.setSessionToken(token)` to inject the token programmatically
+
+## 🎯 Quick Start
+
+### ⚠️ Important: Audio Context Initialization
+
+**Before using any audio-related features, you MUST initialize the audio context in a user gesture context** (e.g., `click`, `touchstart` event handlers). This is required by browser security policies. Calling `initializeAudioContext()` outside a user gesture will fail.
+
+### Basic Usage
 
 ```typescript
 import { 
-  AvatarKit, 
+  AvatarSDK, 
   AvatarManager, 
   AvatarView,
   Configuration,
-  Environment 
+  Environment,
+  DrivingServiceMode,
+  LogLevel
 } from '@spatialwalk/avatarkit'
 
-// 1. 初始化 SDK
+// 1. Initialize SDK
+
 const configuration: Configuration = {
-  environment: Environment.test,
+  environment: Environment.cn,
+  drivingServiceMode: DrivingServiceMode.sdk, // Optional, 'sdk' is default
+  // - DrivingServiceMode.sdk: SDK mode - SDK handles network communication
+  // - DrivingServiceMode.host: Host mode - Host app provides audio and animation data
+  logLevel: LogLevel.off, // Optional, 'off' is default
+  // - LogLevel.off: Disable all logs
+  // - LogLevel.error: Only error logs
+  // - LogLevel.warning: Warning and error logs
+  // - LogLevel.all: All logs (info, warning, error)
+  audioFormat: { // Default is { channelCount: 1, sampleRate: 16000 }
+    channelCount: 1, // Fixed to 1 (mono)
+    sampleRate: 16000 // Supported: 8000, 16000, 22050, 24000, 32000, 44100, 48000 Hz
+    // ⚠️ Must match your actual audio sample rate. Mismatched sample rate will cause playback issues.
+  }
+  // characterApiBaseUrl: 'https://custom-api.example.com' // Optional, internal debug config, can be ignored
 }
 
-await AvatarKit.initialize('your-app-id', configuration)
+await AvatarSDK.initialize('your-app-id', configuration)
 
-// 设置 sessionToken（如果需要，单独调用）
-// AvatarKit.setSessionToken('your-session-token')
+// Set Session Token (required for authentication)
+// You must obtain a valid Session Token from your SDK provider
+// See Authentication section above for more details
+AvatarSDK.setSessionToken('your-session-token')
 
-// 2. 加载角色
-const avatarManager = new AvatarManager()
+// 2. Load avatar
+const avatarManager = AvatarManager.shared
 const avatar = await avatarManager.load('character-id', (progress) => {
   console.log(`Loading progress: ${progress.progress}%`)
 })
 
-// 3. 创建视图（自动创建 Canvas 和 AvatarController）
+// 3. Create view (automatically creates Canvas and AvatarController)
+// The playback mode is determined by drivingServiceMode in AvatarSDK configuration
+// - DrivingServiceMode.sdk: SDK mode - SDK handles network communication
+// - DrivingServiceMode.host: Host mode - Host app provides audio and animation data
 const container = document.getElementById('avatar-container')
 const avatarView = new AvatarView(avatar, container)
 
-// 4. 启动实时通信
-await avatarView.avatarController.start()
+// 4. ⚠️ CRITICAL: Initialize audio context (MUST be called in user gesture context)
+// This method MUST be called within a user gesture event handler (click, touchstart, etc.)
+// to satisfy browser security policies. Calling it outside a user gesture will fail.
+button.addEventListener('click', async () => {
+  // Initialize audio context - MUST be in user gesture context
+  await avatarView.controller.initializeAudioContext()
+  
+  // 5. Start real-time communication (SDK mode only)
+  // Note: start() initiates the WebSocket connection asynchronously.
+  // Wait for onConnectionState === 'connected' before calling send().
+  await avatarView.controller.start()
+
+  // 6. Wait for connection to be ready
+  await new Promise<void>((resolve) => {
+    avatarView.controller.onConnectionState = (state) => {
+      if (state === ConnectionState.connected) resolve()
+    }
+  })
+
+  // 7. Send audio data (SDK mode, must be mono PCM16 format matching configured sample rate)
+  // audioData: ArrayBuffer or Uint8Array containing PCM16 (S16LE) audio samples
+  // ⚠️ Byte length MUST be even (2 bytes per sample). Odd-length data will cause server-side
+  //    validation error and WebSocket disconnect.
+  // - PCM files: Can be directly read as ArrayBuffer
+  // - WAV files: Extract PCM data from WAV format (may require resampling)
+  // - MP3 files: Decode first (e.g., using AudioContext.decodeAudioData()), then convert to PCM16
+  const audioData = new ArrayBuffer(1024) // Placeholder: Replace with actual PCM16 audio data
+  avatarView.controller.send(audioData, false) // Send audio data
+  avatarView.controller.send(audioData, true) // end=true marks the end of current conversation round
+})
+```
+
+### Host Mode Example
+
+```typescript
+
+// 1-3. Same as SDK mode (initialize SDK, load avatar)
+
+// 3. Create view with Host mode
+const container = document.getElementById('avatar-container')
+const avatarView = new AvatarView(avatar, container)
 
-// 5. 发送音频数据
-// 如果音频是 Uint8Array，可以使用 slice().buffer 转换为 ArrayBuffer
-const audioUint8 = new Uint8Array(1024) // 示例：音频数据
-const audioData = audioUint8.slice().buffer // 简化的转换方式，适用于 ArrayBuffer 和 SharedArrayBuffer
-avatarView.avatarController.send(audioData, false) // 发送音频数据，积累到一定量后会自动开始播放
-avatarView.avatarController.send(audioData, true) // end=true 表示立即返回动画数据，不再积累
+// 4. ⚠️ CRITICAL: Initialize audio context (MUST be called in user gesture context)
+// This method MUST be called within a user gesture event handler (click, touchstart, etc.)
+// to satisfy browser security policies. Calling it outside a user gesture will fail.
+button.addEventListener('click', async () => {
+  // Initialize audio context - MUST be in user gesture context
+  await avatarView.controller.initializeAudioContext()
+  
+  // 5. Host Mode Workflow:
+  // Send audio data first to get conversationId, then use it to send animation data
+  const conversationId = avatarView.controller.yieldAudioData(audioData, false)
+  avatarView.controller.yieldFramesData(animationDataArray, conversationId) // animationDataArray: (Uint8Array | ArrayBuffer)[]
 ```
 
-### 完整示例
+### Complete Examples
+
+This SDK supports two usage modes:
+- SDK mode: Real-time audio input with automatic animation data reception
+- Host mode: Custom data sources with manual audio/animation data management
+
+## 🏗️ Architecture Overview
+
+### Core Components
+
+- **AvatarSDK** - SDK initialization and management
+- **AvatarManager** - Avatar resource loading and management
+- **AvatarView** - 3D rendering view
+- **AvatarController** - Audio/animation playback controller
 
-查看 GitHub 仓库中的示例代码了解完整的使用流程。
+### Playback Modes
 
-## 🏗️ 架构概览
+The SDK supports two playback modes, configured in `AvatarSDK.initialize()`:
 
-### 核心组件
+#### 1. SDK Mode (Default)
+- Configured via `drivingServiceMode: DrivingServiceMode.sdk` in `AvatarSDK.initialize()`
+- SDK handles network communication automatically
+- Send audio data via `AvatarController.send()`
+- SDK receives animation data from backend and synchronizes playback
+- Best for: Real-time audio input scenarios
 
-- **AvatarKit** - SDK 初始化和管理
-- **AvatarManager** - 角色资源加载和管理
-- **AvatarView** - 3D 渲染视图（内部包含 AvatarController）
-- **AvatarController** - 实时通信和数据处理
-- **AvatarCoreAdapter** - WASM 模块适配器
+#### 2. Host Mode
+- Configured via `drivingServiceMode: DrivingServiceMode.host` in `AvatarSDK.initialize()`
+- Host application manages its own network/data fetching
+- Host application provides both audio and animation data
+- SDK only handles synchronized playback
+- Best for: Custom data sources, pre-recorded content, or custom network implementations
 
-### 数据流
+**Note:** The playback mode is determined by `drivingServiceMode` in `AvatarSDK.initialize()` configuration.
+
+### Fallback Mechanism
+
+The SDK includes a fallback mechanism to ensure audio playback continues even when animation data is unavailable:
+
+- **SDK Mode Connection Failure**: If connection fails to establish within 15 seconds, the SDK automatically enters fallback mode. Audio data can still be sent and will play normally, even though no animation data will be received. This ensures audio playback is not interrupted.
+- **SDK Mode Server Error**: If the server returns an error after connection is established, the SDK automatically enters audio-only mode for that session.
+- **Host Mode**: If empty animation data is provided (empty array or undefined), the SDK automatically enters audio-only mode.
+- Once in audio-only mode, any subsequent animation data for that session will be ignored, and only audio will continue playing.
+- The fallback mode is interruptible, just like normal playback mode.
+- Connection state callbacks (`onConnectionState`) will notify you when connection fails or times out.
+
+### Data Flow
+
+#### SDK Mode Flow
 
 ```
-用户音频输入（16kHz mono PCM） → AvatarController → WebSocket → 后台处理
-                                              ↓
-后台返回动画数据（FLAME 关键帧） → AvatarController → AnimationPlayer
-                                              ↓
-FLAME 参数 → AvatarCore.computeFrameFlatFromParams() → Splat 数据
-                                              ↓
-Splat 数据 → RenderSystem → WebGPU/WebGL → Canvas 渲染
+Audio input (PCM16 mono)
+    ↓
+AvatarController.send()
+    ↓
+Backend processing → Animation data
+    ↓
+SDK synchronizes audio + animation playback
+    ↓
+GPU rendering → Canvas
 ```
 
-**注意：** 用户需要自己提供音频数据（16kHz mono PCM），SDK 负责接收动画数据并渲染。
+#### Host Mode Flow
 
-## 📚 API 参考
+```
+External data source (audio + animation)
+    ↓
+AvatarController.yieldAudioData(audioChunk)       → returns conversationId
+AvatarController.yieldFramesData(dataArray, conversationId)
+    ↓
+SDK synchronizes audio + animation playback
+    ↓
+GPU rendering → Canvas
+```
 
-### AvatarKit
+### Audio Format Requirements
 
-SDK 的核心管理类，负责初始化和全局配置。
+**⚠️ Important:** The SDK requires audio data to be in **mono PCM16** format:
 
+- **Sample Rate**: Configurable via `audioFormat.sampleRate` in SDK initialization (default: 16000 Hz)
+  - Supported sample rates: 8000, 16000, 22050, 24000, 32000, 44100, 48000 Hz
+  - The configured sample rate will be used for both audio recording and playback
+- **Channels**: Mono (single channel) - Fixed to 1 channel
+- **Format**: PCM16 (16-bit signed integer, little-endian)
+- **Byte Order**: Little-endian
+
+**Audio Data Format:**
+- Each sample is 2 bytes (16-bit signed integer, little-endian)
+- Audio data should be provided as `ArrayBuffer` or `Uint8Array`
+- For example, with 16kHz sample rate: 1 second of audio = 16000 samples × 2 bytes = 32000 bytes
+- For 48kHz sample rate: 1 second of audio = 48000 samples × 2 bytes = 96000 bytes
+
+**Audio Data Source:**
+The `audioData` parameter represents raw PCM16 audio samples in the configured sample rate and mono format. Common audio sources include:
+- **PCM files**: Raw PCM16 files can be directly read as `ArrayBuffer` or `Uint8Array` and sent to the SDK (ensure sample rate matches configuration)
+- **WAV files**: WAV files contain PCM16 audio data in their data chunk. After extracting the PCM data from the WAV file format, it can be sent to the SDK (may require resampling if sample rate differs)
+- **MP3 files**: MP3 files need to be decoded first (e.g., using `AudioContext.decodeAudioData()` or a decoder library), then converted from the decoded format to PCM16 before sending to the SDK
+- **Microphone input**: Real-time microphone audio needs to be captured and converted to PCM16 format at the configured sample rate before sending
+- **Other audio sources**: Any audio source must be converted to mono PCM16 format at the configured sample rate before sending
+
+**Example: Processing WAV and MP3 Files:**
 ```typescript
-// 初始化 SDK
-await AvatarKit.initialize(appId: string, configuration: Configuration)
+// WAV file processing
+async function processWAVFile(wavFile: File): Promise<ArrayBuffer> {
+  const arrayBuffer = await wavFile.arrayBuffer()
+  const view = new DataView(arrayBuffer)
+  
+  // WAV format: Skip header (usually 44 bytes for standard WAV)
+  // Check RIFF header
+  if (view.getUint32(0, true) !== 0x46464952) { // "RIFF"
+    throw new Error('Invalid WAV file')
+  }
+  
+  // Find "data" chunk (offset may vary)
+  let dataOffset = 44 // Standard WAV header size
+  // For non-standard WAV files, you may need to search for "data" chunk
+  // This is a simplified example - production code should parse chunks properly
+  
+  const pcmData = arrayBuffer.slice(dataOffset)
+  return pcmData
+}
 
-// 检查初始化状态
-const isInitialized = AvatarKit.isInitialized
+// MP3 file processing
+async function processMP3File(mp3File: File, targetSampleRate: number): Promise<ArrayBuffer> {
+  const arrayBuffer = await mp3File.arrayBuffer()
+  const audioContext = new AudioContext({ sampleRate: targetSampleRate })
+  
+  // Decode MP3 to AudioBuffer
+  const audioBuffer = await audioContext.decodeAudioData(arrayBuffer.slice(0))
+  
+  // Convert AudioBuffer to PCM16 ArrayBuffer
+  const length = audioBuffer.length
+  const channels = audioBuffer.numberOfChannels
+  const pcm16Buffer = new ArrayBuffer(length * 2)
+  const pcm16View = new DataView(pcm16Buffer)
+  
+  // Mix down to mono if stereo
+  const sourceData = channels === 1 
+    ? audioBuffer.getChannelData(0)
+    : new Float32Array(length)
+  
+  if (channels > 1) {
+    const leftChannel = audioBuffer.getChannelData(0)
+    const rightChannel = audioBuffer.getChannelData(1)
+    for (let i = 0; i < length; i++) {
+      sourceData[i] = (leftChannel[i] + rightChannel[i]) / 2 // Mix to mono
+    }
+  }
+  
+  // Convert float32 (-1.0 to 1.0) to int16 (-32768 to 32767)
+  for (let i = 0; i < length; i++) {
+    const sample = Math.max(-1, Math.min(1, sourceData[i])) // Clamp
+    const int16Sample = sample < 0 ? sample * 0x8000 : sample * 0x7FFF
+    pcm16View.setInt16(i * 2, int16Sample, true) // little-endian
+  }
+  
+  audioContext.close()
+  return pcm16Buffer
+}
+
+// Usage example:
+// const wavPcmData = await processWAVFile(wavFile)
+// avatarView.controller.send(wavPcmData, false)
+//
+// const mp3PcmData = await processMP3File(mp3File, 16000) // 16kHz
+// avatarView.controller.send(mp3PcmData, false)
+```
+
+**Resampling:**
+- If your audio source is at a different sample rate, you must resample it to match the configured sample rate before sending to the SDK
+- For high-quality resampling, we recommend using Web Audio API's `OfflineAudioContext` with anti-aliasing filtering
+- See example projects for resampling implementation
 
-// 清理资源（不再使用时必须调用）
-AvatarKit.cleanup()
+**Configuration Example:**
+```typescript
+const configuration: Configuration = {
+  environment: Environment.cn,
+  audioFormat: {
+    channelCount: 1, // Fixed to 1 (mono)
+    sampleRate: 48000 // Choose from: 8000, 16000, 22050, 24000, 32000, 44100, 48000
+  }
+}
+```
+
+## 📚 API Reference
+
+### AvatarSDK
+
+The core management class of the SDK, responsible for initialization and global configuration.
+
+```typescript
+// Initialize SDK
+await AvatarSDK.initialize(appId: string, configuration: Configuration)
+
+// Check initialization status
+const isInitialized = AvatarSDK.isInitialized
+
+// Get initialized app ID
+const appId = AvatarSDK.appId
+
+// Get configuration
+const config = AvatarSDK.configuration
+
+// Set Session Token (required for authentication)
+// You must obtain a valid Session Token from your SDK provider
+// See Authentication section for more details
+AvatarSDK.setSessionToken('your-session-token')
+
+// Set userId (optional, for telemetry)
+AvatarSDK.setUserId('user-id')
+
+// Get sessionToken
+const sessionToken = AvatarSDK.sessionToken
+
+// Get userId
+const userId = AvatarSDK.userId
+
+// Get SDK version
+const version = AvatarSDK.version
+
+// Cleanup resources (must be called when no longer in use)
+AvatarSDK.cleanup()
 ```
 
 ### AvatarManager
 
-角色资源管理器，负责下载、缓存和加载角色数据。
+Avatar resource manager, responsible for downloading, caching, and loading avatar data. Use the singleton instance via `AvatarManager.shared`.
 
 ```typescript
-const manager = new AvatarManager()
+// Get singleton instance
+const manager = AvatarManager.shared
 
-// 加载角色
+// Load avatar
 const avatar = await manager.load(
-  characterId: string, 
+  id: string, 
   onProgress?: (progress: LoadProgressInfo) => void
 )
 
-// 清理缓存
-manager.clearCache()
+// Clear cache
+manager.clearAll()
 ```
 
 ### AvatarView
 
-3D 渲染视图，内部自动创建和管理 AvatarController。
+3D rendering view, responsible for 3D rendering only. Internally automatically creates and manages `AvatarController`.
+
+```typescript
+constructor(avatar: Avatar, container: HTMLElement)
+```
+
+**Parameters:**
+- `avatar`: Avatar instance
+- `container`: Canvas container element (required)
+  - Canvas automatically uses the full size of the container (width and height)
+  - Canvas aspect ratio adapts to container size - set container size to control aspect ratio
+  - Canvas will be automatically added to the container
+  - SDK automatically handles resize events via ResizeObserver
+
+**Playback Mode:**
+- The playback mode is determined by `drivingServiceMode` in `AvatarSDK.initialize()` configuration
+- The playback mode is fixed when creating `AvatarView` and persists throughout its lifecycle
+- Cannot be changed after creation
 
 ```typescript
-// 创建视图（Canvas 会自动添加到容器中）
-const avatarView = new AvatarView(avatar: Avatar, container?: HTMLElement)
+// Create view (Canvas is automatically added to container)
+const container = document.getElementById('avatar-container')
+const avatarView = new AvatarView(avatar, container)
 
-// 获取 Canvas 元素
-const canvas = avatarView.getCanvas()
+// Wait for first frame to render
+avatarView.onFirstRendering = () => {
+  // First frame rendered
+}
 
-// 设置背景
-avatarView.setBackgroundImage('path/to/image.jpg')
-avatarView.setBackgroundOpaque(true)
+// Get or set avatar transform (position and scale)
+// Get current transform
+const currentTransform = avatarView.avatarTransform // { x: number, y: number, scale: number }
 
-// 更新相机配置
-avatarView.updateCameraConfig(cameraConfig: CameraConfig)
+// Set transform
+avatarView.avatarTransform = { x, y, scale }
+// - x: Horizontal offset in normalized coordinates (-1 to 1, where -1 = left edge, 0 = center, 1 = right edge)
+// - y: Vertical offset in normalized coordinates (-1 to 1, where -1 = bottom edge, 0 = center, 1 = top edge)
+// - scale: Scale factor (1.0 = original size, 2.0 = double size, 0.5 = half size)
 
-// 清理资源
+// Cleanup resources (must be called before switching avatars)
 avatarView.dispose()
 ```
 
+**Switching Avatars:**
+
+To switch avatars, dispose the old view and create a new one. Do NOT attempt to reuse or reset an existing AvatarView.
+- `AvatarSDK.initialize()` and session token do not need to be called again.
+- The old AvatarView's internal state is fully cleaned up by `dispose()`.
+
+```typescript
+// 1. Dispose old avatar
+if (currentAvatarView) {
+  currentAvatarView.dispose()
+}
+
+// 2. Load new avatar (SDK is already initialized, token is still valid)
+const newAvatar = await AvatarManager.shared.load('new-character-id')
+
+// 3. Create new AvatarView
+currentAvatarView = new AvatarView(newAvatar, container)
+
+// 4. Start connection if SDK mode
+await currentAvatarView.controller.start()
+```
+
 ### AvatarController
 
-实时通信控制器，处理 WebSocket 连接和动画数据。
+Audio/animation playback controller, manages synchronized playback of audio and animation. Automatically handles network communication in SDK mode.
+
+**Two Usage Patterns:**
+
+#### SDK Mode Methods
 
 ```typescript
-// 启动连接
-await avatarView.avatarController.start()
+// ⚠️ CRITICAL: Initialize audio context first (MUST be called in user gesture context)
+// This method MUST be called within a user gesture event handler (click, touchstart, etc.)
+// to satisfy browser security policies. Calling it outside a user gesture will fail.
+// All audio operations (start, send, etc.) require prior initialization.
+button.addEventListener('click', async () => {
+  // Initialize audio context - MUST be in user gesture context
+  await avatarView.controller.initializeAudioContext()
+  
+  // Start service
+  await avatarView.controller.start()
+
+  // Send audio data (must be mono PCM16 format matching configured sample rate)
+  const conversationId = avatarView.controller.send(audioData: ArrayBuffer, end: boolean)
+  // Returns: conversationId - Conversation ID for this conversation session
+  // end: false (default) - Continue sending audio data for current conversation
+  // end: true - Mark the end of audio input for current conversation round. The avatar will continue playing remaining animation until finished, then automatically return to idle (notified via onConversationState). After end=true, sending new audio data will interrupt any ongoing playback from the previous conversation round
+})
 
-// 发送音频数据
-avatarView.avatarController.send(audioData: ArrayBuffer, end: boolean)
-// audioData: 音频数据（ArrayBuffer 格式）
-// end: false（默认）- 正常发送音频数据，服务端会积累音频数据，积累到一定量后会自动返回动画数据并开始同步播放动画和音频
-// end: true - 立即返回动画数据，不再积累，用于结束当前对话或需要立即响应的场景
+// Close service
+avatarView.controller.close()
+```
+
+#### Host Mode Methods
+
+```typescript
+// ⚠️ CRITICAL: Initialize audio context first (MUST be called in user gesture context)
+// This method MUST be called within a user gesture event handler (click, touchstart, etc.)
+// to satisfy browser security policies. Calling it outside a user gesture will fail.
+// All audio operations (yieldAudioData, yieldFramesData, etc.) require prior initialization.
+button.addEventListener('click', async () => {
+  // Initialize audio context - MUST be in user gesture context
+  await avatarView.controller.initializeAudioContext()
+  
+  // Stream audio chunks (must be mono PCM16 format matching configured sample rate)
+  const conversationId = avatarView.controller.yieldAudioData(
+    data: Uint8Array,               // Audio chunk data (PCM16 format)
+    isLast: boolean = false         // Whether this is the last chunk
+  )
+  // Returns: conversationId - Conversation ID for this audio session
+
+  // Stream animation keyframes (requires conversationId from audio data)
+  avatarView.controller.yieldFramesData(
+    keyframesDataArray: (Uint8Array | ArrayBuffer)[],  // Animation keyframes binary data array
+    conversationId: string                              // Conversation ID (required)
+  )
+})
+```
 
-// 打断对话
-avatarView.avatarController.interrupt()
+**⚠️ Important: Conversation ID (conversationId) Management**
 
-// 关闭连接
-avatarView.avatarController.close()
+**SDK Mode:**
+- `send()` returns a conversationId to distinguish each conversation round
+- `end=true` marks the end of a conversation round
 
-// 设置事件回调
-avatarView.avatarController.onConnectionState = (state: ConnectionState) => {}
-avatarView.avatarController.onAvatarState = (state: AvatarState) => {}
-avatarView.avatarController.onError = (error: Error) => {}
+**Host Mode:**
+- `yieldAudioData()` returns a conversationId (automatically generates if starting new session)
+   - `yieldFramesData()` requires a valid conversationId parameter
+   - Animation data with mismatched conversationId will be **discarded**
+   - Use `getCurrentConversationId()` to retrieve the current active conversationId
 
-// 注意：不支持 sendText() 方法，调用会抛出错误
+#### Common Methods (Both Modes)
+
+```typescript
+
+// Pause playback (from playing state)
+avatarView.controller.pause()
+
+// Resume playback (from paused state)
+await avatarView.controller.resume()
+
+// Interrupt current playback (stops and clears data)
+avatarView.controller.interrupt()
+
+// Clear all data and resources
+avatarView.controller.clear()
+
+// Get current conversation ID (for Host mode)
+const conversationId = avatarView.controller.getCurrentConversationId()
+// Returns: Current conversationId for the active audio session, or null if no active session
+
+// Volume control (affects only avatar audio player, not system volume)
+avatarView.controller.setVolume(0.5)  // Set volume to 50% (0.0 to 1.0)
+const currentVolume = avatarView.controller.getVolume()  // Get current volume (0.0 to 1.0)
+
+// Set event callbacks
+avatarView.controller.onConnectionState = (state: ConnectionState) => {} // SDK mode only
+avatarView.controller.onConversationState = (state: ConversationState) => {}
+avatarView.controller.onError = (error: AvatarError) => {} // Includes error.code for specific error type
 ```
 
-## 🔧 配置
+#### Avatar Transform Methods
+
+```typescript
+// Get or set avatar transform (position and scale in canvas)
+// Get current transform
+const currentTransform = avatarView.avatarTransform // { x: number, y: number, scale: number }
+
+// Set transform
+avatarView.avatarTransform = { x, y, scale }
+// - x: Horizontal offset in normalized coordinates (-1 to 1, where -1 = left edge, 0 = center, 1 = right edge)
+// - y: Vertical offset in normalized coordinates (-1 to 1, where -1 = bottom edge, 0 = center, 1 = top edge)
+// - scale: Scale factor (1.0 = original size, 2.0 = double size, 0.5 = half size)
+// Example:
+avatarView.avatarTransform = { x: 0, y: 0, scale: 1.0 }  // Center, original size
+avatarView.avatarTransform = { x: 0.5, y: 0, scale: 2.0 } // Right half, double size
+```
+
+**Important Notes:**
+- `start()` and `close()` are only available in SDK mode
+- `yieldAudioData()` and `yieldFramesData()` are only available in Host mode
+- `pause()`, `resume()`, `interrupt()`, `clear()`, `getCurrentConversationId()`, `setVolume()`, and `getVolume()` are available in both modes
+- The playback mode is determined when creating `AvatarView` and cannot be changed
+
+## 🔧 Configuration
 
 ### Configuration
 
 ```typescript
 interface Configuration {
   environment: Environment
+  drivingServiceMode?: DrivingServiceMode  // Optional, default is 'sdk' (SDK mode)
+  logLevel?: LogLevel  // Optional, default is 'off' (no logs)
+  audioFormat?: AudioFormat  // Optional, default is { channelCount: 1, sampleRate: 16000 }
+  characterApiBaseUrl?: string  // Optional, internal debug config, can be ignored
+}
+
+interface AudioFormat {
+  readonly channelCount: 1  // Fixed to 1 (mono)
+  readonly sampleRate: number  // Supported: 8000, 16000, 22050, 24000, 32000, 44100, 48000 Hz, default: 16000
 }
 ```
 
-**说明：**
-- `environment`: 指定环境（cn/us/test），SDK 会根据环境自动使用对应的 API 地址和 WebSocket 地址
-- `sessionToken`: 通过 `AvatarKit.setSessionToken()` 单独设置，而不是在 Configuration 中
+### LogLevel
+
+Control the verbosity of SDK logs:
+
+```typescript
+enum LogLevel {
+  off = 'off',        // Disable all logs
+  error = 'error',    // Only error logs
+  warning = 'warning', // Warning and error logs
+  all = 'all'         // All logs (info, warning, error) - default
+}
+```
 
+**Note:** `LogLevel.off` completely disables all logging, including error logs. Use with caution in production environments.
+
+**Description:**
+- `environment`: Specifies the environment (cn/intl), SDK will automatically use the corresponding server addresses based on the environment
+- `drivingServiceMode`: Specifies the driving service mode
+  - `DrivingServiceMode.sdk` (default): SDK mode - SDK handles network communication automatically
+  - `DrivingServiceMode.host`: Host mode - Host application provides audio and animation data
+- `logLevel`: Controls the verbosity of SDK logs
+  - `LogLevel.off` (default): Disable all logs
+  - `LogLevel.error`: Only error logs
+  - `LogLevel.warning`: Warning and error logs
+  - `LogLevel.all`: All logs (info, warning, error)
+- `audioFormat`: Configures audio sample rate and channel count
+  - `channelCount`: Fixed to 1 (mono channel)
+  - `sampleRate`: Audio sample rate in Hz (default: 16000)
+    - Supported values: 8000, 16000, 22050, 24000, 32000, 44100, 48000
+    - The configured sample rate will be used for both audio recording and playback
+- `characterApiBaseUrl`: Internal debug config, can be ignored
+- `sessionToken`: **Required for authentication**. Set separately via `AvatarSDK.setSessionToken()`, not in Configuration. See [Authentication](#-authentication) section for details
+
+```typescript
 enum Environment {
-  cn = 'cn',    // 中国区
-  us = 'us',    // 美国区
-  test = 'test' // 测试环境
+  cn = 'cn',    // China region
+  intl = 'intl',    // International region
 }
 ```
 
@@ -198,17 +811,17 @@ enum Environment {
 
 ```typescript
 interface CameraConfig {
-  position: [number, number, number]  // 相机位置
-  target: [number, number, number]    // 相机目标
-  fov: number                         // 视野角度
-  near: number                        // 近裁剪面
-  far: number                         // 远裁剪面
-  up?: [number, number, number]       // 上方向
-  aspect?: number                     // 宽高比
+  position: [number, number, number]  // Camera position
+  target: [number, number, number]    // Camera target
+  fov: number                         // Field of view angle
+  near: number                        // Near clipping plane
+  far: number                         // Far clipping plane
+  up?: [number, number, number]       // Up direction
+  aspect?: number                     // Aspect ratio
 }
 ```
 
-## 📊 状态管理
+## 📊 State Management
 
 ### ConnectionState
 
@@ -221,89 +834,42 @@ enum ConnectionState {
 }
 ```
 
-### AvatarState
+### ConversationState
 
 ```typescript
-enum AvatarState {
-  idle = 'idle',      // 空闲状态，呈现呼吸态
-  active = 'active',  // 活跃中，等待可播放内容
-  playing = 'playing' // 播放中
+enum ConversationState {
+  idle = 'idle',      // Idle state (breathing animation)
+  playing = 'playing', // Playing state (active conversation)
+  pausing = 'pausing' // Pausing state (paused during playback)
 }
 ```
 
-## 🎨 渲染系统
+**State Description:**
+- `idle`: Avatar is in idle state (breathing animation), waiting for conversation to start
+- `playing`: Avatar is playing conversation content (including during transition animations)
+- `pausing`: Avatar playback is paused (e.g., when `end=false` and waiting for more audio data)
 
-SDK 支持两种渲染后端：
+**Note:** During transition animations, the target state is notified immediately:
+- When transitioning from `idle` to `playing`, the `playing` state is notified immediately
+- When transitioning from `playing` to `idle`, the `idle` state is notified immediately
 
-- **WebGPU** - 现代浏览器的高性能渲染
-- **WebGL** - 兼容性更好的传统渲染
+## 🎨 Rendering System
 
-渲染系统会自动选择最佳的后端，无需手动配置。
+The SDK automatically selects the best rendering backend for your browser, no manual configuration needed.
 
-## 🔍 调试和监控
+## 🚨 Error Handling
 
-### 日志系统
+### AvatarError
 
-SDK 内置了完整的日志系统，支持不同级别的日志输出：
+The SDK uses custom error types, providing more detailed error information:
 
 ```typescript
-import { logger } from '@spatialwalk/avatarkit'
-
-// 设置日志级别
-logger.setLevel('verbose') // 'basic' | 'verbose'
-
-// 手动日志输出
-logger.log('Info message')
-logger.warn('Warning message')
-logger.error('Error message')
-```
-
-### 性能监控
-
-SDK 提供了性能监控接口，可以监控渲染性能：
-
-```typescript
-// 获取渲染性能统计
-const stats = avatarView.getPerformanceStats()
-
-if (stats) {
-  console.log(`渲染耗时: ${stats.renderTime.toFixed(2)}ms`)
-  console.log(`排序耗时: ${stats.sortTime.toFixed(2)}ms`)
-  console.log(`渲染后端: ${stats.backend}`)
-  
-  // 计算帧率
-  const fps = 1000 / stats.renderTime
-  console.log(`帧率: ${fps.toFixed(2)} FPS`)
-}
-
-// 定期监控性能
-setInterval(() => {
-  const stats = avatarView.getPerformanceStats()
-  if (stats) {
-    // 发送到监控服务或显示在 UI 上
-    console.log('Performance:', stats)
-  }
-}, 1000)
-```
-
-**性能统计说明**：
-- `renderTime`: 总渲染耗时（毫秒），包含排序和 GPU 渲染
-- `sortTime`: 排序耗时（毫秒），使用 Radix Sort 算法对点云进行深度排序
-- `backend`: 当前使用的渲染后端（`'webgpu'` | `'webgl'` | `null`）
-
-## 🚨 错误处理
-
-### SPAvatarError
-
-SDK 使用自定义错误类型，提供更详细的错误信息：
-
-```typescript
-import { SPAvatarError } from '@spatialwalk/avatarkit'
+import { AvatarError } from '@spatialwalk/avatarkit'
 
 try {
-  await avatarView.avatarController.start()
+  await avatarView.controller.start()
 } catch (error) {
-  if (error instanceof SPAvatarError) {
+  if (error instanceof AvatarError) {
     console.error('SDK Error:', error.message, error.code)
   } else {
     console.error('Unknown error:', error)
@@ -311,65 +877,112 @@ try {
 }
 ```
 
-### 错误回调
+### Error Callbacks
 
 ```typescript
-avatarView.avatarController.onError = (error: Error) => {
-  console.error('AvatarController error:', error)
-  // 处理错误，比如重连、用户提示等
+import { AvatarError } from '@spatialwalk/avatarkit'
+
+avatarView.controller.onError = (error: AvatarError) => {
+  console.error('Error:', error.code, error.message)
 }
 ```
 
-## 🔄 资源管理
-
-### 生命周期管理
+`error.code` values (from `ErrorCode` enum):
+
+| Code | Description | Trigger |
+|------|-------------|---------|
+| **Authentication & Authorization** | | |
+| `appIDUnrecognized` | App ID not recognized | Reserved |
+| `sessionTokenInvalid` | Token invalid or appId mismatch | WebSocket close code 4010 |
+| `sessionTokenExpired` | Token expired | WebSocket close code 4010 |
+| `insufficientBalance` | Insufficient balance | WebSocket close code 4001 |
+| `concurrentLimitExceeded` | Concurrent connection limit exceeded | WebSocket close code 4003 |
+| **Resource Loading** | | |
+| `avatarIDUnrecognized` | Avatar ID not found | Server error |
+| `failedToFetchAvatarMetadata` | Metadata fetch failed | Network/server error |
+| `failedToDownloadAvatarAssets` | Asset download failed | Network/server error |
+| **Connection** | | |
+| `websocketError` | WebSocket handshake or network error | Connection failure |
+| `websocketClosedAbnormally` | Connection closed abnormally | Close code 1006 |
+| `websocketClosedUnexpected` | Unexpected close code | Unknown close code |
+| `sessionTimeout` | Session timeout | WebSocket close code 4002 |
+| `connectionInProgress` | Connection already in progress | Duplicate `start()` call |
+| **Playback** | | |
+| `networkLayerNotAvailable` | Network layer not available | `send()` in host mode |
+| `playbackStartFailed` | Failed to start playback | Internal error |
+| `playbackInitFailed` | Playback initialization failed | Internal error |
+| `audioOnlyInitFailed` | Audio-only playback init failed | Fallback mode error |
+| `noAudio` | No audio data to play | Empty audio input |
+| `audioContextNotInitialized` | Audio context not initialized | `send()` before `initializeAudioContext()` |
+| `animationPlayerNotInitialized` | Animation player not initialized | Internal error |
+| **Server** | | |
+| `serverError` | Server-side error | Server MESSAGE_SERVER_ERROR |
+
+## 🔄 Resource Management
+
+### Lifecycle Management
+
+#### SDK Mode Lifecycle
 
 ```typescript
-// 初始化
+// Initialize
+const container = document.getElementById('avatar-container')
 const avatarView = new AvatarView(avatar, container)
-await avatarView.avatarController.start()
+await avatarView.controller.start()
 
-// 使用
-avatarView.avatarController.send(audioData, false)
+// Use
+avatarView.controller.send(audioData, false)
 
-// 清理
-avatarView.dispose() // 自动清理所有资源
+// Cleanup - dispose() automatically cleans up all resources including connections
+avatarView.dispose()
 ```
 
-### 内存优化
+#### Host Mode Lifecycle
 
-- SDK 自动管理 WASM 内存分配
-- 支持角色和动画资源的动态加载/卸载
-- 提供内存使用监控接口
+```typescript
+// Initialize
+const container = document.getElementById('avatar-container')
+const avatarView = new AvatarView(avatar, container)
+
+// Use
+const conversationId = avatarView.controller.yieldAudioData(audioChunk, false)
+avatarView.controller.yieldFramesData(keyframesDataArray, conversationId)
+
+// Cleanup - dispose() automatically cleans up all resources including playback data
+avatarView.dispose()
+```
 
-### 音频数据发送
+**⚠️ Important Notes:**
+- `dispose()` automatically cleans up all resources, including:
+  - Network connections (SDK mode)
+  - Playback data and animation resources (both modes)
+  - Render system and canvas elements
+  - All event listeners and callbacks
+- Not properly calling `dispose()` may cause resource leaks and rendering errors
+- If you need to manually close connections or clear playback data before disposing, you can call `avatarView.controller.close()` (SDK mode) or `avatarView.controller.clear()` (both modes) first, but it's not required as `dispose()` handles this automatically
 
-`send()` 方法接收 `ArrayBuffer` 格式的音频数据：
+### Memory Optimization
 
-**使用说明：**
-- `audioData`: 音频数据（ArrayBuffer 格式）
-- `end=false`（默认）- 正常发送音频数据，服务端会积累音频数据，积累到一定量后会自动返回动画数据并开始同步播放动画和音频
-- `end=true` - 立即返回动画数据，不再积累，用于结束当前对话或需要立即响应的场景
-- **重要**：不需要等待 `end=true` 才开始播放，积累到一定音频数据后就会自动开始播放
+- SDK automatically manages memory allocation
+- Supports dynamic loading/unloading of avatar and animation resources
 
-## 🌐 浏览器兼容性
+## 🌐 Browser Compatibility
 
-- **Chrome/Edge** 90+ (推荐 WebGPU)
+- **Chrome/Edge** 90+ (WebGPU recommended)
 - **Firefox** 90+ (WebGL)
 - **Safari** 14+ (WebGL)
-- **移动端** iOS 14+, Android 8+
+- **Mobile** iOS 14+, Android 8+
 
-## 📝 许可证
+## 📝 License
 
 MIT License
 
-## 🤝 贡献
+## 🤝 Contributing
 
-欢迎提交 Issue 和 Pull Request！
+Issues and Pull Requests are welcome!
 
-## 📞 支持
+## 📞 Support
 
-如有问题，请联系：
-- 邮箱：support@spavatar.com
-- 文档：https://docs.spavatar.com
-- GitHub：https://github.com/spavatar/sdk
+For questions, please contact:
+- Email: code@spatialwalk.net
+- Documentation: https://docs.spatialreal.ai