@spatialwalk/avatarkit 1.0.0-beta.7 → 1.0.0-beta.71

package/README.md

@@ -1,4 +1,4 @@
-# SPAvatarKit SDK
+# AvatarKit SDK
 
 Real-time virtual avatar rendering SDK based on 3D Gaussian Splatting, supporting audio-driven animation rendering and high-quality 3D rendering.
 
@@ -6,6 +6,7 @@ Real-time virtual avatar rendering SDK based on 3D Gaussian Splatting, supportin
 
 - **3D Gaussian Splatting Rendering** - Based on the latest point cloud rendering technology, providing high-quality 3D virtual avatars
 - **Audio-Driven Real-Time Animation Rendering** - Users provide audio data, SDK handles receiving animation data and rendering
+- **Multi-Avatar Support** - Support multiple avatar instances simultaneously, each with independent state and rendering
 - **WebGPU/WebGL Dual Rendering Backend** - Automatically selects the best rendering backend for compatibility
 - **WASM High-Performance Computing** - Uses C++ compiled WebAssembly modules for geometric calculations
 - **TypeScript Support** - Complete type definitions and IntelliSense
@@ -17,90 +18,179 @@ Real-time virtual avatar rendering SDK based on 3D Gaussian Splatting, supportin
 npm install @spatialwalk/avatarkit
 ```
 
+## 🔧 Vite 配置（推荐）
+
+如果你使用 Vite 作为构建工具，强烈推荐使用我们的 Vite 插件来自动处理 WASM 文件配置。插件会自动处理所有必要的配置，让你无需手动设置。
+
+### 使用插件
+
+在 `vite.config.ts` 中添加插件：
+
+```typescript
+import { defineConfig } from 'vite'
+import { avatarkitVitePlugin } from '@spatialwalk/avatarkit/vite'
+
+export default defineConfig({
+  plugins: [
+    avatarkitVitePlugin(), // 添加这一行即可
+  ],
+})
+```
+
+### 插件功能
+
+插件会自动处理：
+
+- ✅ **开发服务器**：自动设置 WASM 文件的正确 MIME 类型 (`application/wasm`)
+- ✅ **构建时**：自动复制 WASM 文件到 `dist/assets/` 目录
+  - 智能检测：从 JS glue 文件中提取引用的 WASM 文件名（包括 hash）
+  - 自动匹配：确保复制的 WASM 文件与 JS glue 文件中的引用匹配
+  - 支持 hash：正确处理带 hash 的 WASM 文件（如 `avatar_core_wasm-{hash}.wasm`）
+- ✅ **WASM JS Glue**：自动复制 WASM JS glue 文件到 `dist/assets/` 目录
+- ✅ **Cloudflare Pages**：自动生成 `_headers` 文件，确保 WASM 文件使用正确的 MIME 类型
+- ✅ **Vite 配置**：自动配置 `optimizeDeps`、`assetsInclude`、`assetsInlineLimit` 等选项
+
+### 手动配置（不使用插件）
+
+如果你不使用 Vite 插件，需要手动配置以下内容：
+
+```typescript
+// vite.config.ts
+export default defineConfig({
+  optimizeDeps: {
+    exclude: ['@spatialwalk/avatarkit'],
+  },
+  assetsInclude: ['**/*.wasm'],
+  build: {
+    assetsInlineLimit: 0,
+    rollupOptions: {
+      output: {
+        assetFileNames: (assetInfo) => {
+          if (assetInfo.name?.endsWith('.wasm')) {
+            return 'assets/[name][extname]'
+          }
+          return 'assets/[name]-[hash][extname]'
+        },
+      },
+    },
+  },
+  // 开发服务器需要手动配置中间件设置 WASM MIME 类型
+  configureServer(server) {
+    server.middlewares.use((req, res, next) => {
+      if (req.url?.endsWith('.wasm')) {
+        res.setHeader('Content-Type', 'application/wasm')
+      }
+      next()
+    })
+  },
+})
+```
+
 ## 🎯 Quick Start
 
+### ⚠️ Important: Audio Context Initialization
+
+**Before using any audio-related features, you MUST initialize the audio context in a user gesture context** (e.g., `click`, `touchstart` event handlers). This is required by browser security policies. Calling `initializeAudioContext()` outside a user gesture will fail.
+
 ### Basic Usage
 
 ```typescript
 import { 
-  AvatarKit, 
+  AvatarSDK, 
   AvatarManager, 
   AvatarView,
   Configuration,
-  Environment 
+  Environment,
+  DrivingServiceMode,
+  LogLevel
 } from '@spatialwalk/avatarkit'
 
 // 1. Initialize SDK
+
 const configuration: Configuration = {
-  environment: Environment.test,
+  environment: Environment.cn,
+  drivingServiceMode: DrivingServiceMode.sdk, // Optional, 'sdk' is default
+  // - DrivingServiceMode.sdk: SDK mode - SDK handles WebSocket communication
+  // - DrivingServiceMode.host: Host mode - Host app provides audio and animation data
+  logLevel: LogLevel.off, // Optional, 'off' is default
+  // - LogLevel.off: Disable all logs
+  // - LogLevel.error: Only error logs
+  // - LogLevel.warning: Warning and error logs
+  // - LogLevel.all: All logs (info, warning, error)
+  audioFormat: { // Optional, default is { channelCount: 1, sampleRate: 16000 }
+    channelCount: 1, // Fixed to 1 (mono)
+    sampleRate: 16000 // Supported: 8000, 16000, 22050, 24000, 32000, 44100, 48000 Hz
+  }
+  // characterApiBaseUrl: 'https://custom-api.example.com' // Optional, internal debug config, can be ignored
 }
 
-await AvatarKit.initialize('your-app-id', configuration)
+await AvatarSDK.initialize('your-app-id', configuration)
 
 // Set sessionToken (if needed, call separately)
-// AvatarKit.setSessionToken('your-session-token')
+// AvatarSDK.setSessionToken('your-session-token')
 
-// 2. Load character
-const avatarManager = new AvatarManager()
+// 2. Load avatar
+const avatarManager = AvatarManager.shared
 const avatar = await avatarManager.load('character-id', (progress) => {
   console.log(`Loading progress: ${progress.progress}%`)
 })
 
 // 3. Create view (automatically creates Canvas and AvatarController)
-// Network mode (default)
+// The playback mode is determined by drivingServiceMode in AvatarSDK configuration
+// - DrivingServiceMode.sdk: SDK mode - SDK handles WebSocket communication
+// - DrivingServiceMode.host: Host mode - Host app provides audio and animation data
 const container = document.getElementById('avatar-container')
-const avatarView = new AvatarView(avatar, {
-  container: container,
-  playbackMode: 'network' // Optional, 'network' is default
+const avatarView = new AvatarView(avatar, container)
+
+// 4. ⚠️ CRITICAL: Initialize audio context (MUST be called in user gesture context)
+// This method MUST be called within a user gesture event handler (click, touchstart, etc.)
+// to satisfy browser security policies. Calling it outside a user gesture will fail.
+button.addEventListener('click', async () => {
+  // Initialize audio context - MUST be in user gesture context
+  await avatarView.controller.initializeAudioContext()
+  
+  // 5. Start real-time communication (SDK mode only)
+  await avatarView.controller.start()
+  
+  // 6. Send audio data (SDK mode, must be mono PCM16 format matching configured sample rate)
+  // audioData: ArrayBuffer or Uint8Array containing PCM16 audio samples
+  // - PCM files: Can be directly read as ArrayBuffer
+  // - WAV files: Extract PCM data from WAV format (may require resampling)
+  // - MP3 files: Decode first (e.g., using AudioContext.decodeAudioData()), then convert to PCM16
+  const audioData = new ArrayBuffer(1024) // Placeholder: Replace with actual PCM16 audio data
+  avatarView.controller.send(audioData, false) // Send audio data
+  avatarView.controller.send(audioData, true) // end=true marks the end of current conversation round
 })
-
-// 4. Start real-time communication (network mode only)
-await avatarView.avatarController.start()
-
-// 5. Send audio data (network mode)
-// ⚠️ Important: Audio must be 16kHz mono PCM16 format
-// If audio is Uint8Array, you can use slice().buffer to convert to ArrayBuffer
-const audioUint8 = new Uint8Array(1024) // Example: 16kHz PCM16 audio data (512 samples = 1024 bytes)
-const audioData = audioUint8.slice().buffer // Simplified conversion, works for ArrayBuffer and SharedArrayBuffer
-avatarView.avatarController.send(audioData, false) // Send audio data, will automatically start playing after accumulating enough data
-avatarView.avatarController.send(audioData, true) // end=true means immediately return animation data, no longer accumulating
 ```
 
-### External Data Mode Example
+### Host Mode Example
 
 ```typescript
-import { AvatarPlaybackMode } from '@spatialwalk/avatarkit'
 
-// 1-3. Same as network mode (initialize SDK, load character)
+// 1-3. Same as SDK mode (initialize SDK, load avatar)
 
-// 3. Create view with external data mode
+// 3. Create view with Host mode
 const container = document.getElementById('avatar-container')
-const avatarView = new AvatarView(avatar, {
-  container: container,
-  playbackMode: AvatarPlaybackMode.external
-})
-
-// 4. Start playback with initial data (obtained from your service)
-// Note: Audio and animation data should be obtained from your backend service
-const initialAudioChunks = [{ data: audioData1, isLast: false }, { data: audioData2, isLast: false }]
-const initialKeyframes = animationData1 // Animation keyframes from your service
-
-await avatarView.avatarController.play(initialAudioChunks, initialKeyframes)
-
-// 5. Stream additional data as needed
-avatarView.avatarController.sendAudioChunk(audioData3, false)
-avatarView.avatarController.sendKeyframes(animationData2)
+const avatarView = new AvatarView(avatar, container)
+
+// 4. ⚠️ CRITICAL: Initialize audio context (MUST be called in user gesture context)
+// This method MUST be called within a user gesture event handler (click, touchstart, etc.)
+// to satisfy browser security policies. Calling it outside a user gesture will fail.
+button.addEventListener('click', async () => {
+  // Initialize audio context - MUST be in user gesture context
+  await avatarView.controller.initializeAudioContext()
+  
+  // 5. Host Mode Workflow:
+  // Send audio data first to get conversationId, then use it to send animation data
+  const conversationId = avatarView.controller.yieldAudioData(audioData, false)
+  avatarView.controller.yieldFramesData(animationDataArray, conversationId) // animationDataArray: (Uint8Array | ArrayBuffer)[]
 ```
 
 ### Complete Examples
 
-Check the example code in the GitHub repository for complete usage flows for both modes.
-
-**Example Project:** [Avatarkit-web-demo](https://github.com/spatialwalk/Avatarkit-web-demo)
-
-This repository contains complete examples for Vanilla JS, Vue 3, and React, demonstrating:
-- Network mode: Real-time audio input with automatic animation data reception
-- External data mode: Custom data sources with manual audio/animation data management
+This SDK supports two usage modes:
+- SDK mode: Real-time audio input with automatic animation data reception
+- Host mode: Custom data sources with manual audio/animation data management
 
 ## 🏗️ Architecture Overview
 
@@ -110,47 +200,60 @@ The SDK uses a three-layer architecture for clear separation of concerns:
 
 1. **Rendering Layer (AvatarView)** - Responsible for 3D rendering only
 2. **Playback Layer (AvatarController)** - Manages audio/animation synchronization and playback
-3. **Network Layer (NetworkLayer)** - Handles WebSocket communication (only in network mode)
+3. **Network Layer** - Handles WebSocket communication (only in SDK mode, internal implementation)
 
 ### Core Components
 
-- **AvatarKit** - SDK initialization and management
-- **AvatarManager** - Character resource loading and management
+- **AvatarSDK** - SDK initialization and management
+- **AvatarManager** - Avatar resource loading and management
 - **AvatarView** - 3D rendering view (rendering layer)
 - **AvatarController** - Audio/animation playback controller (playback layer)
-- **NetworkLayer** - WebSocket communication (network layer, automatically composed in network mode)
-- **AvatarCoreAdapter** - WASM module adapter
 
 ### Playback Modes
 
-The SDK supports two playback modes, configured when creating `AvatarView`:
+The SDK supports two playback modes, configured in `AvatarSDK.initialize()`:
 
-#### 1. Network Mode (Default)
+#### 1. SDK Mode (Default)
+- Configured via `drivingServiceMode: DrivingServiceMode.sdk` in `AvatarSDK.initialize()`
 - SDK handles WebSocket communication automatically
 - Send audio data via `AvatarController.send()`
 - SDK receives animation data from backend and synchronizes playback
 - Best for: Real-time audio input scenarios
 
-#### 2. External Data Mode
-- External components manage their own network/data fetching
-- External components provide both audio and animation data
+#### 2. Host Mode
+- Configured via `drivingServiceMode: DrivingServiceMode.host` in `AvatarSDK.initialize()`
+- Host application manages its own network/data fetching
+- Host application provides both audio and animation data
 - SDK only handles synchronized playback
 - Best for: Custom data sources, pre-recorded content, or custom network implementations
 
+**Note:** The playback mode is determined by `drivingServiceMode` in `AvatarSDK.initialize()` configuration.
+
+### Fallback Mechanism
+
+The SDK includes a fallback mechanism to ensure audio playback continues even when animation data is unavailable:
+
+- **SDK Mode Connection Failure**: If WebSocket connection fails to establish within 15 seconds, the SDK automatically enters fallback mode. In this mode, audio data can still be sent and will play normally, even though no animation data will be received from the server. This ensures that audio playback is not interrupted even when the service connection fails.
+- **SDK Mode Server Error**: If the server returns an error after connection is established, the SDK automatically enters audio-only mode for that session and continues playing audio independently.
+- **Host Mode**: If empty animation data is provided (empty array or undefined), the SDK automatically enters audio-only mode.
+- Once in audio-only mode, any subsequent animation data for that session will be ignored, and only audio will continue playing.
+- The fallback mode is interruptible, just like normal playback mode.
+- Connection state callbacks (`onConnectionState`) will notify you when connection fails or times out, allowing you to handle the fallback state appropriately.
+
 ### Data Flow
 
-#### Network Mode Flow
+#### SDK Mode Flow
 
 ```
 User audio input (16kHz mono PCM16) 
     ↓
 AvatarController.send() 
     ↓
-NetworkLayer → WebSocket → Backend processing
+WebSocket → Backend processing
     ↓
 Backend returns animation data (FLAME keyframes) 
     ↓
-NetworkLayer → AvatarController → AnimationPlayer
+AvatarController → AnimationPlayer
     ↓
 FLAME parameters → AvatarCore.computeFrameFlatFromParams() → Splat data
     ↓
@@ -159,15 +262,14 @@ AvatarController (playback loop) → AvatarView.renderRealtimeFrame()
 RenderSystem → WebGPU/WebGL → Canvas rendering
 ```
 
-#### External Data Mode Flow
+#### Host Mode Flow
 
 ```
 External data source (audio + animation)
     ↓
-AvatarController.play(initialAudio, initialKeyframes) // Start playback
+AvatarController.yieldAudioData(audioChunk) // Returns conversationId
     ↓
-AvatarController.sendAudioChunk() // Stream additional audio
-AvatarController.sendKeyframes() // Stream additional animation
+AvatarController.yieldFramesData(keyframesDataArray, conversationId) // keyframesDataArray: (Uint8Array | ArrayBuffer)[] - each element is a protobuf encoded Message
     ↓
 AvatarController → AnimationPlayer (synchronized playback)
     ↓
@@ -178,205 +280,348 @@ AvatarController (playback loop) → AvatarView.renderRealtimeFrame()
 RenderSystem → WebGPU/WebGL → Canvas rendering
 ```
 
-**Note:** 
-- In network mode, users provide audio data, SDK handles network communication and animation data reception
-- In external data mode, users provide both audio and animation data, SDK handles synchronized playback only
-
 ### Audio Format Requirements
 
-**⚠️ Important:** The SDK requires audio data to be in **16kHz mono PCM16** format:
+**⚠️ Important:** The SDK requires audio data to be in **mono PCM16** format:
 
-- **Sample Rate**: 16kHz (16000 Hz) - This is a backend requirement
-- **Channels**: Mono (single channel)
+- **Sample Rate**: Configurable via `audioFormat.sampleRate` in SDK initialization (default: 16000 Hz)
+  - Supported sample rates: 8000, 16000, 22050, 24000, 32000, 44100, 48000 Hz
+  - The configured sample rate will be used for both audio recording and playback
+- **Channels**: Mono (single channel) - Fixed to 1 channel
 - **Format**: PCM16 (16-bit signed integer, little-endian)
 - **Byte Order**: Little-endian
 
 **Audio Data Format:**
-- Each sample is 2 bytes (16-bit)
+- Each sample is 2 bytes (16-bit signed integer, little-endian)
 - Audio data should be provided as `ArrayBuffer` or `Uint8Array`
-- For example: 1 second of audio = 16000 samples × 2 bytes = 32000 bytes
+- For example, with 16kHz sample rate: 1 second of audio = 16000 samples × 2 bytes = 32000 bytes
+- For 48kHz sample rate: 1 second of audio = 48000 samples × 2 bytes = 96000 bytes
+
+**Audio Data Source:**
+The `audioData` parameter represents raw PCM16 audio samples in the configured sample rate and mono format. Common audio sources include:
+- **PCM files**: Raw PCM16 files can be directly read as `ArrayBuffer` or `Uint8Array` and sent to the SDK (ensure sample rate matches configuration)
+- **WAV files**: WAV files contain PCM16 audio data in their data chunk. After extracting the PCM data from the WAV file format, it can be sent to the SDK (may require resampling if sample rate differs)
+- **MP3 files**: MP3 files need to be decoded first (e.g., using `AudioContext.decodeAudioData()` or a decoder library), then converted from the decoded format to PCM16 before sending to the SDK
+- **Microphone input**: Real-time microphone audio needs to be captured and converted to PCM16 format at the configured sample rate before sending
+- **Other audio sources**: Any audio source must be converted to mono PCM16 format at the configured sample rate before sending
+
+**Example: Processing WAV and MP3 Files:**
+```typescript
+// WAV file processing
+async function processWAVFile(wavFile: File): Promise<ArrayBuffer> {
+  const arrayBuffer = await wavFile.arrayBuffer()
+  const view = new DataView(arrayBuffer)
+  
+  // WAV format: Skip header (usually 44 bytes for standard WAV)
+  // Check RIFF header
+  if (view.getUint32(0, true) !== 0x46464952) { // "RIFF"
+    throw new Error('Invalid WAV file')
+  }
+  
+  // Find "data" chunk (offset may vary)
+  let dataOffset = 44 // Standard WAV header size
+  // For non-standard WAV files, you may need to search for "data" chunk
+  // This is a simplified example - production code should parse chunks properly
+  
+  const pcmData = arrayBuffer.slice(dataOffset)
+  return pcmData
+}
+
+// MP3 file processing
+async function processMP3File(mp3File: File, targetSampleRate: number): Promise<ArrayBuffer> {
+  const arrayBuffer = await mp3File.arrayBuffer()
+  const audioContext = new AudioContext({ sampleRate: targetSampleRate })
+  
+  // Decode MP3 to AudioBuffer
+  const audioBuffer = await audioContext.decodeAudioData(arrayBuffer.slice(0))
+  
+  // Convert AudioBuffer to PCM16 ArrayBuffer
+  const length = audioBuffer.length
+  const channels = audioBuffer.numberOfChannels
+  const pcm16Buffer = new ArrayBuffer(length * 2)
+  const pcm16View = new DataView(pcm16Buffer)
+  
+  // Mix down to mono if stereo
+  const sourceData = channels === 1 
+    ? audioBuffer.getChannelData(0)
+    : new Float32Array(length)
+  
+  if (channels > 1) {
+    const leftChannel = audioBuffer.getChannelData(0)
+    const rightChannel = audioBuffer.getChannelData(1)
+    for (let i = 0; i < length; i++) {
+      sourceData[i] = (leftChannel[i] + rightChannel[i]) / 2 // Mix to mono
+    }
+  }
+  
+  // Convert float32 (-1.0 to 1.0) to int16 (-32768 to 32767)
+  for (let i = 0; i < length; i++) {
+    const sample = Math.max(-1, Math.min(1, sourceData[i])) // Clamp
+    const int16Sample = sample < 0 ? sample * 0x8000 : sample * 0x7FFF
+    pcm16View.setInt16(i * 2, int16Sample, true) // little-endian
+  }
+  
+  audioContext.close()
+  return pcm16Buffer
+}
+
+// Usage example:
+// const wavPcmData = await processWAVFile(wavFile)
+// avatarView.controller.send(wavPcmData, false)
+//
+// const mp3PcmData = await processMP3File(mp3File, 16000) // 16kHz
+// avatarView.controller.send(mp3PcmData, false)
+```
 
 **Resampling:**
-- If your audio source is at a different sample rate (e.g., 24kHz, 48kHz), you must resample it to 16kHz before sending to the SDK
+- If your audio source is at a different sample rate, you must resample it to match the configured sample rate before sending to the SDK
 - For high-quality resampling, we recommend using Web Audio API's `OfflineAudioContext` with anti-aliasing filtering
 - See example projects for resampling implementation
 
+**Configuration Example:**
+```typescript
+const configuration: Configuration = {
+  environment: Environment.cn,
+  audioFormat: {
+    channelCount: 1, // Fixed to 1 (mono)
+    sampleRate: 48000 // Choose from: 8000, 16000, 22050, 24000, 32000, 44100, 48000
+  }
+}
+```
+
 ## 📚 API Reference
 
-### AvatarKit
+### AvatarSDK
 
 The core management class of the SDK, responsible for initialization and global configuration.
 
 ```typescript
 // Initialize SDK
-await AvatarKit.initialize(appId: string, configuration: Configuration)
+await AvatarSDK.initialize(appId: string, configuration: Configuration)
 
 // Check initialization status
-const isInitialized = AvatarKit.isInitialized
+const isInitialized = AvatarSDK.isInitialized
+
+// Get initialized app ID
+const appId = AvatarSDK.appId
+
+// Get configuration
+const config = AvatarSDK.configuration
+
+// Set sessionToken (if needed, call separately)
+AvatarSDK.setSessionToken('your-session-token')
+
+// Set userId (optional, for telemetry)
+AvatarSDK.setUserId('user-id')
+
+// Get sessionToken
+const sessionToken = AvatarSDK.sessionToken
+
+// Get userId
+const userId = AvatarSDK.userId
+
+// Get SDK version
+const version = AvatarSDK.version
 
 // Cleanup resources (must be called when no longer in use)
-AvatarKit.cleanup()
+AvatarSDK.cleanup()
 ```
 
 ### AvatarManager
 
-Character resource manager, responsible for downloading, caching, and loading character data.
+Avatar resource manager, responsible for downloading, caching, and loading avatar data. Use the singleton instance via `AvatarManager.shared`.
 
 ```typescript
-const manager = new AvatarManager()
+// Get singleton instance
+const manager = AvatarManager.shared
 
-// Load character
+// Load avatar
 const avatar = await manager.load(
   characterId: string, 
   onProgress?: (progress: LoadProgressInfo) => void
 )
 
 // Clear cache
-manager.clearCache()
+manager.clearAll()
 ```
 
 ### AvatarView
 
 3D rendering view (rendering layer), responsible for 3D rendering only. Internally automatically creates and manages `AvatarController`.
 
-**⚠️ Important Limitation:** Currently, the SDK only supports one AvatarView instance at a time. If you need to switch characters, you must first call the `dispose()` method to clean up the current AvatarView, then create a new instance.
+```typescript
+constructor(avatar: Avatar, container: HTMLElement)
+```
 
-**Playback Mode Configuration:**
+**Parameters:**
+- `avatar`: Avatar instance
+- `container`: Canvas container element (required)
+  - Canvas automatically uses the full size of the container (width and height)
+  - Canvas aspect ratio adapts to container size - set container size to control aspect ratio
+  - Canvas will be automatically added to the container
+  - SDK automatically handles resize events via ResizeObserver
+
+**Playback Mode:**
+- The playback mode is determined by `drivingServiceMode` in `AvatarSDK.initialize()` configuration
 - The playback mode is fixed when creating `AvatarView` and persists throughout its lifecycle
 - Cannot be changed after creation
 
 ```typescript
-import { AvatarPlaybackMode } from '@spatialwalk/avatarkit'
-
 // Create view (Canvas is automatically added to container)
-// Network mode (default)
 const container = document.getElementById('avatar-container')
-const avatarView = new AvatarView(avatar: Avatar, {
-  container: container,
-  playbackMode: AvatarPlaybackMode.network // Optional, default is 'network'
-})
-
-// External data mode
-const avatarView = new AvatarView(avatar: Avatar, {
-  container: container,
-  playbackMode: AvatarPlaybackMode.external
-})
+const avatarView = new AvatarView(avatar, container)
 
-// Get Canvas element
-const canvas = avatarView.getCanvas()
+// Wait for first frame to render
+avatarView.onFirstRendering = () => {
+  // First frame rendered
+}
 
-// Get playback mode
-const mode = avatarView.playbackMode // 'network' | 'external'
+// Get or set avatar transform (position and scale)
+// Get current transform
+const currentTransform = avatarView.transform // { x: number, y: number, scale: number }
 
-// Update camera configuration
-avatarView.updateCameraConfig(cameraConfig: CameraConfig)
+// Set transform
+avatarView.transform = { x, y, scale }
+// - x: Horizontal offset in normalized coordinates (-1 to 1, where -1 = left edge, 0 = center, 1 = right edge)
+// - y: Vertical offset in normalized coordinates (-1 to 1, where -1 = bottom edge, 0 = center, 1 = top edge)
+// - scale: Scale factor (1.0 = original size, 2.0 = double size, 0.5 = half size)
 
-// Cleanup resources (must be called before switching characters)
+// Cleanup resources (must be called before switching avatars)
 avatarView.dispose()
 ```
 
-**Character Switching Example:**
+**Avatar Switching Example:**
 
 ```typescript
-// Before switching characters, must clean up old AvatarView first
+// To switch avatars, simply dispose the old view and create a new one
 if (currentAvatarView) {
   currentAvatarView.dispose()
-  currentAvatarView = null
 }
 
-// Load new character
+// Load new avatar
 const newAvatar = await avatarManager.load('new-character-id')
 
-// Create new AvatarView (with same or different playback mode)
-currentAvatarView = new AvatarView(newAvatar, {
-  container: container,
-  playbackMode: AvatarPlaybackMode.network
-})
+// Create new AvatarView
+currentAvatarView = new AvatarView(newAvatar, container)
 
-// Network mode: start connection
-if (currentAvatarView.playbackMode === AvatarPlaybackMode.network) {
-  await currentAvatarView.avatarController.start()
-}
+// SDK mode: start connection (will throw error if not in SDK mode)
+  await currentAvatarView.controller.start()
 ```
 
 ### AvatarController
 
-Audio/animation playback controller (playback layer), manages synchronized playback of audio and animation. Automatically composes `NetworkLayer` in network mode.
+Audio/animation playback controller (playback layer), manages synchronized playback of audio and animation. Automatically handles WebSocket communication in SDK mode.
 
 **Two Usage Patterns:**
 
-#### Network Mode Methods
+#### SDK Mode Methods
 
 ```typescript
-// Start WebSocket service
-await avatarView.avatarController.start()
-
-// Send audio data (SDK handles receiving animation data automatically)
-avatarView.avatarController.send(audioData: ArrayBuffer, end: boolean)
-// audioData: Audio data (ArrayBuffer format, must be 16kHz mono PCM16)
-//   - Sample rate: 16kHz (16000 Hz) - backend requirement
-//   - Format: PCM16 (16-bit signed integer, little-endian)
-//   - Channels: Mono (single channel)
-//   - Example: 1 second = 16000 samples × 2 bytes = 32000 bytes
-// end: false (default) - Normal audio data sending, server will accumulate audio data, automatically returns animation data and starts synchronized playback of animation and audio after accumulating enough data
-// end: true - Immediately return animation data, no longer accumulating, used for ending current conversation or scenarios requiring immediate response
+// ⚠️ CRITICAL: Initialize audio context first (MUST be called in user gesture context)
+// This method MUST be called within a user gesture event handler (click, touchstart, etc.)
+// to satisfy browser security policies. Calling it outside a user gesture will fail.
+// All audio operations (start, send, etc.) require prior initialization.
+button.addEventListener('click', async () => {
+  // Initialize audio context - MUST be in user gesture context
+  await avatarView.controller.initializeAudioContext()
+  
+  // Start WebSocket service
+  await avatarView.controller.start()
+  
+  // Send audio data (must be mono PCM16 format matching configured sample rate)
+  const conversationId = avatarView.controller.send(audioData: ArrayBuffer, end: boolean)
+  // Returns: conversationId - Conversation ID for this conversation session
+  // end: false (default) - Continue sending audio data for current conversation
+  // end: true - Mark the end of current conversation round. After end=true, sending new audio data will interrupt any ongoing playback from the previous conversation round
+})
 
 // Close WebSocket service
-avatarView.avatarController.close()
+avatarView.controller.close()
 ```
 
-#### External Data Mode Methods
+#### Host Mode Methods
 
 ```typescript
-// Start playback with initial audio and animation data
-await avatarView.avatarController.play(
-  initialAudioChunks?: Array<{ data: Uint8Array, isLast: boolean }>,  // Initial audio chunks (16kHz mono PCM16)
-  initialKeyframes?: any[]  // Initial animation keyframes (obtained from your service)
-)
+// ⚠️ CRITICAL: Initialize audio context first (MUST be called in user gesture context)
+// This method MUST be called within a user gesture event handler (click, touchstart, etc.)
+// to satisfy browser security policies. Calling it outside a user gesture will fail.
+// All audio operations (yieldAudioData, yieldFramesData, etc.) require prior initialization.
+button.addEventListener('click', async () => {
+  // Initialize audio context - MUST be in user gesture context
+  await avatarView.controller.initializeAudioContext()
+  
+  // Stream audio chunks (must be mono PCM16 format matching configured sample rate)
+  const conversationId = avatarView.controller.yieldAudioData(
+    data: Uint8Array,               // Audio chunk data (PCM16 format)
+    isLast: boolean = false         // Whether this is the last chunk
+  )
+  // Returns: conversationId - Conversation ID for this audio session
+
+  // Stream animation keyframes (requires conversationId from audio data)
+  avatarView.controller.yieldFramesData(
+    keyframesDataArray: (Uint8Array | ArrayBuffer)[],  // Animation keyframes binary data array (each element is a protobuf encoded Message)
+    conversationId: string                              // Conversation ID (required)
+  )
+})
+```
 
-// Stream additional audio chunks (after play() is called)
-avatarView.avatarController.sendAudioChunk(
-  data: Uint8Array,               // Audio chunk data
-  isLast: boolean = false         // Whether this is the last chunk
-)
+**⚠️ Important: Conversation ID (conversationId) Management**
 
-// Stream additional animation keyframes (after play() is called)
-avatarView.avatarController.sendKeyframes(
-  keyframes: any[]                 // Additional animation keyframes (obtained from your service)
-)
-```
+**SDK Mode:**
+- `send()` returns a conversationId to distinguish each conversation round
+- `end=true` marks the end of a conversation round
+
+**Host Mode:**
+- `yieldAudioData()` returns a conversationId (automatically generates if starting new session)
+   - `yieldFramesData()` requires a valid conversationId parameter
+   - Animation data with mismatched conversationId will be **discarded**
+   - Use `getCurrentConversationId()` to retrieve the current active conversationId
 
 #### Common Methods (Both Modes)
 
 ```typescript
+
 // Interrupt current playback (stops and clears data)
 avatarView.avatarController.interrupt()
 
 // Clear all data and resources
 avatarView.avatarController.clear()
 
-// Get connection state (network mode only)
-const isConnected = avatarView.avatarController.connected
-
-// Start service (network mode only)
-await avatarView.avatarController.start()
-
-// Close service (network mode only)
-avatarView.avatarController.close()
+// Get current conversation ID (for Host mode)
+const conversationId = avatarView.avatarController.getCurrentConversationId()
+// Returns: Current conversationId for the active audio session, or null if no active session
 
-// Get current avatar state
-const state = avatarView.avatarController.state
+// Volume control (affects only avatar audio player, not system volume)
+avatarView.avatarController.setVolume(0.5)  // Set volume to 50% (0.0 to 1.0)
+const currentVolume = avatarView.avatarController.getVolume()  // Get current volume (0.0 to 1.0)
 
 // Set event callbacks
-avatarView.avatarController.onConnectionState = (state: ConnectionState) => {} // Network mode only
-avatarView.avatarController.onAvatarState = (state: AvatarState) => {}
+avatarView.avatarController.onConnectionState = (state: ConnectionState) => {} // SDK mode only
+avatarView.avatarController.onConversationState = (state: ConversationState) => {}
 avatarView.avatarController.onError = (error: Error) => {}
 ```
 
+#### Avatar Transform Methods
+
+```typescript
+// Get or set avatar transform (position and scale in canvas)
+// Get current transform
+const currentTransform = avatarView.transform // { x: number, y: number, scale: number }
+
+// Set transform
+avatarView.transform = { x, y, scale }
+// - x: Horizontal offset in normalized coordinates (-1 to 1, where -1 = left edge, 0 = center, 1 = right edge)
+// - y: Vertical offset in normalized coordinates (-1 to 1, where -1 = bottom edge, 0 = center, 1 = top edge)
+// - scale: Scale factor (1.0 = original size, 2.0 = double size, 0.5 = half size)
+// Example:
+avatarView.transform = { x: 0, y: 0, scale: 1.0 }  // Center, original size
+avatarView.transform = { x: 0.5, y: 0, scale: 2.0 } // Right half, double size
+```
+
 **Important Notes:**
-- `start()` and `close()` are only available in network mode
-- `play()`, `sendAudioChunk()`, and `sendKeyframes()` are only available in external data mode
-- `interrupt()` and `clear()` are available in both modes
+- `start()` and `close()` are only available in SDK mode
+- `yieldAudioData()` and `yieldFramesData()` are only available in Host mode
+- `pause()`, `resume()`, `interrupt()`, `clear()`, `getCurrentConversationId()`, `setVolume()`, and `getVolume()` are available in both modes
 - The playback mode is determined when creating `AvatarView` and cannot be changed
 
 ## 🔧 Configuration
@@ -386,40 +631,55 @@ avatarView.avatarController.onError = (error: Error) => {}
 ```typescript
 interface Configuration {
   environment: Environment
+  drivingServiceMode?: DrivingServiceMode  // Optional, default is 'sdk' (SDK mode)
+  logLevel?: LogLevel  // Optional, default is 'off' (no logs)
+  audioFormat?: AudioFormat  // Optional, default is { channelCount: 1, sampleRate: 16000 }
+  characterApiBaseUrl?: string  // Optional, internal debug config, can be ignored
 }
-```
-
-**Description:**
-- `environment`: Specifies the environment (cn/us/test), SDK will automatically use the corresponding API address and WebSocket address based on the environment
-- `sessionToken`: Set separately via `AvatarKit.setSessionToken()`, not in Configuration
 
-```typescript
-enum Environment {
-  cn = 'cn',    // China region
-  us = 'us',    // US region
-  test = 'test' // Test environment
+interface AudioFormat {
+  readonly channelCount: 1  // Fixed to 1 (mono)
+  readonly sampleRate: number  // Supported: 8000, 16000, 22050, 24000, 32000, 44100, 48000 Hz, default: 16000
 }
 ```
 
-### AvatarViewOptions
+### LogLevel
+
+Control the verbosity of SDK logs:
 
 ```typescript
-interface AvatarViewOptions {
-  playbackMode?: AvatarPlaybackMode  // Playback mode, default is 'network'
-  container?: HTMLElement            // Canvas container element
+enum LogLevel {
+  off = 'off',        // Disable all logs
+  error = 'error',    // Only error logs
+  warning = 'warning', // Warning and error logs
+  all = 'all'         // All logs (info, warning, error) - default
 }
 ```
 
+**Note:** `LogLevel.off` completely disables all logging, including error logs. Use with caution in production environments.
+
 **Description:**
-- `playbackMode`: Specifies the playback mode (`'network'` or `'external'`), default is `'network'`
-  - `'network'`: SDK handles WebSocket communication, send audio via `send()`
-  - `'external'`: External components provide audio and animation data, SDK handles synchronized playback
-- `container`: Optional container element for Canvas, if not provided, Canvas will be created but not added to DOM
+- `environment`: Specifies the environment (cn/intl), SDK will automatically use the corresponding API address and WebSocket address based on the environment
+- `drivingServiceMode`: Specifies the driving service mode
+  - `DrivingServiceMode.sdk` (default): SDK mode - SDK handles WebSocket communication automatically
+  - `DrivingServiceMode.host`: Host mode - Host application provides audio and animation data
+- `logLevel`: Controls the verbosity of SDK logs
+  - `LogLevel.off` (default): Disable all logs
+  - `LogLevel.error`: Only error logs
+  - `LogLevel.warning`: Warning and error logs
+  - `LogLevel.all`: All logs (info, warning, error)
+- `audioFormat`: Configures audio sample rate and channel count
+  - `channelCount`: Fixed to 1 (mono channel)
+  - `sampleRate`: Audio sample rate in Hz (default: 16000)
+    - Supported values: 8000, 16000, 22050, 24000, 32000, 44100, 48000
+    - The configured sample rate will be used for both audio recording and playback
+- `characterApiBaseUrl`: Internal debug config, can be ignored
+- `sessionToken`: Set separately via `AvatarSDK.setSessionToken()`, not in Configuration
 
 ```typescript
-enum AvatarPlaybackMode {
-  network = 'network',   // Network mode: SDK handles WebSocket communication
-  external = 'external'  // External data mode: External provides data, SDK handles playback
+enum Environment {
+  cn = 'cn',    // China region
+  intl = 'intl',    // International region
 }
 ```
 
@@ -450,16 +710,25 @@ enum ConnectionState {
 }
 ```
 
-### AvatarState
+### ConversationState
 
 ```typescript
-enum AvatarState {
-  idle = 'idle',      // Idle state, showing breathing animation
-  active = 'active',  // Active, waiting for playable content
-  playing = 'playing' // Playing
+enum ConversationState {
+  idle = 'idle',      // Idle state (breathing animation)
+  playing = 'playing', // Playing state (active conversation)
+  pausing = 'pausing' // Pausing state (paused during playback)
 }
 ```
 
+**State Description:**
+- `idle`: Avatar is in idle state (breathing animation), waiting for conversation to start
+- `playing`: Avatar is playing conversation content (including during transition animations)
+- `pausing`: Avatar playback is paused (e.g., when `end=false` and waiting for more audio data)
+
+**Note:** During transition animations, the target state is notified immediately:
+- When transitioning from `idle` to `playing`, the `playing` state is notified immediately
+- When transitioning from `playing` to `idle`, the `idle` state is notified immediately
+
 ## 🎨 Rendering System
 
 The SDK supports two rendering backends:
@@ -469,70 +738,19 @@ The SDK supports two rendering backends:
 
 The rendering system automatically selects the best backend, no manual configuration needed.
 
-## 🔍 Debugging and Monitoring
-
-### Logging System
-
-The SDK has a built-in complete logging system, supporting different levels of log output:
-
-```typescript
-import { logger } from '@spatialwalk/avatarkit'
-
-// Set log level
-logger.setLevel('verbose') // 'basic' | 'verbose'
-
-// Manual log output
-logger.log('Info message')
-logger.warn('Warning message')
-logger.error('Error message')
-```
-
-### Performance Monitoring
-
-The SDK provides performance monitoring interfaces to monitor rendering performance:
-
-```typescript
-// Get rendering performance statistics
-const stats = avatarView.getPerformanceStats()
-
-if (stats) {
-  console.log(`Render time: ${stats.renderTime.toFixed(2)}ms`)
-  console.log(`Sort time: ${stats.sortTime.toFixed(2)}ms`)
-  console.log(`Rendering backend: ${stats.backend}`)
-  
-  // Calculate frame rate
-  const fps = 1000 / stats.renderTime
-  console.log(`Frame rate: ${fps.toFixed(2)} FPS`)
-}
-
-// Regular performance monitoring
-setInterval(() => {
-  const stats = avatarView.getPerformanceStats()
-  if (stats) {
-    // Send to monitoring service or display on UI
-    console.log('Performance:', stats)
-  }
-}, 1000)
-```
-
-**Performance Statistics Description:**
-- `renderTime`: Total rendering time (milliseconds), includes sorting and GPU rendering
-- `sortTime`: Sorting time (milliseconds), uses Radix Sort algorithm to depth-sort point cloud
-- `backend`: Currently used rendering backend (`'webgpu'` | `'webgl'` | `null`)
-
 ## 🚨 Error Handling
 
-### SPAvatarError
+### AvatarError
 
 The SDK uses custom error types, providing more detailed error information:
 
 ```typescript
-import { SPAvatarError } from '@spatialwalk/avatarkit'
+import { AvatarError } from '@spatialwalk/avatarkit'
 
 try {
   await avatarView.avatarController.start()
 } catch (error) {
-  if (error instanceof SPAvatarError) {
+  if (error instanceof AvatarError) {
     console.error('SDK Error:', error.message, error.code)
   } else {
     console.error('Unknown error:', error)
@@ -553,15 +771,12 @@ avatarView.avatarController.onError = (error: Error) => {
 
 ### Lifecycle Management
 
-#### Network Mode Lifecycle
+#### SDK Mode Lifecycle
 
 ```typescript
 // Initialize
 const container = document.getElementById('avatar-container')
-const avatarView = new AvatarView(avatar, {
-  container: container,
-  playbackMode: AvatarPlaybackMode.network
-})
+const avatarView = new AvatarView(avatar, container)
 await avatarView.avatarController.start()
 
 // Use
@@ -572,21 +787,16 @@ avatarView.avatarController.close()
 avatarView.dispose() // Automatically cleans up all resources
 ```
 
-#### External Data Mode Lifecycle
+#### Host Mode Lifecycle
 
 ```typescript
 // Initialize
 const container = document.getElementById('avatar-container')
-const avatarView = new AvatarView(avatar, {
-  container: container,
-  playbackMode: AvatarPlaybackMode.external
-})
+const avatarView = new AvatarView(avatar, container)
 
 // Use
-const initialAudioChunks = [{ data: audioData1, isLast: false }]
-await avatarView.avatarController.play(initialAudioChunks, initialKeyframes)
-avatarView.avatarController.sendAudioChunk(audioChunk, false)
-avatarView.avatarController.sendKeyframes(keyframes)
+const conversationId = avatarView.avatarController.yieldAudioData(audioChunk, false)
+avatarView.avatarController.yieldFramesData(keyframesDataArray, conversationId) // keyframesDataArray: (Uint8Array | ArrayBuffer)[]
 
 // Cleanup
 avatarView.avatarController.clear() // Clear all data and resources
@@ -594,63 +804,17 @@ avatarView.dispose() // Automatically cleans up all resources
 ```
 
 **⚠️ Important Notes:** 
-- SDK currently only supports one AvatarView instance at a time
-- When switching characters, must first call `dispose()` to clean up old AvatarView, then create new instance
+- When disposing AvatarView instances, must call `dispose()` to properly clean up resources
 - Not properly cleaning up may cause resource leaks and rendering errors
-- In network mode, call `close()` before `dispose()` to properly close WebSocket connections
-- In external data mode, call `clear()` before `dispose()` to clear all playback data
+- In SDK mode, call `close()` before `dispose()` to properly close WebSocket connections
+- In Host mode, call `clear()` before `dispose()` to clear all playback data
 
 ### Memory Optimization
 
 - SDK automatically manages WASM memory allocation
-- Supports dynamic loading/unloading of character and animation resources
+- Supports dynamic loading/unloading of avatar and animation resources
 - Provides memory usage monitoring interface
 
-### Audio Data Sending
-
-#### Network Mode
-
-The `send()` method receives audio data in `ArrayBuffer` format:
-
-**Audio Format Requirements:**
-- **Sample Rate**: 16kHz (16000 Hz) - **Backend requirement, must be exactly 16kHz**
-- **Format**: PCM16 (16-bit signed integer, little-endian)
-- **Channels**: Mono (single channel)
-- **Data Size**: Each sample is 2 bytes, so 1 second of audio = 16000 samples × 2 bytes = 32000 bytes
-
-**Usage:**
-- `audioData`: Audio data (ArrayBuffer format, must be 16kHz mono PCM16)
-- `end=false` (default) - Normal audio data sending, server will accumulate audio data, automatically returns animation data and starts synchronized playback of animation and audio after accumulating enough data
-- `end=true` - Immediately return animation data, no longer accumulating, used for ending current conversation or scenarios requiring immediate response
-- **Important**: No need to wait for `end=true` to start playing, it will automatically start playing after accumulating enough audio data
-
-#### External Data Mode
-
-The `play()` method starts playback with initial data, then use `sendAudioChunk()` to stream additional audio:
-
-**Audio Format Requirements:**
-- Same as network mode: 16kHz mono PCM16 format
-- Audio data should be provided as `Uint8Array` in chunks with `isLast` flag
-
-**Usage:**
-```typescript
-// Start playback with initial audio and animation data
-// Note: Audio and animation data should be obtained from your backend service
-const initialAudioChunks = [
-  { data: audioData1, isLast: false },
-  { data: audioData2, isLast: false }
-]
-await avatarController.play(initialAudioChunks, initialKeyframes)
-
-// Stream additional audio chunks
-avatarController.sendAudioChunk(audioChunk, isLast)
-```
-
-**Resampling (Both Modes):**
-- If your audio source is at a different sample rate (e.g., 24kHz, 48kHz), you **must** resample it to 16kHz before sending
-- For high-quality resampling, use Web Audio API's `OfflineAudioContext` with anti-aliasing filtering
-- See example projects (`vanilla`, `react`, `vue`) for complete resampling implementation
-
 ## 🌐 Browser Compatibility
 
 - **Chrome/Edge** 90+ (WebGPU recommended)
@@ -669,6 +833,5 @@ Issues and Pull Requests are welcome!
 ## 📞 Support
 
 For questions, please contact:
-- Email: support@spavatar.com
-- Documentation: https://docs.spavatar.com
-- GitHub: https://github.com/spavatar/sdk
+- Email: code@spatialwalk.net
+- Documentation: https://docs.spatialreal.ai