@spatialwalk/avatarkit 1.0.0-beta.3 → 1.0.0-beta.5

package/CHANGELOG.md

@@ -0,0 +1,127 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [1.0.0-beta.5] - 2025-11-14
+
+### 🐛 Bug Fixes
+- Fixed missing `AvatarPlaybackMode` enum export in published package
+
+---
+
+## [1.0.0-beta.4] - 2025-11-14
+
+### 🔄 Updates
+- Updated WASM module (avatar_core_wasm.wasm)
+
+---
+
+## [1.0.0-beta.3] - 2025-11-13
+
+### 🎉 Major Changes
+
+#### Architecture Refactoring - Three-Layer Architecture
+- **Rendering Layer (AvatarView)**: Pure rendering responsibility, receives rendering instructions from the playback layer
+- **Playback Layer (AvatarController)**: Unified data controller responsible for audio playback and animation rendering synchronization
+- **Network Layer (NetworkLayer)**: Optional network communication layer, automatically composed only in network mode
+
+#### Dual Playback Mode Support
+- **Network Mode**: 
+  - Send audio data to server via WebSocket
+  - Automatically receive and play animation data returned from server
+  - Use `controller.start()` and `controller.send()` methods
+  
+- **External Data Mode**:
+  - External components fully control audio and animation data acquisition
+  - SDK only responsible for synchronized playback of externally provided data
+  - Use `controller.play()`, `controller.sendAudioChunk()` and `controller.sendKeyframes()` methods
+
+### ✨ New Features
+
+- **Transition Animation Optimization**: 
+  - Transition animation duration adjusted to 200ms
+  - Improved audio-animation synchronization logic, ensuring audio and animation start playing together after transition animation ends
+  - Optimized transition frame generation and alignment logic
+
+- **Camera Configuration Enhancement**:
+  - Default transparent background, background rendering functionality removed
+  - Camera configuration aligned with iOS (Reversed-Z projection, near=0.01, far=100)
+  - Support dynamic camera configuration loading from characterSettings and camera.json
+
+- **Audio Sample Rate Configuration**:
+  - Updated default audio sample rate to 16kHz (previously 24kHz) to match backend requirements
+  - Audio format requirement: 16kHz mono PCM16
+
+- **Telemetry Migration**:
+  - Migrated from PostHog to Tencent Cloud CLS (Cloud Log Service)
+  - Support environment-based dynamic configuration (cn/test/us)
+  - Support both Token and SecretId/SecretKey authentication methods
+  - Aligned with iOS SDK configuration
+
+### 🔧 API Changes
+
+#### Breaking Changes
+
+- **AvatarView Constructor**:
+  ```typescript
+  // Old API
+  new AvatarView(avatar, container)
+  
+  // New API
+  new AvatarView(avatar, {
+    container: container,
+    playbackMode: AvatarPlaybackMode.network // or AvatarPlaybackMode.external
+  })
+  ```
+
+- **AvatarController Methods**:
+  - Network Mode: `start()` / `close()` / `send()`
+  - External Data Mode: `play()` / `sendAudioChunk()` / `sendKeyframes()`
+  - Common Methods: `interrupt()` / `clear()` / `dispose()`
+
+- **Removed Properties**:
+  - `realtimeStartTime` property removed (no longer needed)
+
+### 🐛 Bug Fixes
+
+- Fixed audio-animation desynchronization during transition animation
+- Fixed animation freezing issue in external data mode
+- Fixed audio playback speed abnormality caused by sample rate mismatch
+- Fixed camera configuration priority logic
+- Fixed WebSocket connection state callback timing issues
+
+### 📚 Documentation
+
+- Updated README.md with detailed explanation of three-layer architecture and two playback modes
+- Added audio format requirements documentation (16kHz mono PCM16)
+- Updated API reference documentation and code examples
+- Added architecture diagrams and flowcharts
+
+### 🧪 Testing
+
+- Updated unit tests to cover new architecture
+- Added unit tests for external data mode
+- Updated E2E tests to use new API
+- All tests passing (69 unit tests + 24 E2E tests)
+
+### 🔄 Internal Changes
+
+- Refactored `AvatarController` into unified playback layer
+- Extracted `NetworkLayer` as independent component
+- Optimized animation playback loop and audio-animation synchronization logic
+- Updated WASM module (avatar_core_wasm.wasm)
+- Improved error handling and logging
+
+### 📦 Dependencies
+
+- No new dependencies added
+- Maintained existing dependency versions
+
+---
+
+## [1.0.0-beta.2] - Previous Version
+
+(Previous version changelog entries...)

package/README.md

@@ -47,51 +47,159 @@ const avatar = await avatarManager.load('character-id', (progress) => {
 })
 
 // 3. Create view (automatically creates Canvas and AvatarController)
+// Network mode (default)
 const container = document.getElementById('avatar-container')
-const avatarView = new AvatarView(avatar, container)
+const avatarView = new AvatarView(avatar, {
+  container: container,
+  playbackMode: 'network' // Optional, 'network' is default
+})
 
-// 4. Start real-time communication
+// 4. Start real-time communication (network mode only)
 await avatarView.avatarController.start()
 
-// 5. Send audio data
+// 5. Send audio data (network mode)
+// ⚠️ Important: Audio must be 16kHz mono PCM16 format
 // If audio is Uint8Array, you can use slice().buffer to convert to ArrayBuffer
-const audioUint8 = new Uint8Array(1024) // Example: audio data
+const audioUint8 = new Uint8Array(1024) // Example: 16kHz PCM16 audio data (512 samples = 1024 bytes)
 const audioData = audioUint8.slice().buffer // Simplified conversion, works for ArrayBuffer and SharedArrayBuffer
 avatarView.avatarController.send(audioData, false) // Send audio data, will automatically start playing after accumulating enough data
 avatarView.avatarController.send(audioData, true) // end=true means immediately return animation data, no longer accumulating
 ```
 
-### Complete Example
+### External Data Mode Example
+
+```typescript
+import { AvatarPlaybackMode } from '@spatialwalk/avatarkit'
+
+// 1-3. Same as network mode (initialize SDK, load character)
+
+// 3. Create view with external data mode
+const container = document.getElementById('avatar-container')
+const avatarView = new AvatarView(avatar, {
+  container: container,
+  playbackMode: AvatarPlaybackMode.external
+})
 
-Check the example code in the GitHub repository for the complete usage flow.
+// 4. Start playback with initial data (obtained from your service)
+// Note: Audio and animation data should be obtained from your backend service
+const initialAudioChunks = [{ data: audioData1, isLast: false }, { data: audioData2, isLast: false }]
+const initialKeyframes = animationData1 // Animation keyframes from your service
+
+await avatarView.avatarController.play(initialAudioChunks, initialKeyframes)
+
+// 5. Stream additional data as needed
+avatarView.avatarController.sendAudioChunk(audioData3, false)
+avatarView.avatarController.sendKeyframes(animationData2)
+```
+
+### Complete Examples
+
+Check the example code in the GitHub repository for complete usage flows for both modes.
 
 **Example Project:** [Avatarkit-web-demo](https://github.com/spatialwalk/Avatarkit-web-demo)
 
-This repository contains complete examples for Vanilla JS, Vue 3, and React, demonstrating how to integrate and use SPAvatarKit SDK in different frameworks.
+This repository contains complete examples for Vanilla JS, Vue 3, and React, demonstrating:
+- Network mode: Real-time audio input with automatic animation data reception
+- External data mode: Custom data sources with manual audio/animation data management
 
 ## 🏗️ Architecture Overview
 
+### Three-Layer Architecture
+
+The SDK uses a three-layer architecture for clear separation of concerns:
+
+1. **Rendering Layer (AvatarView)** - Responsible for 3D rendering only
+2. **Playback Layer (AvatarController)** - Manages audio/animation synchronization and playback
+3. **Network Layer (NetworkLayer)** - Handles WebSocket communication (only in network mode)
+
 ### Core Components
 
 - **AvatarKit** - SDK initialization and management
 - **AvatarManager** - Character resource loading and management
-- **AvatarView** - 3D rendering view (internally contains AvatarController)
-- **AvatarController** - Real-time communication and data processing
+- **AvatarView** - 3D rendering view (rendering layer)
+- **AvatarController** - Audio/animation playback controller (playback layer)
+- **NetworkLayer** - WebSocket communication (network layer, automatically composed in network mode)
 - **AvatarCoreAdapter** - WASM module adapter
 
+### Playback Modes
+
+The SDK supports two playback modes, configured when creating `AvatarView`:
+
+#### 1. Network Mode (Default)
+- SDK handles WebSocket communication automatically
+- Send audio data via `AvatarController.send()`
+- SDK receives animation data from backend and synchronizes playback
+- Best for: Real-time audio input scenarios
+
+#### 2. External Data Mode
+- External components manage their own network/data fetching
+- External components provide both audio and animation data
+- SDK only handles synchronized playback
+- Best for: Custom data sources, pre-recorded content, or custom network implementations
+
 ### Data Flow
 
+#### Network Mode Flow
+
+```
+User audio input (16kHz mono PCM16) 
+    ↓
+AvatarController.send() 
+    ↓
+NetworkLayer → WebSocket → Backend processing
+    ↓
+Backend returns animation data (FLAME keyframes) 
+    ↓
+NetworkLayer → AvatarController → AnimationPlayer
+    ↓
+FLAME parameters → AvatarCore.computeFrameFlatFromParams() → Splat data
+    ↓
+AvatarController (playback loop) → AvatarView.renderRealtimeFrame()
+    ↓
+RenderSystem → WebGPU/WebGL → Canvas rendering
+```
+
+#### External Data Mode Flow
+
 ```
-User audio input (16kHz mono PCM) → AvatarController → WebSocket → Backend processing
-                                              ↓
-Backend returns animation data (FLAME keyframes) → AvatarController → AnimationPlayer
-                                              ↓
+External data source (audio + animation)
+    ↓
+AvatarController.play(initialAudio, initialKeyframes) // Start playback
+    ↓
+AvatarController.sendAudioChunk() // Stream additional audio
+AvatarController.sendKeyframes() // Stream additional animation
+    ↓
+AvatarController → AnimationPlayer (synchronized playback)
+    ↓
 FLAME parameters → AvatarCore.computeFrameFlatFromParams() → Splat data
-                                              ↓
-Splat data → RenderSystem → WebGPU/WebGL → Canvas rendering
+    ↓
+AvatarController (playback loop) → AvatarView.renderRealtimeFrame()
+    ↓
+RenderSystem → WebGPU/WebGL → Canvas rendering
 ```
 
-**Note:** Users need to provide audio data themselves (16kHz mono PCM), SDK handles receiving animation data and rendering.
+**Note:** 
+- In network mode, users provide audio data, SDK handles network communication and animation data reception
+- In external data mode, users provide both audio and animation data, SDK handles synchronized playback only
+
+### Audio Format Requirements
+
+**⚠️ Important:** The SDK requires audio data to be in **16kHz mono PCM16** format:
+
+- **Sample Rate**: 16kHz (16000 Hz) - This is a backend requirement
+- **Channels**: Mono (single channel)
+- **Format**: PCM16 (16-bit signed integer, little-endian)
+- **Byte Order**: Little-endian
+
+**Audio Data Format:**
+- Each sample is 2 bytes (16-bit)
+- Audio data should be provided as `ArrayBuffer` or `Uint8Array`
+- For example: 1 second of audio = 16000 samples × 2 bytes = 32000 bytes
+
+**Resampling:**
+- If your audio source is at a different sample rate (e.g., 24kHz, 48kHz), you must resample it to 16kHz before sending to the SDK
+- For high-quality resampling, we recommend using Web Audio API's `OfflineAudioContext` with anti-aliasing filtering
+- See example projects for resampling implementation
 
 ## 📚 API Reference
 
@@ -129,20 +237,36 @@ manager.clearCache()
 
 ### AvatarView
 
-3D rendering view, internally automatically creates and manages AvatarController.
+3D rendering view (rendering layer), responsible for 3D rendering only. Internally automatically creates and manages `AvatarController`.
 
 **⚠️ Important Limitation:** Currently, the SDK only supports one AvatarView instance at a time. If you need to switch characters, you must first call the `dispose()` method to clean up the current AvatarView, then create a new instance.
 
+**Playback Mode Configuration:**
+- The playback mode is fixed when creating `AvatarView` and persists throughout its lifecycle
+- Cannot be changed after creation
+
 ```typescript
+import { AvatarPlaybackMode } from '@spatialwalk/avatarkit'
+
 // Create view (Canvas is automatically added to container)
-const avatarView = new AvatarView(avatar: Avatar, container?: HTMLElement)
+// Network mode (default)
+const container = document.getElementById('avatar-container')
+const avatarView = new AvatarView(avatar: Avatar, {
+  container: container,
+  playbackMode: AvatarPlaybackMode.network // Optional, default is 'network'
+})
+
+// External data mode
+const avatarView = new AvatarView(avatar: Avatar, {
+  container: container,
+  playbackMode: AvatarPlaybackMode.external
+})
 
 // Get Canvas element
 const canvas = avatarView.getCanvas()
 
-// Set background
-avatarView.setBackgroundImage('path/to/image.jpg')
-avatarView.setBackgroundOpaque(true)
+// Get playback mode
+const mode = avatarView.playbackMode // 'network' | 'external'
 
 // Update camera configuration
 avatarView.updateCameraConfig(cameraConfig: CameraConfig)
@@ -163,39 +287,98 @@ if (currentAvatarView) {
 // Load new character
 const newAvatar = await avatarManager.load('new-character-id')
 
-// Create new AvatarView
-currentAvatarView = new AvatarView(newAvatar, container)
-await currentAvatarView.avatarController.start()
+// Create new AvatarView (with same or different playback mode)
+currentAvatarView = new AvatarView(newAvatar, {
+  container: container,
+  playbackMode: AvatarPlaybackMode.network
+})
+
+// Network mode: start connection
+if (currentAvatarView.playbackMode === AvatarPlaybackMode.network) {
+  await currentAvatarView.avatarController.start()
+}
 ```
 
 ### AvatarController
 
-Real-time communication controller, handles WebSocket connections and animation data.
+Audio/animation playback controller (playback layer), manages synchronized playback of audio and animation. Automatically composes `NetworkLayer` in network mode.
+
+**Two Usage Patterns:**
+
+#### Network Mode Methods
 
 ```typescript
-// Start connection
+// Start WebSocket service
 await avatarView.avatarController.start()
 
-// Send audio data
+// Send audio data (SDK handles receiving animation data automatically)
 avatarView.avatarController.send(audioData: ArrayBuffer, end: boolean)
-// audioData: Audio data (ArrayBuffer format)
+// audioData: Audio data (ArrayBuffer format, must be 16kHz mono PCM16)
+//   - Sample rate: 16kHz (16000 Hz) - backend requirement
+//   - Format: PCM16 (16-bit signed integer, little-endian)
+//   - Channels: Mono (single channel)
+//   - Example: 1 second = 16000 samples × 2 bytes = 32000 bytes
 // end: false (default) - Normal audio data sending, server will accumulate audio data, automatically returns animation data and starts synchronized playback of animation and audio after accumulating enough data
 // end: true - Immediately return animation data, no longer accumulating, used for ending current conversation or scenarios requiring immediate response
 
-// Interrupt conversation
+// Close WebSocket service
+avatarView.avatarController.close()
+```
+
+#### External Data Mode Methods
+
+```typescript
+// Start playback with initial audio and animation data
+await avatarView.avatarController.play(
+  initialAudioChunks?: Array<{ data: Uint8Array, isLast: boolean }>,  // Initial audio chunks (16kHz mono PCM16)
+  initialKeyframes?: any[]  // Initial animation keyframes (obtained from your service)
+)
+
+// Stream additional audio chunks (after play() is called)
+avatarView.avatarController.sendAudioChunk(
+  data: Uint8Array,               // Audio chunk data
+  isLast: boolean = false         // Whether this is the last chunk
+)
+
+// Stream additional animation keyframes (after play() is called)
+avatarView.avatarController.sendKeyframes(
+  keyframes: any[]                 // Additional animation keyframes (obtained from your service)
+)
+```
+
+#### Common Methods (Both Modes)
+
+```typescript
+// Interrupt current playback (stops and clears data)
 avatarView.avatarController.interrupt()
 
-// Close connection
+// Clear all data and resources
+avatarView.avatarController.clear()
+
+// Get connection state (network mode only)
+const isConnected = avatarView.avatarController.connected
+
+// Start service (network mode only)
+await avatarView.avatarController.start()
+
+// Close service (network mode only)
 avatarView.avatarController.close()
 
+// Get current avatar state
+const state = avatarView.avatarController.state
+
 // Set event callbacks
-avatarView.avatarController.onConnectionState = (state: ConnectionState) => {}
+avatarView.avatarController.onConnectionState = (state: ConnectionState) => {} // Network mode only
 avatarView.avatarController.onAvatarState = (state: AvatarState) => {}
 avatarView.avatarController.onError = (error: Error) => {}
-
-// Note: sendText() method is not supported, calling it will throw an error
 ```
 
+**Important Notes:**
+- `start()` and `close()` are only available in network mode
+- `play()`, `sendAudioChunk()`, and `sendKeyframes()` are only available in external data mode
+- `interrupt()` and `clear()` are available in both modes
+- The playback mode is determined when creating `AvatarView` and cannot be changed
+
 ## 🔧 Configuration
 
 ### Configuration
@@ -218,6 +401,28 @@ enum Environment {
 }
 ```
 
+### AvatarViewOptions
+
+```typescript
+interface AvatarViewOptions {
+  playbackMode?: AvatarPlaybackMode  // Playback mode, default is 'network'
+  container?: HTMLElement            // Canvas container element
+}
+```
+
+**Description:**
+- `playbackMode`: Specifies the playback mode (`'network'` or `'external'`), default is `'network'`
+  - `'network'`: SDK handles WebSocket communication, send audio via `send()`
+  - `'external'`: External components provide audio and animation data, SDK handles synchronized playback
+- `container`: Optional container element for Canvas, if not provided, Canvas will be created but not added to DOM
+
+```typescript
+enum AvatarPlaybackMode {
+  network = 'network',   // Network mode: SDK handles WebSocket communication
+  external = 'external'  // External data mode: External provides data, SDK handles playback
+}
+```
+
 ### CameraConfig
 
 ```typescript
@@ -348,15 +553,43 @@ avatarView.avatarController.onError = (error: Error) => {
 
 ### Lifecycle Management
 
+#### Network Mode Lifecycle
+
 ```typescript
 // Initialize
-const avatarView = new AvatarView(avatar, container)
+const container = document.getElementById('avatar-container')
+const avatarView = new AvatarView(avatar, {
+  container: container,
+  playbackMode: AvatarPlaybackMode.network
+})
 await avatarView.avatarController.start()
 
 // Use
 avatarView.avatarController.send(audioData, false)
 
-// Cleanup (must be called before switching characters)
+// Cleanup
+avatarView.avatarController.close()
+avatarView.dispose() // Automatically cleans up all resources
+```
+
+#### External Data Mode Lifecycle
+
+```typescript
+// Initialize
+const container = document.getElementById('avatar-container')
+const avatarView = new AvatarView(avatar, {
+  container: container,
+  playbackMode: AvatarPlaybackMode.external
+})
+
+// Use
+const initialAudioChunks = [{ data: audioData1, isLast: false }]
+await avatarView.avatarController.play(initialAudioChunks, initialKeyframes)
+avatarView.avatarController.sendAudioChunk(audioChunk, false)
+avatarView.avatarController.sendKeyframes(keyframes)
+
+// Cleanup
+avatarView.avatarController.clear() // Clear all data and resources
 avatarView.dispose() // Automatically cleans up all resources
 ```
 
@@ -364,6 +597,8 @@ avatarView.dispose() // Automatically cleans up all resources
 - SDK currently only supports one AvatarView instance at a time
 - When switching characters, must first call `dispose()` to clean up old AvatarView, then create new instance
 - Not properly cleaning up may cause resource leaks and rendering errors
+- In network mode, call `close()` before `dispose()` to properly close WebSocket connections
+- In external data mode, call `clear()` before `dispose()` to clear all playback data
 
 ### Memory Optimization
 
@@ -373,14 +608,49 @@ avatarView.dispose() // Automatically cleans up all resources
 
 ### Audio Data Sending
 
+#### Network Mode
+
 The `send()` method receives audio data in `ArrayBuffer` format:
 
+**Audio Format Requirements:**
+- **Sample Rate**: 16kHz (16000 Hz) - **Backend requirement, must be exactly 16kHz**
+- **Format**: PCM16 (16-bit signed integer, little-endian)
+- **Channels**: Mono (single channel)
+- **Data Size**: Each sample is 2 bytes, so 1 second of audio = 16000 samples × 2 bytes = 32000 bytes
+
 **Usage:**
-- `audioData`: Audio data (ArrayBuffer format)
+- `audioData`: Audio data (ArrayBuffer format, must be 16kHz mono PCM16)
 - `end=false` (default) - Normal audio data sending, server will accumulate audio data, automatically returns animation data and starts synchronized playback of animation and audio after accumulating enough data
 - `end=true` - Immediately return animation data, no longer accumulating, used for ending current conversation or scenarios requiring immediate response
 - **Important**: No need to wait for `end=true` to start playing, it will automatically start playing after accumulating enough audio data
 
+#### External Data Mode
+
+The `play()` method starts playback with initial data, then use `sendAudioChunk()` to stream additional audio:
+
+**Audio Format Requirements:**
+- Same as network mode: 16kHz mono PCM16 format
+- Audio data should be provided as `Uint8Array` in chunks with `isLast` flag
+
+**Usage:**
+```typescript
+// Start playback with initial audio and animation data
+// Note: Audio and animation data should be obtained from your backend service
+const initialAudioChunks = [
+  { data: audioData1, isLast: false },
+  { data: audioData2, isLast: false }
+]
+await avatarController.play(initialAudioChunks, initialKeyframes)
+
+// Stream additional audio chunks
+avatarController.sendAudioChunk(audioChunk, isLast)
+```
+
+**Resampling (Both Modes):**
+- If your audio source is at a different sample rate (e.g., 24kHz, 48kHz), you **must** resample it to 16kHz before sending
+- For high-quality resampling, use Web Audio API's `OfflineAudioContext` with anti-aliasing filtering
+- See example projects (`vanilla`, `react`, `vue`) for complete resampling implementation
+
 ## 🌐 Browser Compatibility
 
 - **Chrome/Edge** 90+ (WebGPU recommended)