@spatius/avatarkit 1.2.0 → 1.3.1-beta.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,965 +1,20 @@
1
1
  # @spatius/avatarkit
2
2
 
3
- Real-time virtual avatar rendering SDK for Web, supporting audio-driven animation and high-quality 3D rendering.
3
+ Real-time virtual avatar rendering SDK for Web audio-driven animation and high-quality 3D rendering.
4
4
 
5
- ## 🚀 Features
6
-
7
- - **High-Quality 3D Rendering** - GPU-accelerated avatar rendering with automatic backend selection
8
- - **Audio-Driven Real-Time Animation** - Send audio data, SDK handles animation and rendering
9
- - **Multi-Avatar Support** - Support multiple avatar instances simultaneously, each with independent state and rendering
10
- - **TypeScript Support** - Complete type definitions and IntelliSense
11
- - **Modular Architecture** - Clear component separation, easy to integrate and extend
12
-
13
- ## 📦 Installation
5
+ ## Installation
14
6
 
15
7
  ```bash
16
8
  npm install @spatius/avatarkit
17
9
  ```
18
10
 
19
- ## 🚧 Release Gate (Hard Rule)
20
-
21
- Release must pass gates before publish. Do not publish by manual ad-hoc commands.
22
-
23
- Required gate checks:
24
-
25
- ```bash
26
- pnpm typecheck
27
- pnpm test
28
- pnpm build
29
- ./tools/check_perf_baseline_release_gate.sh
30
- ```
31
-
32
- If iteration includes bugfixes, `docs/bugfix-history.md` must have completed rows (test mapping + red/green evidence).
33
-
34
- Hotfix bypass is allowed only for emergency and must be recorded:
35
-
36
- ```bash
37
- HOTFIX_BYPASS=1 ./tools/check_perf_baseline_release_gate.sh
38
- ```
39
-
40
- ## 🧪 Benchmark Demo (Web SDK)
41
-
42
- Use the dedicated benchmark demo (independent from `vanilla/`) for perf/render baseline runs:
43
-
44
- ```bash
45
- pnpm demo:benchmark
46
- ```
47
-
48
- ## 🚀 Demo Repository
49
-
50
- <div align="center">
51
-
52
- ### 📌 **Quick Start: Check Out Our Demo Repository**
53
-
54
- We provide complete example code and best practices to help you quickly integrate the SDK.
55
-
56
- **The demo repository includes:**
57
- - ✅ Complete integration examples
58
- - ✅ Usage examples for both Direct mode and Backend mode
59
- - ✅ Audio processing examples (PCM16, WAV, MP3, etc.)
60
- - ✅ Vite configuration examples
61
- - ✅ Next.js configuration examples
62
- - ✅ Best practices for common scenarios
63
-
64
- **[👉 View Demo Repository](https://github.com/spatius-ai/avatarkit-demos)** | *If not yet created, please contact the team*
65
-
66
- </div>
67
-
68
- ---
69
-
70
- ## 🔧 Vite Configuration (Recommended)
71
-
72
- If you are using Vite as your build tool, we strongly recommend using our Vite plugin to automatically handle WASM file configuration. The plugin automatically handles all necessary configurations, so you don't need to set them up manually.
73
-
74
- ### Using the Plugin
75
-
76
- Add the plugin to `vite.config.ts`:
77
-
78
- ```typescript
79
- import { defineConfig } from 'vite'
80
- import { avatarkitVitePlugin } from '@spatius/avatarkit/vite'
81
-
82
- export default defineConfig({
83
- plugins: [
84
- avatarkitVitePlugin(), // Just add this line
85
- ],
86
- })
87
- ```
88
-
89
- ### Plugin Features
90
-
91
- The plugin automatically handles:
92
-
93
- - ✅ **Development Server**: Automatically sets the correct MIME type (`application/wasm`) for WASM files
94
- - ✅ **Build Time**: Automatically copies WASM files to `dist/assets/` directory
95
- - ✅ **Cloudflare Pages**: Automatically generates `_headers` file to ensure WASM files use the correct MIME type
96
- - ✅ **Vite Configuration**: Automatically configures `optimizeDeps`, `assetsInclude`, `assetsInlineLimit`, and other options
97
-
98
- ### Manual Configuration (Without Plugin)
99
-
100
- If you don't use the Vite plugin, you need to manually configure the following:
101
-
102
- ```typescript
103
- // vite.config.ts
104
- export default defineConfig({
105
- optimizeDeps: {
106
- exclude: ['@spatius/avatarkit'],
107
- },
108
- assetsInclude: ['**/*.wasm'],
109
- build: {
110
- assetsInlineLimit: 0,
111
- rollupOptions: {
112
- output: {
113
- assetFileNames: (assetInfo) => {
114
- if (assetInfo.name?.endsWith('.wasm')) {
115
- return 'assets/[name][extname]'
116
- }
117
- return 'assets/[name]-[hash][extname]'
118
- },
119
- },
120
- },
121
- },
122
- // Development server needs to manually configure middleware to set WASM MIME type
123
- configureServer(server) {
124
- server.middlewares.use((req, res, next) => {
125
- if (req.url?.endsWith('.wasm')) {
126
- res.setHeader('Content-Type', 'application/wasm')
127
- }
128
- next()
129
- })
130
- },
131
- })
132
- ```
133
-
134
- ## 🔧 Next.js Configuration
135
-
136
- For Next.js projects, use the `withAvatarkit` wrapper to automatically handle WASM file configuration with webpack.
137
-
138
- ### Using the Plugin
139
-
140
- Wrap your Next.js config in `next.config.mjs`:
141
-
142
- ```javascript
143
- import { withAvatarkit } from '@spatius/avatarkit/next'
144
-
145
- export default withAvatarkit({
146
- // ...your existing Next.js config
147
- })
148
- ```
149
-
150
- ### Plugin Features
151
-
152
- The plugin automatically handles:
153
-
154
- - ✅ **Path Fix**: Patches asset path resolution so WASM files are correctly loaded at `/_next/static/chunks/`
155
- - ✅ **WASM Copying**: Copies `.wasm` files into `static/chunks/` via a custom webpack plugin (client build only)
156
- - ✅ **Content-Type Headers**: Adds `application/wasm` response header for `/_next/static/chunks/*.wasm`
157
- - ✅ **Config Chaining**: Preserves your existing `webpack` and `headers` configurations
158
-
159
- ## 🔐 Authentication
160
-
161
- All environments require an **App ID** and **Session Token** for authentication.
162
-
163
- ### App ID
164
-
165
- The App ID is used to identify your application. You can obtain your App ID by:
166
-
167
- 1. **For Testing**: Use the default test App ID provided in demo repositories (paired with test Session Token, only works with publicly available test avatars like Rohan, Dr.Kellan, Priya, Josh, etc.)
168
- 2. **For Production**: Visit the [Developer Platform](https://app.spatius.ai) to create your own App and avatars. You will receive your own App ID after creating an App.
169
-
170
- ### Session Token
171
-
172
- The Session Token is required for authentication and must be obtained from your SDK provider.
173
-
174
- **⚠️ Important Notes:**
175
- - The Session Token must be valid and not expired
176
- - In production applications, you **must** manually inject a valid Session Token obtained from your SDK provider
177
- - The default Session Token provided in demo repositories is **only for demonstration purposes** and can only be used with test avatars
178
- - If you want to create your own avatars and test them, please visit the [Developer Platform](https://app.spatius.ai) to create your own App and generate Session Tokens
179
-
180
- **How to Set Session Token:**
181
-
182
- ```typescript
183
- // Initialize SDK with App ID
184
- await AvatarSDK.initialize('your-app-id', configuration)
185
-
186
- // Set Session Token (can be called before or after initialization)
187
- // If called before initialization, the token will be automatically set when you initialize the SDK
188
- AvatarSDK.setSessionToken('your-session-token')
189
-
190
- // Get current Session Token
191
- const sessionToken = AvatarSDK.sessionToken
192
- ```
193
-
194
- **Token Management:**
195
- - The Session Token can be set at any time using `AvatarSDK.setSessionToken(token)`
196
- - If you set the token before initializing the SDK, it will be automatically applied during initialization
197
- - If you set the token after initialization, it will be applied immediately
198
- - Handle token refresh logic in your application as needed (e.g., when token expires)
199
-
200
- **For Production Integration:**
201
- - Obtain a valid Session Token from your SDK provider
202
- - Store the token securely (never expose it in client-side code if possible)
203
- - Implement token refresh logic to handle token expiration
204
- - Use `AvatarSDK.setSessionToken(token)` to inject the token programmatically
205
-
206
- ## 🎯 Quick Start
207
-
208
- ### ⚠️ Important: Audio Context Initialization
209
-
210
- **Before using any audio-related features, you MUST initialize the audio context in a user gesture context** (e.g., `click`, `touchstart` event handlers). This is required by browser security policies. Calling `initializeAudioContext()` outside a user gesture will fail.
211
-
212
- ### Basic Usage
213
-
214
- ```typescript
215
- import {
216
- AvatarSDK,
217
- AvatarManager,
218
- AvatarView,
219
- Configuration,
220
- DrivingServiceMode,
221
- LogLevel
222
- } from '@spatius/avatarkit'
223
-
224
- // 1. Initialize SDK
225
-
226
- const configuration: Configuration = {
227
- drivingServiceMode: DrivingServiceMode.direct, // Optional, `direct` is default
228
- // - DrivingServiceMode.direct: Direct mode - SDK handles network communication
229
- // - DrivingServiceMode.backend: Backend mode - Host app provides audio and animation data
230
- logLevel: LogLevel.off, // Optional, 'off' is default
231
- // - LogLevel.off: Disable all logs
232
- // - LogLevel.error: Only error logs
233
- // - LogLevel.warning: Warning and error logs
234
- // - LogLevel.all: All logs (info, warning, error)
235
- audioFormat: { // Default is { channelCount: 1, sampleRate: 16000 }
236
- channelCount: 1, // Fixed to 1 (mono)
237
- sampleRate: 16000 // Supported: 8000, 16000, 22050, 24000, 32000, 44100, 48000 Hz
238
- // ⚠️ Must match your actual audio sample rate. Mismatched sample rate will cause playback issues.
239
- }
240
- }
241
-
242
- await AvatarSDK.initialize('your-app-id', configuration)
243
-
244
- // Set Session Token (required for authentication)
245
- // You must obtain a valid Session Token from your SDK provider
246
- // See Authentication section above for more details
247
- AvatarSDK.setSessionToken('your-session-token')
248
-
249
- // 2. Load avatar
250
- const avatarManager = AvatarManager.shared
251
- const avatar = await avatarManager.load('character-id', (progress) => {
252
- console.log(`Loading progress: ${progress.progress}%`)
253
- })
254
-
255
- // 3. Create view (automatically creates Canvas and AvatarController)
256
- // The playback mode is determined by drivingServiceMode in AvatarSDK configuration
257
- // - DrivingServiceMode.direct: Direct mode - SDK handles network communication
258
- // - DrivingServiceMode.backend: Backend mode - Host app provides audio and animation data
259
- const container = document.getElementById('avatar-container')
260
- const avatarView = new AvatarView(avatar, container)
261
-
262
- // 4. ⚠️ CRITICAL: Initialize audio context (MUST be called in user gesture context)
263
- // This method MUST be called within a user gesture event handler (click, touchstart, etc.)
264
- // to satisfy browser security policies. Calling it outside a user gesture will fail.
265
- button.addEventListener('click', async () => {
266
- // Initialize audio context - MUST be in user gesture context
267
- await avatarView.controller.initializeAudioContext()
268
-
269
- // 5. Start real-time communication (Direct mode only)
270
- // Note: start() initiates the WebSocket connection asynchronously.
271
- // Wait for onConnectionState === 'connected' before calling send().
272
- await avatarView.controller.start()
273
-
274
- // 6. Wait for connection to be ready
275
- await new Promise<void>((resolve) => {
276
- avatarView.controller.onConnectionState = (state) => {
277
- if (state === ConnectionState.connected) resolve()
278
- }
279
- })
280
-
281
- // 7. Send audio data (Direct mode, must be mono PCM16 format matching configured sample rate)
282
- // audioData: ArrayBuffer or Uint8Array containing PCM16 (S16LE) audio samples
283
- // ⚠️ Byte length MUST be even (2 bytes per sample). Odd-length data will cause server-side
284
- // validation error and WebSocket disconnect.
285
- // - PCM files: Can be directly read as ArrayBuffer
286
- // - WAV files: Extract PCM data from WAV format (may require resampling)
287
- // - MP3 files: Decode first (e.g., using AudioContext.decodeAudioData()), then convert to PCM16
288
- const audioData = new ArrayBuffer(1024) // Placeholder: Replace with actual PCM16 audio data
289
- avatarView.controller.send(audioData, false) // Send audio data
290
- avatarView.controller.send(audioData, true) // end=true marks the end of current conversation round
291
- })
292
- ```
293
-
294
- ### Host Mode Example
295
-
296
- ```typescript
297
-
298
- // 1-3. Same as Direct mode (initialize SDK, load avatar)
299
-
300
- // 3. Create view with Backend mode
301
- const container = document.getElementById('avatar-container')
302
- const avatarView = new AvatarView(avatar, container)
303
-
304
- // 4. ⚠️ CRITICAL: Initialize audio context (MUST be called in user gesture context)
305
- // This method MUST be called within a user gesture event handler (click, touchstart, etc.)
306
- // to satisfy browser security policies. Calling it outside a user gesture will fail.
307
- button.addEventListener('click', async () => {
308
- // Initialize audio context - MUST be in user gesture context
309
- await avatarView.controller.initializeAudioContext()
310
-
311
- // 5. Host Mode Workflow:
312
- // Send audio data first to get conversationId, then use it to send animation data
313
- const conversationId = avatarView.controller.yieldAudioData(audioData, false)
314
- avatarView.controller.yieldFramesData(animationDataArray, conversationId) // animationDataArray: (Uint8Array | ArrayBuffer)[]
315
- ```
316
-
317
- ### Complete Examples
318
-
319
- This SDK supports two usage modes:
320
- - Direct mode: Real-time audio input with automatic animation data reception
321
- - Backend mode: Custom data sources with manual audio/animation data management
322
-
323
- ## 🏗️ Architecture Overview
324
-
325
- ### Core Components
326
-
327
- - **AvatarSDK** - SDK initialization and management
328
- - **AvatarManager** - Avatar resource loading and management
329
- - **AvatarView** - 3D rendering view
330
- - **AvatarController** - Audio/animation playback controller
331
-
332
- ### Playback Modes
333
-
334
- The SDK supports two playback modes, configured in `AvatarSDK.initialize()`:
335
-
336
- #### 1. SDK Mode (Default)
337
- - Configured via `drivingServiceMode: DrivingServiceMode.direct` in `AvatarSDK.initialize()`
338
- - SDK handles network communication automatically
339
- - Send audio data via `AvatarController.send()`
340
- - SDK receives animation data from backend and synchronizes playback
341
- - Best for: Real-time audio input scenarios
342
-
343
- #### 2. Host Mode
344
- - Configured via `drivingServiceMode: DrivingServiceMode.backend` in `AvatarSDK.initialize()`
345
- - Host application manages its own network/data fetching
346
- - Host application provides both audio and animation data
347
- - SDK only handles synchronized playback
348
- - Best for: Custom data sources, pre-recorded content, or custom network implementations
349
-
350
- **Note:** The playback mode is determined by `drivingServiceMode` in `AvatarSDK.initialize()` configuration.
351
-
352
- ### Fallback Mechanism
353
-
354
- The SDK includes a fallback mechanism to ensure audio playback continues even when animation data is unavailable:
355
-
356
- - **SDK Mode Connection Failure**: If connection fails to establish within 15 seconds, the SDK automatically enters fallback mode. Audio data can still be sent and will play normally, even though no animation data will be received. This ensures audio playback is not interrupted.
357
- - **SDK Mode Server Error**: If the server returns an error after connection is established, the SDK automatically enters audio-only mode for that session.
358
- - **Host Mode**: If empty animation data is provided (empty array or undefined), the SDK automatically enters audio-only mode.
359
- - Once in audio-only mode, any subsequent animation data for that session will be ignored, and only audio will continue playing.
360
- - The fallback mode is interruptible, just like normal playback mode.
361
- - Connection state callbacks (`onConnectionState`) will notify you when connection fails or times out.
362
-
363
- ### Data Flow
364
-
365
- #### SDK Mode Flow
366
-
367
- ```
368
- Audio input (PCM16 mono)
369
-
370
- AvatarController.send()
371
-
372
- Backend processing → Animation data
373
-
374
- SDK synchronizes audio + animation playback
375
-
376
- GPU rendering → Canvas
377
- ```
378
-
379
- #### Host Mode Flow
380
-
381
- ```
382
- External data source (audio + animation)
383
-
384
- AvatarController.yieldAudioData(audioChunk) → returns conversationId
385
- AvatarController.yieldFramesData(dataArray, conversationId)
386
-
387
- SDK synchronizes audio + animation playback
388
-
389
- GPU rendering → Canvas
390
- ```
391
-
392
- ### Audio Format Requirements
393
-
394
- **⚠️ Important:** The SDK requires audio data to be in **mono PCM16** format:
395
-
396
- - **Sample Rate**: Configurable via `audioFormat.sampleRate` in SDK initialization (default: 16000 Hz)
397
- - Supported sample rates: 8000, 16000, 22050, 24000, 32000, 44100, 48000 Hz
398
- - The configured sample rate will be used for both audio recording and playback
399
- - **Channels**: Mono (single channel) - Fixed to 1 channel
400
- - **Format**: PCM16 (16-bit signed integer, little-endian)
401
- - **Byte Order**: Little-endian
402
-
403
- **Audio Data Format:**
404
- - Each sample is 2 bytes (16-bit signed integer, little-endian)
405
- - Audio data should be provided as `ArrayBuffer` or `Uint8Array`
406
- - For example, with 16kHz sample rate: 1 second of audio = 16000 samples × 2 bytes = 32000 bytes
407
- - For 48kHz sample rate: 1 second of audio = 48000 samples × 2 bytes = 96000 bytes
408
-
409
- **Audio Data Source:**
410
- The `audioData` parameter represents raw PCM16 audio samples in the configured sample rate and mono format. Common audio sources include:
411
- - **PCM files**: Raw PCM16 files can be directly read as `ArrayBuffer` or `Uint8Array` and sent to the SDK (ensure sample rate matches configuration)
412
- - **WAV files**: WAV files contain PCM16 audio data in their data chunk. After extracting the PCM data from the WAV file format, it can be sent to the SDK (may require resampling if sample rate differs)
413
- - **MP3 files**: MP3 files need to be decoded first (e.g., using `AudioContext.decodeAudioData()` or a decoder library), then converted from the decoded format to PCM16 before sending to the SDK
414
- - **Microphone input**: Real-time microphone audio needs to be captured and converted to PCM16 format at the configured sample rate before sending
415
- - **Other audio sources**: Any audio source must be converted to mono PCM16 format at the configured sample rate before sending
416
-
417
- **Example: Processing WAV and MP3 Files:**
418
- ```typescript
419
- // WAV file processing
420
- async function processWAVFile(wavFile: File): Promise<ArrayBuffer> {
421
- const arrayBuffer = await wavFile.arrayBuffer()
422
- const view = new DataView(arrayBuffer)
423
-
424
- // WAV format: Skip header (usually 44 bytes for standard WAV)
425
- // Check RIFF header
426
- if (view.getUint32(0, true) !== 0x46464952) { // "RIFF"
427
- throw new Error('Invalid WAV file')
428
- }
429
-
430
- // Find "data" chunk (offset may vary)
431
- let dataOffset = 44 // Standard WAV header size
432
- // For non-standard WAV files, you may need to search for "data" chunk
433
- // This is a simplified example - production code should parse chunks properly
434
-
435
- const pcmData = arrayBuffer.slice(dataOffset)
436
- return pcmData
437
- }
438
-
439
- // MP3 file processing
440
- async function processMP3File(mp3File: File, targetSampleRate: number): Promise<ArrayBuffer> {
441
- const arrayBuffer = await mp3File.arrayBuffer()
442
- const audioContext = new AudioContext({ sampleRate: targetSampleRate })
443
-
444
- // Decode MP3 to AudioBuffer
445
- const audioBuffer = await audioContext.decodeAudioData(arrayBuffer.slice(0))
446
-
447
- // Convert AudioBuffer to PCM16 ArrayBuffer
448
- const length = audioBuffer.length
449
- const channels = audioBuffer.numberOfChannels
450
- const pcm16Buffer = new ArrayBuffer(length * 2)
451
- const pcm16View = new DataView(pcm16Buffer)
452
-
453
- // Mix down to mono if stereo
454
- const sourceData = channels === 1
455
- ? audioBuffer.getChannelData(0)
456
- : new Float32Array(length)
457
-
458
- if (channels > 1) {
459
- const leftChannel = audioBuffer.getChannelData(0)
460
- const rightChannel = audioBuffer.getChannelData(1)
461
- for (let i = 0; i < length; i++) {
462
- sourceData[i] = (leftChannel[i] + rightChannel[i]) / 2 // Mix to mono
463
- }
464
- }
465
-
466
- // Convert float32 (-1.0 to 1.0) to int16 (-32768 to 32767)
467
- for (let i = 0; i < length; i++) {
468
- const sample = Math.max(-1, Math.min(1, sourceData[i])) // Clamp
469
- const int16Sample = sample < 0 ? sample * 0x8000 : sample * 0x7FFF
470
- pcm16View.setInt16(i * 2, int16Sample, true) // little-endian
471
- }
472
-
473
- audioContext.close()
474
- return pcm16Buffer
475
- }
476
-
477
- // Usage example:
478
- // const wavPcmData = await processWAVFile(wavFile)
479
- // avatarView.controller.send(wavPcmData, false)
480
- //
481
- // const mp3PcmData = await processMP3File(mp3File, 16000) // 16kHz
482
- // avatarView.controller.send(mp3PcmData, false)
483
- ```
484
-
485
- **Resampling:**
486
- - If your audio source is at a different sample rate, you must resample it to match the configured sample rate before sending to the SDK
487
- - For high-quality resampling, we recommend using Web Audio API's `OfflineAudioContext` with anti-aliasing filtering
488
- - See example projects for resampling implementation
489
-
490
- **Configuration Example:**
491
- ```typescript
492
- const configuration: Configuration = {
493
- audioFormat: {
494
- channelCount: 1, // Fixed to 1 (mono)
495
- sampleRate: 48000 // Choose from: 8000, 16000, 22050, 24000, 32000, 44100, 48000
496
- }
497
- }
498
- ```
499
-
500
- ## 📚 API Reference
501
-
502
- ### AvatarSDK
503
-
504
- The core management class of the SDK, responsible for initialization and global configuration.
505
-
506
- ```typescript
507
- // Initialize SDK
508
- await AvatarSDK.initialize(appId: string, configuration: Configuration)
509
-
510
- // Check initialization status
511
- const isInitialized = AvatarSDK.isInitialized
512
-
513
- // Get initialized app ID
514
- const appId = AvatarSDK.appId
515
-
516
- // Get configuration
517
- const config = AvatarSDK.configuration
518
-
519
- // Set Session Token (required for authentication)
520
- // You must obtain a valid Session Token from your SDK provider
521
- // See Authentication section for more details
522
- AvatarSDK.setSessionToken('your-session-token')
523
-
524
- // Set userId (optional, for telemetry)
525
- AvatarSDK.setUserId('user-id')
526
-
527
- // Get sessionToken
528
- const sessionToken = AvatarSDK.sessionToken
529
-
530
- // Get userId
531
- const userId = AvatarSDK.userId
532
-
533
- // Get SDK version
534
- const version = AvatarSDK.version
535
-
536
- // Cleanup resources (must be called when no longer in use)
537
- AvatarSDK.cleanup()
538
- ```
539
-
540
- ### AvatarManager
541
-
542
- Avatar resource manager, responsible for downloading, caching, and loading avatar data. Use the singleton instance via `AvatarManager.shared`.
543
-
544
- ```typescript
545
- // Get singleton instance
546
- const manager = AvatarManager.shared
547
-
548
- // Load avatar
549
- const avatar = await manager.load(
550
- id: string,
551
- onProgress?: (progress: LoadProgressInfo) => void
552
- )
553
-
554
- // Clear cache
555
- manager.clearAll()
556
- ```
557
-
558
- ### AvatarView
559
-
560
- 3D rendering view, responsible for 3D rendering only. Internally automatically creates and manages `AvatarController`.
561
-
562
- ```typescript
563
- constructor(avatar: Avatar, container: HTMLElement)
564
- ```
565
-
566
- **Parameters:**
567
- - `avatar`: Avatar instance
568
- - `container`: Canvas container element (required)
569
- - Canvas automatically uses the full size of the container (width and height)
570
- - Canvas aspect ratio adapts to container size - set container size to control aspect ratio
571
- - Canvas will be automatically added to the container
572
- - SDK automatically handles resize events via ResizeObserver
573
-
574
- **Playback Mode:**
575
- - The playback mode is determined by `drivingServiceMode` in `AvatarSDK.initialize()` configuration
576
- - The playback mode is fixed when creating `AvatarView` and persists throughout its lifecycle
577
- - Cannot be changed after creation
578
-
579
- ```typescript
580
- // Create view (Canvas is automatically added to container)
581
- const container = document.getElementById('avatar-container')
582
- const avatarView = new AvatarView(avatar, container)
583
-
584
- // Wait for first frame to render
585
- avatarView.onFirstRendering = () => {
586
- // First frame rendered
587
- }
588
-
589
- // Get or set avatar transform (position and scale)
590
- // Get current transform
591
- const currentTransform = avatarView.avatarTransform // { x: number, y: number, scale: number }
592
-
593
- // Set transform
594
- avatarView.avatarTransform = { x, y, scale }
595
- // - x: Horizontal offset in normalized coordinates (-1 to 1, where -1 = left edge, 0 = center, 1 = right edge)
596
- // - y: Vertical offset in normalized coordinates (-1 to 1, where -1 = bottom edge, 0 = center, 1 = top edge)
597
- // - scale: Scale factor (1.0 = original size, 2.0 = double size, 0.5 = half size)
598
-
599
- // Cleanup resources (must be called before switching avatars)
600
- avatarView.dispose()
601
- ```
602
-
603
- **Switching Avatars:**
604
-
605
- To switch avatars, dispose the old view and create a new one. Do NOT attempt to reuse or reset an existing AvatarView.
606
- - `AvatarSDK.initialize()` and session token do not need to be called again.
607
- - The old AvatarView's internal state is fully cleaned up by `dispose()`.
608
-
609
- ```typescript
610
- // 1. Dispose old avatar
611
- if (currentAvatarView) {
612
- currentAvatarView.dispose()
613
- }
614
-
615
- // 2. Load new avatar (SDK is already initialized, token is still valid)
616
- const newAvatar = await AvatarManager.shared.load('new-character-id')
617
-
618
- // 3. Create new AvatarView
619
- currentAvatarView = new AvatarView(newAvatar, container)
620
-
621
- // 4. Start connection if Direct mode
622
- await currentAvatarView.controller.start()
623
- ```
624
-
625
- ### AvatarController
626
-
627
- Audio/animation playback controller, manages synchronized playback of audio and animation. Automatically handles network communication in Direct mode.
628
-
629
- **Two Usage Patterns:**
630
-
631
- #### SDK Mode Methods
632
-
633
- ```typescript
634
- // ⚠️ CRITICAL: Initialize audio context first (MUST be called in user gesture context)
635
- // This method MUST be called within a user gesture event handler (click, touchstart, etc.)
636
- // to satisfy browser security policies. Calling it outside a user gesture will fail.
637
- // All audio operations (start, send, etc.) require prior initialization.
638
- button.addEventListener('click', async () => {
639
- // Initialize audio context - MUST be in user gesture context
640
- await avatarView.controller.initializeAudioContext()
641
-
642
- // Start service
643
- await avatarView.controller.start()
644
-
645
- // Send audio data (must be mono PCM16 format matching configured sample rate)
646
- const conversationId = avatarView.controller.send(audioData: ArrayBuffer, end: boolean)
647
- // Returns: conversationId - Conversation ID for this conversation session
648
- // end: false (default) - Continue sending audio data for current conversation
649
- // end: true - Mark the end of audio input for current conversation round. The avatar will continue playing remaining animation until finished, then automatically return to idle (notified via onConversationState). After end=true, sending new audio data will interrupt any ongoing playback from the previous conversation round
650
- })
651
-
652
- // Close service
653
- avatarView.controller.close()
654
- ```
655
-
656
- #### Host Mode Methods
657
-
658
- ```typescript
659
- // ⚠️ CRITICAL: Initialize audio context first (MUST be called in user gesture context)
660
- // This method MUST be called within a user gesture event handler (click, touchstart, etc.)
661
- // to satisfy browser security policies. Calling it outside a user gesture will fail.
662
- // All audio operations (yieldAudioData, yieldFramesData, etc.) require prior initialization.
663
- button.addEventListener('click', async () => {
664
- // Initialize audio context - MUST be in user gesture context
665
- await avatarView.controller.initializeAudioContext()
666
-
667
- // Stream audio chunks (must be mono PCM16 format matching configured sample rate)
668
- const conversationId = avatarView.controller.yieldAudioData(
669
- data: Uint8Array, // Audio chunk data (PCM16 format)
670
- isLast: boolean = false // Whether this is the last chunk
671
- )
672
- // Returns: conversationId - Conversation ID for this audio session
673
-
674
- // Stream animation keyframes (requires conversationId from audio data)
675
- avatarView.controller.yieldFramesData(
676
- keyframesDataArray: (Uint8Array | ArrayBuffer)[], // Animation keyframes binary data array
677
- conversationId: string // Conversation ID (required)
678
- )
679
- })
680
- ```
681
-
682
- **⚠️ Important: Conversation ID (conversationId) Management**
683
-
684
- **SDK Mode:**
685
- - `send()` returns a conversationId to distinguish each conversation round
686
- - `end=true` marks the end of a conversation round
687
-
688
- **Host Mode:**
689
- - `yieldAudioData()` returns a conversationId (automatically generates if starting new session)
690
- - `yieldFramesData()` requires a valid conversationId parameter
691
- - Animation data with mismatched conversationId will be **discarded**
692
-
693
- #### Common Methods (Both Modes)
694
-
695
- ```typescript
696
-
697
- // Pause playback (from playing state)
698
- avatarView.controller.pause()
699
-
700
- // Resume playback (from paused state)
701
- await avatarView.controller.resume()
702
-
703
- // Interrupt current playback (stops and clears data)
704
- avatarView.controller.interrupt()
705
-
706
- // Volume control (affects only avatar audio player, not system volume)
707
- avatarView.controller.setVolume(0.5) // Set volume to 50% (0.0 to 1.0)
708
- const currentVolume = avatarView.controller.getVolume() // Get current volume (0.0 to 1.0)
709
-
710
- // Set event callbacks
711
- avatarView.controller.onConnectionState = (state: ConnectionState) => {} // Direct mode only
712
- avatarView.controller.onConversationState = (state: ConversationState) => {}
713
- avatarView.controller.onError = (error: AvatarError) => {} // Includes error.code for specific error type
714
- ```
715
-
716
- #### Avatar Transform Methods
717
-
718
- ```typescript
719
- // Get or set avatar transform (position and scale in canvas)
720
- // Get current transform
721
- const currentTransform = avatarView.avatarTransform // { x: number, y: number, scale: number }
722
-
723
- // Set transform
724
- avatarView.avatarTransform = { x, y, scale }
725
- // - x: Horizontal offset in normalized coordinates (-1 to 1, where -1 = left edge, 0 = center, 1 = right edge)
726
- // - y: Vertical offset in normalized coordinates (-1 to 1, where -1 = bottom edge, 0 = center, 1 = top edge)
727
- // - scale: Scale factor (1.0 = original size, 2.0 = double size, 0.5 = half size)
728
- // Example:
729
- avatarView.avatarTransform = { x: 0, y: 0, scale: 1.0 } // Center, original size
730
- avatarView.avatarTransform = { x: 0.5, y: 0, scale: 2.0 } // Right half, double size
731
- ```
732
-
733
- **Important Notes:**
734
- - `start()` and `close()` are only available in Direct mode
735
- - `yieldAudioData()` and `yieldFramesData()` are only available in Backend mode
736
- - `pause()`, `resume()`, `interrupt()`, `setVolume()`, and `getVolume()` are available in both modes
737
- - The playback mode is determined when creating `AvatarView` and cannot be changed
738
-
739
- ## 🔧 Configuration
740
-
741
- ### Configuration
742
-
743
- ```typescript
744
- interface Configuration {
745
- drivingServiceMode?: DrivingServiceMode // Optional, default is `direct` (Direct mode)
746
- logLevel?: LogLevel // Optional, default is 'off' (no logs)
747
- audioFormat?: AudioFormat // Optional, default is { channelCount: 1, sampleRate: 16000 }
748
- }
749
-
750
- interface AudioFormat {
751
- readonly channelCount: 1 // Fixed to 1 (mono)
752
- readonly sampleRate: number // Supported: 8000, 16000, 22050, 24000, 32000, 44100, 48000 Hz, default: 16000
753
- }
754
- ```
755
-
756
- ### LogLevel
757
-
758
- Control the verbosity of SDK logs:
759
-
760
- ```typescript
761
- enum LogLevel {
762
- off = 'off', // Disable all logs
763
- error = 'error', // Only error logs
764
- warning = 'warning', // Warning and error logs
765
- all = 'all' // All logs (info, warning, error) - default
766
- }
767
- ```
768
-
769
- **Note:** `LogLevel.off` completely disables all logging, including error logs. Use with caution in production environments.
770
-
771
- **Description:**
772
- - `drivingServiceMode`: Specifies the driving service mode
773
- - `DrivingServiceMode.direct` (default): Direct mode - SDK handles network communication automatically
774
- - `DrivingServiceMode.backend`: Backend mode - Host application provides audio and animation data
775
- - `logLevel`: Controls the verbosity of SDK logs
776
- - `LogLevel.off` (default): Disable all logs
777
- - `LogLevel.error`: Only error logs
778
- - `LogLevel.warning`: Warning and error logs
779
- - `LogLevel.all`: All logs (info, warning, error)
780
- - `audioFormat`: Configures audio sample rate and channel count
781
- - `channelCount`: Fixed to 1 (mono channel)
782
- - `sampleRate`: Audio sample rate in Hz (default: 16000)
783
- - Supported values: 8000, 16000, 22050, 24000, 32000, 44100, 48000
784
- - The configured sample rate will be used for both audio recording and playback
785
- - `sessionToken`: **Required for authentication**. Set separately via `AvatarSDK.setSessionToken()`, not in Configuration. See [Authentication](#-authentication) section for details
786
-
787
- ### CameraConfig
788
-
789
- ```typescript
790
- interface CameraConfig {
791
- position: [number, number, number] // Camera position
792
- target: [number, number, number] // Camera target
793
- fov: number // Field of view angle
794
- near: number // Near clipping plane
795
- far: number // Far clipping plane
796
- up?: [number, number, number] // Up direction
797
- aspect?: number // Aspect ratio
798
- }
799
- ```
800
-
801
- ## 📊 State Management
802
-
803
- ### ConnectionState
804
-
805
- ```typescript
806
- enum ConnectionState {
807
- disconnected = 'disconnected',
808
- connecting = 'connecting',
809
- connected = 'connected',
810
- failed = 'failed'
811
- }
812
- ```
813
-
814
- ### ConversationState
815
-
816
- ```typescript
817
- enum ConversationState {
818
- idle = 'idle', // Idle state (breathing animation)
819
- playing = 'playing', // Playing state (active conversation)
820
- pausing = 'pausing' // Pausing state (paused during playback)
821
- }
822
- ```
823
-
824
- **State Description:**
825
- - `idle`: Avatar is in idle state (breathing animation), waiting for conversation to start
826
- - `playing`: Avatar is playing conversation content (including during transition animations)
827
- - `pausing`: Avatar playback is paused (e.g., when `end=false` and waiting for more audio data)
828
-
829
- **Note:** During transition animations, the target state is notified immediately:
830
- - When transitioning from `idle` to `playing`, the `playing` state is notified immediately
831
- - When transitioning from `playing` to `idle`, the `idle` state is notified immediately
832
-
833
- ## 🎨 Rendering System
834
-
835
- The SDK automatically selects the best rendering backend for your browser, no manual configuration needed.
836
-
837
- ## 🚨 Error Handling
838
-
839
- ### AvatarError
840
-
841
- The SDK uses custom error types, providing more detailed error information:
842
-
843
- ```typescript
844
- import { AvatarError } from '@spatius/avatarkit'
845
-
846
- try {
847
- await avatarView.controller.start()
848
- } catch (error) {
849
- if (error instanceof AvatarError) {
850
- console.error('SDK Error:', error.message, error.code)
851
- } else {
852
- console.error('Unknown error:', error)
853
- }
854
- }
855
- ```
856
-
857
- ### Error Callbacks
858
-
859
- ```typescript
860
- import { AvatarError } from '@spatius/avatarkit'
861
-
862
- avatarView.controller.onError = (error: AvatarError) => {
863
- console.error('Error:', error.code, error.message)
864
- }
865
- ```
866
-
867
- `error.code` values (from `ErrorCode` enum):
868
-
869
- | Code | Description | Trigger |
870
- |------|-------------|---------|
871
- | **Authentication & Authorization** | | |
872
- | `appIDUnrecognized` | App ID not recognized | Reserved |
873
- | `sessionTokenInvalid` | Token invalid or appId mismatch | WebSocket close code 4010 |
874
- | `sessionTokenExpired` | Token expired | WebSocket close code 4010 |
875
- | `insufficientBalance` | Insufficient balance | WebSocket close code 4001 |
876
- | `concurrentLimitExceeded` | Concurrent connection limit exceeded | WebSocket close code 4003 |
877
- | **Resource Loading** | | |
878
- | `avatarIDUnrecognized` | Avatar ID not found | Server error |
879
- | `failedToFetchAvatarMetadata` | Metadata fetch failed | Network/server error |
880
- | `failedToDownloadAvatarAssets` | Asset download failed | Network/server error |
881
- | **Connection** | | |
882
- | `websocketError` | WebSocket handshake or network error | Connection failure |
883
- | `websocketClosedAbnormally` | Connection closed abnormally | Close code 1006 |
884
- | `websocketClosedUnexpected` | Unexpected close code | Unknown close code |
885
- | `sessionTimeout` | Session timeout | WebSocket close code 4002 |
886
- | `connectionInProgress` | Connection already in progress | Duplicate `start()` call |
887
- | **Playback** | | |
888
- | `networkLayerNotAvailable` | Network layer not available | `send()` in backend mode |
889
- | `playbackStartFailed` | Failed to start playback | Internal error |
890
- | `playbackInitFailed` | Playback initialization failed | Internal error |
891
- | `audioOnlyInitFailed` | Audio-only playback init failed | Fallback mode error |
892
- | `noAudio` | No audio data to play | Empty audio input |
893
- | `audioContextNotInitialized` | Audio context not initialized | `send()` before `initializeAudioContext()` |
894
- | `animationPlayerNotInitialized` | Animation player not initialized | Internal error |
895
- | **Server** | | |
896
- | `serverError` | Server-side error | Server MESSAGE_SERVER_ERROR |
897
-
898
- ## 🔄 Resource Management
899
-
900
- ### Lifecycle Management
901
-
902
- #### SDK Mode Lifecycle
903
-
904
- ```typescript
905
- // Initialize
906
- const container = document.getElementById('avatar-container')
907
- const avatarView = new AvatarView(avatar, container)
908
- await avatarView.controller.start()
909
-
910
- // Use
911
- avatarView.controller.send(audioData, false)
912
-
913
- // Cleanup - dispose() automatically cleans up all resources including connections
914
- avatarView.dispose()
915
- ```
916
-
917
- #### Host Mode Lifecycle
918
-
919
- ```typescript
920
- // Initialize
921
- const container = document.getElementById('avatar-container')
922
- const avatarView = new AvatarView(avatar, container)
923
-
924
- // Use
925
- const conversationId = avatarView.controller.yieldAudioData(audioChunk, false)
926
- avatarView.controller.yieldFramesData(keyframesDataArray, conversationId)
927
-
928
- // Cleanup - dispose() automatically cleans up all resources including playback data
929
- avatarView.dispose()
930
- ```
931
-
932
- **⚠️ Important Notes:**
933
- - `dispose()` automatically cleans up all resources, including:
934
- - Network connections (Direct mode)
935
- - Playback data and animation resources (both modes)
936
- - Render system and canvas elements
937
- - All event listeners and callbacks
938
- - Not properly calling `dispose()` may cause resource leaks and rendering errors
939
- - If you need to manually close connections before disposing, you can call `avatarView.controller.close()` (Direct mode) first, but it's not required as `dispose()` handles this automatically
940
-
941
- ### Memory Optimization
942
-
943
- - SDK automatically manages memory allocation
944
- - Supports dynamic loading/unloading of avatar and animation resources
945
-
946
- ## 🌐 Browser Compatibility
947
-
948
- - **Chrome/Edge** 90+ (WebGPU recommended)
949
- - **Firefox** 90+ (WebGL)
950
- - **Safari** 14+ (WebGL)
951
- - **Mobile** iOS 14+, Android 8+
952
-
953
- ## 📝 License
954
-
955
- MIT License
11
+ ## Documentation
956
12
 
957
- ## 🤝 Contributing
13
+ Full quickstart, API reference, and integration guides:
958
14
 
959
- Issues and Pull Requests are welcome!
15
+ **https://docs.spatius.ai/quickstarts/web-sdk**
960
16
 
961
- ## 📞 Support
17
+ ## Support
962
18
 
963
- For questions, please contact:
964
- - Email: code@spatius.net
965
- - Documentation: https://docs.spatius.ai
19
+ - Email: hello@spatialwalk.net
20
+ - Docs: https://docs.spatius.ai