@spatialwalk/avatarkit 1.0.0-beta.62 → 1.0.0-beta.63

package/CHANGELOG.md

@@ -2,6 +2,24 @@
 
 All notable changes to this project will be documented in this file.
 
+## [1.0.0-beta.63] - 2026-01-14
+
+### ✨ New Features
+- **Audio Context Initialization API** - Added `initializeAudioContext()` method to `AvatarController`
+  - Must be called in a user gesture context (click, touchstart, etc.) before any audio operations
+  - Ensures AudioContext is created and initialized in a user gesture context, preventing browser security policy issues
+  - All audio operations (`send()`, `yieldAudioData()`, `start()`, `playback()`, etc.) now require prior initialization
+
+### 🔧 Improvements
+- **Initialization Flow** - Removed all lazy initialization logic for audio context
+  - Audio context initialization is now centralized in `initializeAudioContext()` method
+  - All audio operations check for initialization before proceeding
+  - Clear error messages when audio operations are attempted without initialization
+
+### 🐛 Bugfixes
+- **Audio Context User Gesture Requirement** - Fixed issue where AudioContext could not be properly initialized when external applications request recording permissions
+  - Audio context must now be initialized in user gesture context, ensuring browser security policies are satisfied
+
 ## [1.0.0-beta.62] - 2026-01-14
 
 ### ✨ New Features

package/README.md

@@ -20,6 +20,10 @@ npm install @spatialwalk/avatarkit
 
 ## 🎯 Quick Start
 
+### ⚠️ Important: Audio Context Initialization
+
+**Before using any audio-related features, you MUST initialize the audio context in a user gesture context** (e.g., `click`, `touchstart` event handlers). This is required by browser security policies. Calling `initializeAudioContext()` outside a user gesture will fail.
+
 ### Basic Usage
 
 ```typescript
@@ -70,13 +74,21 @@ const avatar = await avatarManager.load('character-id', (progress) => {
 const container = document.getElementById('avatar-container')
 const avatarView = new AvatarView(avatar, container)
 
-// 4. Start real-time communication (SDK mode only)
-await avatarView.avatarController.start()
-
-// 5. Send audio data (SDK mode, must be mono PCM16 format matching configured sample rate)
-const audioData = new ArrayBuffer(1024) // Example: PCM16 audio data at configured sample rate
-avatarView.avatarController.send(audioData, false) // Send audio data
-avatarView.avatarController.send(audioData, true) // end=true marks the end of current conversation round
+// 4. ⚠️ CRITICAL: Initialize audio context (MUST be called in user gesture context)
+// This method MUST be called within a user gesture event handler (click, touchstart, etc.)
+// to satisfy browser security policies. Calling it outside a user gesture will fail.
+button.addEventListener('click', async () => {
+  // Initialize audio context - MUST be in user gesture context
+  await avatarView.controller.initializeAudioContext()
+  
+  // 5. Start real-time communication (SDK mode only)
+  await avatarView.controller.start()
+  
+  // 6. Send audio data (SDK mode, must be mono PCM16 format matching configured sample rate)
+  const audioData = new ArrayBuffer(1024) // Example: PCM16 audio data at configured sample rate
+  avatarView.controller.send(audioData, false) // Send audio data
+  avatarView.controller.send(audioData, true) // end=true marks the end of current conversation round
+})
 ```
 
 ### Host Mode Example
@@ -89,10 +101,17 @@ avatarView.avatarController.send(audioData, true) // end=true marks the end of c
 const container = document.getElementById('avatar-container')
 const avatarView = new AvatarView(avatar, container)
 
-// 4. Host Mode Workflow:
-// Send audio data first to get conversationId, then use it to send animation data
-const conversationId = avatarView.avatarController.yieldAudioData(audioData, false)
-avatarView.avatarController.yieldFramesData(animationDataArray, conversationId) // animationDataArray: (Uint8Array | ArrayBuffer)[]
+// 4. ⚠️ CRITICAL: Initialize audio context (MUST be called in user gesture context)
+// This method MUST be called within a user gesture event handler (click, touchstart, etc.)
+// to satisfy browser security policies. Calling it outside a user gesture will fail.
+button.addEventListener('click', async () => {
+  // Initialize audio context - MUST be in user gesture context
+  await avatarView.controller.initializeAudioContext()
+  
+  // 5. Host Mode Workflow:
+  // Send audio data first to get conversationId, then use it to send animation data
+  const conversationId = avatarView.controller.yieldAudioData(audioData, false)
+  avatarView.controller.yieldFramesData(animationDataArray, conversationId) // animationDataArray: (Uint8Array | ArrayBuffer)[]
 ```
 
 ### Complete Examples
@@ -350,34 +369,52 @@ Audio/animation playback controller (playback layer), manages synchronized playb
 #### SDK Mode Methods
 
 ```typescript
-// Start WebSocket service
-await avatarView.avatarController.start()
-
-// Send audio data (must be 16kHz mono PCM16 format)
-const conversationId = avatarView.avatarController.send(audioData: ArrayBuffer, end: boolean)
-// Returns: conversationId - Conversation ID for this conversation session
-// end: false (default) - Continue sending audio data for current conversation
-// end: true - Mark the end of current conversation round. After end=true, sending new audio data will interrupt any ongoing playback from the previous conversation round
+// ⚠️ CRITICAL: Initialize audio context first (MUST be called in user gesture context)
+// This method MUST be called within a user gesture event handler (click, touchstart, etc.)
+// to satisfy browser security policies. Calling it outside a user gesture will fail.
+// All audio operations (start, send, etc.) require prior initialization.
+button.addEventListener('click', async () => {
+  // Initialize audio context - MUST be in user gesture context
+  await avatarView.controller.initializeAudioContext()
+  
+  // Start WebSocket service
+  await avatarView.controller.start()
+  
+  // Send audio data (must be 16kHz mono PCM16 format)
+  const conversationId = avatarView.controller.send(audioData: ArrayBuffer, end: boolean)
+  // Returns: conversationId - Conversation ID for this conversation session
+  // end: false (default) - Continue sending audio data for current conversation
+  // end: true - Mark the end of current conversation round. After end=true, sending new audio data will interrupt any ongoing playback from the previous conversation round
+})
 
 // Close WebSocket service
-avatarView.avatarController.close()
+avatarView.controller.close()
 ```
 
 #### Host Mode Methods
 
 ```typescript
-// Stream audio chunks (must be 16kHz mono PCM16 format)
-const conversationId = avatarView.avatarController.yieldAudioData(
-  data: Uint8Array,               // Audio chunk data
-  isLast: boolean = false         // Whether this is the last chunk
-)
-// Returns: conversationId - Conversation ID for this audio session
-
-// Stream animation keyframes (requires conversationId from audio data)
-avatarView.avatarController.yieldFramesData(
-  keyframesDataArray: (Uint8Array | ArrayBuffer)[],  // Animation keyframes binary data array (each element is a protobuf encoded Message)
-  conversationId: string                              // Conversation ID (required)
-)
+// ⚠️ CRITICAL: Initialize audio context first (MUST be called in user gesture context)
+// This method MUST be called within a user gesture event handler (click, touchstart, etc.)
+// to satisfy browser security policies. Calling it outside a user gesture will fail.
+// All audio operations (yieldAudioData, yieldFramesData, etc.) require prior initialization.
+button.addEventListener('click', async () => {
+  // Initialize audio context - MUST be in user gesture context
+  await avatarView.controller.initializeAudioContext()
+  
+  // Stream audio chunks (must be 16kHz mono PCM16 format)
+  const conversationId = avatarView.controller.yieldAudioData(
+    data: Uint8Array,               // Audio chunk data
+    isLast: boolean = false         // Whether this is the last chunk
+  )
+  // Returns: conversationId - Conversation ID for this audio session
+
+  // Stream animation keyframes (requires conversationId from audio data)
+  avatarView.controller.yieldFramesData(
+    keyframesDataArray: (Uint8Array | ArrayBuffer)[],  // Animation keyframes binary data array (each element is a protobuf encoded Message)
+    conversationId: string                              // Conversation ID (required)
+  )
+})
 ```
 
 **⚠️ Important: Conversation ID (conversationId) Management**

package/dist/{StreamingAudioPlayer-BWsAt_s7.js → StreamingAudioPlayer-CO9WTktN.js}

@@ -1,7 +1,7 @@
 var __defProp = Object.defineProperty;
 var __defNormalProp = (obj, key, value) => key in obj ? __defProp(obj, key, { enumerable: true, configurable: true, writable: true, value }) : obj[key] = value;
 var __publicField = (obj, key, value) => __defNormalProp(obj, typeof key !== "symbol" ? key + "" : key, value);
-import { A as APP_CONFIG, l as logger, e as errorToMessage, a as logEvent } from "./index-Bhjn1nq3.js";
+import { A as APP_CONFIG, l as logger, e as errorToMessage, a as logEvent } from "./index-C1md-jKJ.js";
 class StreamingAudioPlayer {
   constructor(options) {
     __publicField(this, "audioContext", null);

package/dist/core/AvatarController.d.ts

@@ -44,6 +44,8 @@ export declare class AvatarController {
         playbackMode?: DrivingServiceMode;
     });
     getCurrentConversationId(): string | null;
+    initializeAudioContext(): Promise<void>;
+    private checkAudioContextInitialized;
     start(): Promise<void>;
     send(audioData: ArrayBuffer, end?: boolean): string | null;
     close(): void;

package/dist/{index-Bhjn1nq3.js → index-C1md-jKJ.js}

@@ -7624,7 +7624,7 @@ const _AnimationPlayer = class _AnimationPlayer {
     if (this.streamingPlayer) {
       return;
     }
-    const { StreamingAudioPlayer } = await import("./StreamingAudioPlayer-BWsAt_s7.js");
+    const { StreamingAudioPlayer } = await import("./StreamingAudioPlayer-CO9WTktN.js");
     const { AvatarSDK: AvatarSDK2 } = await Promise.resolve().then(() => AvatarSDK$1);
     const audioFormat = AvatarSDK2.getAudioFormat();
     this.streamingPlayer = new StreamingAudioPlayer({
@@ -8961,7 +8961,7 @@ class AvatarSDK {
 }
 __publicField(AvatarSDK, "_isInitialized", false);
 __publicField(AvatarSDK, "_configuration", null);
-__publicField(AvatarSDK, "_version", "1.0.0-beta.62");
+__publicField(AvatarSDK, "_version", "1.0.0-beta.63");
 __publicField(AvatarSDK, "_avatarCore", null);
 __publicField(AvatarSDK, "_dynamicSdkConfig", null);
 const AvatarSDK$1 = Object.freeze(Object.defineProperty({
@@ -10741,38 +10741,71 @@ class AvatarController {
   getCurrentConversationId() {
     return this.getEffectiveConversationId();
   }
-  async start() {
-    if (!this.networkLayer) {
-      throw new SPAvatarError(
-        "Network layer not available. Use SDK mode.",
-        "NETWORK_LAYER_NOT_AVAILABLE"
-      );
+  async initializeAudioContext() {
+    var _a;
+    if ((_a = this.animationPlayer) == null ? void 0 : _a.isStreamingReady()) {
+      return;
     }
     if (!this.animationPlayer) {
       this.animationPlayer = new AnimationPlayer();
+    }
+    if (!this.animationPlayer.isStreamingReady()) {
       try {
         await this.animationPlayer.createAndInitializeStreamingPlayer();
       } catch (error) {
         const message = error instanceof Error ? error.message : String(error);
-        logger.error("[AvatarController] Failed to create streaming player:", message);
-        logEvent("character_player", "error", {
-          avatar_id: this.avatar.id,
-          event: "streaming_player_init_failed",
-          reason: message
-        });
-        throw error;
+        logger.error("[AvatarController] Failed to initialize audio context:", message);
+        throw new SPAvatarError(
+          `Failed to initialize audio context: ${message}`,
+          "AUDIO_CONTEXT_INIT_FAILED"
+        );
+      }
+    }
+    const streamingPlayer = this.animationPlayer.getStreamingPlayer();
+    if (streamingPlayer) {
+      const audioContext = streamingPlayer.audioContext;
+      if (audioContext && audioContext.state === "suspended") {
+        try {
+          await audioContext.resume();
+        } catch (err) {
+          logger.warn("[AvatarController] Failed to resume AudioContext during initialization:", err);
+        }
       }
     }
+  }
+  checkAudioContextInitialized() {
+    var _a;
+    if (!((_a = this.animationPlayer) == null ? void 0 : _a.isStreamingReady())) {
+      throw new SPAvatarError(
+        "Audio context not initialized. Call initializeAudioContext() in a user gesture context first.",
+        "AUDIO_CONTEXT_NOT_INITIALIZED"
+      );
+    }
+  }
+  async start() {
+    if (!this.networkLayer) {
+      throw new SPAvatarError(
+        "Network layer not available. Use SDK mode.",
+        "NETWORK_LAYER_NOT_AVAILABLE"
+      );
+    }
+    this.checkAudioContextInitialized();
     await this.networkLayer.connect(this.avatar.id);
   }
   send(audioData, end = false) {
-    var _a, _b, _c;
+    var _a, _b, _c, _d;
+    try {
+      this.checkAudioContextInitialized();
+    } catch (error) {
+      (_a = this.onError) == null ? void 0 : _a.call(this, error);
+      return null;
+    }
     if (!this.networkLayer) {
-      (_a = this.onError) == null ? void 0 : _a.call(this, new SPAvatarError("Network layer not available", "NETWORK_LAYER_NOT_AVAILABLE"));
+      (_b = this.onError) == null ? void 0 : _b.call(this, new SPAvatarError("Network layer not available", "NETWORK_LAYER_NOT_AVAILABLE"));
       return null;
     }
     if (!this.networkLayer.canSend()) {
-      (_b = this.onError) == null ? void 0 : _b.call(this, new SPAvatarError("Service not connected", "NOT_CONNECTED"));
+      (_c = this.onError) == null ? void 0 : _c.call(this, new SPAvatarError("Service not connected", "NOT_CONNECTED"));
       logEvent("character_manager", "warning", {
         avatar_id: this.avatar.id,
         event: "send_not_connected"
@@ -10790,7 +10823,7 @@ class AvatarController {
     }
     if (!this.isPlaying && this.currentState === AvatarState.idle) {
       this.currentState = AvatarState.active;
-      (_c = this.onConversationState) == null ? void 0 : _c.call(this, this.mapToConversationState(AvatarState.active));
+      (_d = this.onConversationState) == null ? void 0 : _d.call(this, this.mapToConversationState(AvatarState.active));
     }
     return this.networkLayer.getCurrentConversationId();
   }
@@ -10810,18 +10843,13 @@ class AvatarController {
     (_a = this.onConnectionState) == null ? void 0 : _a.call(this, ConnectionState.disconnected);
   }
   async playback(initialAudioChunks, initialKeyframes) {
+    this.checkAudioContextInitialized();
     if (this.isPlaying || this.currentConversationId) {
       this.interrupt();
     }
     this.currentConversationId = this.generateAndLogNewConversationId();
     this.reqEnd = false;
     this.clearPlaybackData();
-    if (!this.animationPlayer) {
-      this.animationPlayer = new AnimationPlayer();
-    }
-    if (!this.animationPlayer.isStreamingReady()) {
-      await this.animationPlayer.createAndInitializeStreamingPlayer();
-    }
     if (initialAudioChunks && initialAudioChunks.length > 0) {
       this.pendingAudioChunks.push(...initialAudioChunks);
     }
@@ -10848,7 +10876,13 @@ class AvatarController {
     return this.currentConversationId;
   }
   yieldAudioData(data, isLast = false) {
-    var _a, _b;
+    var _a, _b, _c;
+    try {
+      this.checkAudioContextInitialized();
+    } catch (error) {
+      (_a = this.onError) == null ? void 0 : _a.call(this, error);
+      return null;
+    }
     if (this.reqEnd && this.isPlaying && this.currentConversationId) {
       this.interrupt();
       this.currentConversationId = this.generateAndLogNewConversationId();
@@ -10879,12 +10913,12 @@ class AvatarController {
         metrics.tap2Timestamp = Date.now();
       }
     }
-    if (this.isPlaying && ((_a = this.animationPlayer) == null ? void 0 : _a.isStreamingReady())) {
+    if (this.isPlaying && ((_b = this.animationPlayer) == null ? void 0 : _b.isStreamingReady())) {
       this.animationPlayer.addAudioChunk(data, isLast);
     } else {
       if (data.length > 0 || isLast) {
         this.pendingAudioChunks.push({ data, isLast });
-        (_b = this.onConversationState) == null ? void 0 : _b.call(this, this.mapToConversationState(AvatarState.active));
+        (_c = this.onConversationState) == null ? void 0 : _c.call(this, this.mapToConversationState(AvatarState.active));
       }
     }
     return this.currentConversationId;
@@ -11237,6 +11271,7 @@ class AvatarController {
   }
   async startStreamingPlaybackInternal() {
     var _a, _b, _c;
+    this.checkAudioContextInitialized();
     if (this.isPlaying) {
       this.isStartingPlayback = false;
       return;
@@ -11245,30 +11280,15 @@ class AvatarController {
       return;
     }
     this.isStartingPlayback = true;
-    if (!this.animationPlayer) {
-      this.animationPlayer = new AnimationPlayer();
-    }
-    if (!this.animationPlayer.isStreamingReady()) {
-      try {
-        await this.animationPlayer.createAndInitializeStreamingPlayer();
-      } catch (error) {
-        this.isStartingPlayback = false;
-        const message = error instanceof Error ? error.message : String(error);
-        logger.error("[AvatarController] Failed to create streaming player:", message);
-        logEvent("character_player", "error", {
-          avatar_id: this.avatar.id,
-          event: "streaming_player_init_failed",
-          reason: message
-        });
-        throw error;
-      }
-    }
     if (!this.currentKeyframes || this.currentKeyframes.length === 0) {
       this.isStartingPlayback = false;
       logger.warn("[AvatarController] No animation data to play");
       return;
     }
     try {
+      if (!this.animationPlayer) {
+        throw new SPAvatarError("Animation player not initialized", "ANIMATION_PLAYER_NOT_INITIALIZED");
+      }
       await this.animationPlayer.prepareStreamingPlayer(() => {
         var _a2, _b2;
         this.isPlaying = false;
@@ -11490,21 +11510,7 @@ class AvatarController {
   async startAudioOnlyPlayback() {
     var _a, _b;
     if (!this.animationPlayer) {
-      this.animationPlayer = new AnimationPlayer();
-    }
-    if (!this.animationPlayer.isStreamingReady()) {
-      try {
-        await this.animationPlayer.createAndInitializeStreamingPlayer();
-      } catch (error) {
-        const message = error instanceof Error ? error.message : String(error);
-        logger.error("[AvatarController] Failed to create streaming player for audio-only mode:", message);
-        logEvent("character_player", "error", {
-          avatar_id: this.avatar.id,
-          event: "audio_only_streaming_player_init_failed",
-          reason: message
-        });
-        throw error;
-      }
+      throw new SPAvatarError("Animation player not initialized", "ANIMATION_PLAYER_NOT_INITIALIZED");
     }
     try {
       await this.animationPlayer.prepareStreamingPlayer(() => {
@@ -11587,7 +11593,9 @@ class AvatarController {
   }
   addAudioChunkToBuffer(data, isLast) {
     if (!this.animationPlayer) {
-      this.animationPlayer = new AnimationPlayer();
+      logger.warn("[AvatarController] animationPlayer is null in addAudioChunkToBuffer, this should not happen");
+      this.pendingAudioChunks.push({ data, isLast });
+      return;
     }
     if (this.isPlaying && this.animationPlayer.isStreamingReady()) {
       this.animationPlayer.addAudioChunk(data, isLast);

package/dist/index.js

@@ -1,4 +1,4 @@
-import { b, c, f, d, j, g, C, i, D, E, k, h, L, R, S, m } from "./index-Bhjn1nq3.js";
+import { b, c, f, d, j, g, C, i, D, E, k, h, L, R, S, m } from "./index-C1md-jKJ.js";
 export {
   b as Avatar,
   c as AvatarController,

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@spatialwalk/avatarkit",
   "type": "module",
-  "version": "1.0.0-beta.62",
+  "version": "1.0.0-beta.63",
   "description": "SPAvatar SDK - 3D Gaussian Splatting Avatar Rendering SDK",
   "author": "SPAvatar Team",
   "license": "MIT",