@spatialwalk/avatarkit 1.0.0-beta.1

package/README.md

@@ -0,0 +1,375 @@
+# SPAvatarKit SDK
+
+基于 3D Gaussian Splatting 的实时虚拟人物头像渲染 SDK，支持音频驱动的动画渲染和高质量 3D 渲染。
+
+## 🚀 特性
+
+- **3D Gaussian Splatting 渲染** - 基于最新的点云渲染技术，提供高质量的 3D 虚拟人物
+- **音频驱动的实时动画渲染** - 用户提供音频数据，SDK 负责接收动画数据并渲染
+- **WebGPU/WebGL 双渲染后端** - 自动选择最佳渲染后端，确保兼容性
+- **WASM 高性能计算** - 使用 C++ 编译的 WebAssembly 模块进行几何计算
+- **TypeScript 支持** - 完整的类型定义和智能提示
+- **模块化架构** - 清晰的组件分离，易于集成和扩展
+
+## 📦 安装
+
+```bash
+npm install @spatialwalk/avatarkit
+```
+
+## 🎯 快速开始
+
+### 基础使用
+
+```typescript
+import { 
+  AvatarKit, 
+  AvatarManager, 
+  AvatarView,
+  Configuration,
+  Environment 
+} from '@spatialwalk/avatarkit'
+
+// 1. 初始化 SDK
+const configuration: Configuration = {
+  environment: Environment.test,
+}
+
+await AvatarKit.initialize('your-app-id', configuration)
+
+// 设置 sessionToken（如果需要，单独调用）
+// AvatarKit.setSessionToken('your-session-token')
+
+// 2. 加载角色
+const avatarManager = new AvatarManager()
+const avatar = await avatarManager.load('character-id', (progress) => {
+  console.log(`Loading progress: ${progress.progress}%`)
+})
+
+// 3. 创建视图（自动创建 Canvas 和 AvatarController）
+const container = document.getElementById('avatar-container')
+const avatarView = new AvatarView(avatar, container)
+
+// 4. 启动实时通信
+await avatarView.avatarController.start()
+
+// 5. 发送音频数据
+// 如果音频是 Uint8Array，可以使用 slice().buffer 转换为 ArrayBuffer
+const audioUint8 = new Uint8Array(1024) // 示例：音频数据
+const audioData = audioUint8.slice().buffer // 简化的转换方式，适用于 ArrayBuffer 和 SharedArrayBuffer
+avatarView.avatarController.send(audioData, false) // 发送音频数据，积累到一定量后会自动开始播放
+avatarView.avatarController.send(audioData, true) // end=true 表示立即返回动画数据，不再积累
+```
+
+### 完整示例
+
+查看 GitHub 仓库中的示例代码了解完整的使用流程。
+
+## 🏗️ 架构概览
+
+### 核心组件
+
+- **AvatarKit** - SDK 初始化和管理
+- **AvatarManager** - 角色资源加载和管理
+- **AvatarView** - 3D 渲染视图（内部包含 AvatarController）
+- **AvatarController** - 实时通信和数据处理
+- **AvatarCoreAdapter** - WASM 模块适配器
+
+### 数据流
+
+```
+用户音频输入（16kHz mono PCM） → AvatarController → WebSocket → 后台处理
+                                              ↓
+后台返回动画数据（FLAME 关键帧） → AvatarController → AnimationPlayer
+                                              ↓
+FLAME 参数 → AvatarCore.computeFrameFlatFromParams() → Splat 数据
+                                              ↓
+Splat 数据 → RenderSystem → WebGPU/WebGL → Canvas 渲染
+```
+
+**注意：** 用户需要自己提供音频数据（16kHz mono PCM），SDK 负责接收动画数据并渲染。
+
+## 📚 API 参考
+
+### AvatarKit
+
+SDK 的核心管理类，负责初始化和全局配置。
+
+```typescript
+// 初始化 SDK
+await AvatarKit.initialize(appId: string, configuration: Configuration)
+
+// 检查初始化状态
+const isInitialized = AvatarKit.isInitialized
+
+// 清理资源（不再使用时必须调用）
+AvatarKit.cleanup()
+```
+
+### AvatarManager
+
+角色资源管理器，负责下载、缓存和加载角色数据。
+
+```typescript
+const manager = new AvatarManager()
+
+// 加载角色
+const avatar = await manager.load(
+  characterId: string, 
+  onProgress?: (progress: LoadProgressInfo) => void
+)
+
+// 清理缓存
+manager.clearCache()
+```
+
+### AvatarView
+
+3D 渲染视图，内部自动创建和管理 AvatarController。
+
+```typescript
+// 创建视图（Canvas 会自动添加到容器中）
+const avatarView = new AvatarView(avatar: Avatar, container?: HTMLElement)
+
+// 获取 Canvas 元素
+const canvas = avatarView.getCanvas()
+
+// 设置背景
+avatarView.setBackgroundImage('path/to/image.jpg')
+avatarView.setBackgroundOpaque(true)
+
+// 更新相机配置
+avatarView.updateCameraConfig(cameraConfig: CameraConfig)
+
+// 清理资源
+avatarView.dispose()
+```
+
+### AvatarController
+
+实时通信控制器，处理 WebSocket 连接和动画数据。
+
+```typescript
+// 启动连接
+await avatarView.avatarController.start()
+
+// 发送音频数据
+avatarView.avatarController.send(audioData: ArrayBuffer, end: boolean)
+// audioData: 音频数据（ArrayBuffer 格式）
+// end: false（默认）- 正常发送音频数据，服务端会积累音频数据，积累到一定量后会自动返回动画数据并开始同步播放动画和音频
+// end: true - 立即返回动画数据，不再积累，用于结束当前对话或需要立即响应的场景
+
+// 打断对话
+avatarView.avatarController.interrupt()
+
+// 关闭连接
+avatarView.avatarController.close()
+
+// 设置事件回调
+avatarView.avatarController.onConnectionState = (state: ConnectionState) => {}
+avatarView.avatarController.onAvatarState = (state: AvatarState) => {}
+avatarView.avatarController.onError = (error: Error) => {}
+
+// 注意：不支持 sendText() 方法，调用会抛出错误
+```
+
+## 🔧 配置
+
+### Configuration
+
+```typescript
+interface Configuration {
+  environment: Environment
+}
+```
+
+**说明：**
+- `environment`: 指定环境（cn/us/test），SDK 会根据环境自动使用对应的 API 地址和 WebSocket 地址
+- `sessionToken`: 通过 `AvatarKit.setSessionToken()` 单独设置，而不是在 Configuration 中
+
+enum Environment {
+  cn = 'cn',    // 中国区
+  us = 'us',    // 美国区
+  test = 'test' // 测试环境
+}
+```
+
+### CameraConfig
+
+```typescript
+interface CameraConfig {
+  position: [number, number, number]  // 相机位置
+  target: [number, number, number]    // 相机目标
+  fov: number                         // 视野角度
+  near: number                        // 近裁剪面
+  far: number                         // 远裁剪面
+  up?: [number, number, number]       // 上方向
+  aspect?: number                     // 宽高比
+}
+```
+
+## 📊 状态管理
+
+### ConnectionState
+
+```typescript
+enum ConnectionState {
+  disconnected = 'disconnected',
+  connecting = 'connecting',
+  connected = 'connected',
+  failed = 'failed'
+}
+```
+
+### AvatarState
+
+```typescript
+enum AvatarState {
+  idle = 'idle',      // 空闲状态，呈现呼吸态
+  active = 'active',  // 活跃中，等待可播放内容
+  playing = 'playing' // 播放中
+}
+```
+
+## 🎨 渲染系统
+
+SDK 支持两种渲染后端：
+
+- **WebGPU** - 现代浏览器的高性能渲染
+- **WebGL** - 兼容性更好的传统渲染
+
+渲染系统会自动选择最佳的后端，无需手动配置。
+
+## 🔍 调试和监控
+
+### 日志系统
+
+SDK 内置了完整的日志系统，支持不同级别的日志输出：
+
+```typescript
+import { logger } from '@spatialwalk/avatarkit'
+
+// 设置日志级别
+logger.setLevel('verbose') // 'basic' | 'verbose'
+
+// 手动日志输出
+logger.log('Info message')
+logger.warn('Warning message')
+logger.error('Error message')
+```
+
+### 性能监控
+
+SDK 提供了性能监控接口，可以监控渲染性能：
+
+```typescript
+// 获取渲染性能统计
+const stats = avatarView.getPerformanceStats()
+
+if (stats) {
+  console.log(`渲染耗时: ${stats.renderTime.toFixed(2)}ms`)
+  console.log(`排序耗时: ${stats.sortTime.toFixed(2)}ms`)
+  console.log(`渲染后端: ${stats.backend}`)
+  
+  // 计算帧率
+  const fps = 1000 / stats.renderTime
+  console.log(`帧率: ${fps.toFixed(2)} FPS`)
+}
+
+// 定期监控性能
+setInterval(() => {
+  const stats = avatarView.getPerformanceStats()
+  if (stats) {
+    // 发送到监控服务或显示在 UI 上
+    console.log('Performance:', stats)
+  }
+}, 1000)
+```
+
+**性能统计说明**：
+- `renderTime`: 总渲染耗时（毫秒），包含排序和 GPU 渲染
+- `sortTime`: 排序耗时（毫秒），使用 Radix Sort 算法对点云进行深度排序
+- `backend`: 当前使用的渲染后端（`'webgpu'` | `'webgl'` | `null`）
+
+## 🚨 错误处理
+
+### SPAvatarError
+
+SDK 使用自定义错误类型，提供更详细的错误信息：
+
+```typescript
+import { SPAvatarError } from '@spatialwalk/avatarkit'
+
+try {
+  await avatarView.avatarController.start()
+} catch (error) {
+  if (error instanceof SPAvatarError) {
+    console.error('SDK Error:', error.message, error.code)
+  } else {
+    console.error('Unknown error:', error)
+  }
+}
+```
+
+### 错误回调
+
+```typescript
+avatarView.avatarController.onError = (error: Error) => {
+  console.error('AvatarController error:', error)
+  // 处理错误，比如重连、用户提示等
+}
+```
+
+## 🔄 资源管理
+
+### 生命周期管理
+
+```typescript
+// 初始化
+const avatarView = new AvatarView(avatar, container)
+await avatarView.avatarController.start()
+
+// 使用
+avatarView.avatarController.send(audioData, false)
+
+// 清理
+avatarView.dispose() // 自动清理所有资源
+```
+
+### 内存优化
+
+- SDK 自动管理 WASM 内存分配
+- 支持角色和动画资源的动态加载/卸载
+- 提供内存使用监控接口
+
+### 音频数据发送
+
+`send()` 方法接收 `ArrayBuffer` 格式的音频数据：
+
+**使用说明：**
+- `audioData`: 音频数据（ArrayBuffer 格式）
+- `end=false`（默认）- 正常发送音频数据，服务端会积累音频数据，积累到一定量后会自动返回动画数据并开始同步播放动画和音频
+- `end=true` - 立即返回动画数据，不再积累，用于结束当前对话或需要立即响应的场景
+- **重要**：不需要等待 `end=true` 才开始播放，积累到一定音频数据后就会自动开始播放
+
+## 🌐 浏览器兼容性
+
+- **Chrome/Edge** 90+ (推荐 WebGPU)
+- **Firefox** 90+ (WebGL)
+- **Safari** 14+ (WebGL)
+- **移动端** iOS 14+, Android 8+
+
+## 📝 许可证
+
+MIT License
+
+## 🤝 贡献
+
+欢迎提交 Issue 和 Pull Request！
+
+## 📞 支持
+
+如有问题，请联系：
+- 邮箱：support@spavatar.com
+- 文档：https://docs.spavatar.com
+- GitHub：https://github.com/spavatar/sdk

package/dist/StreamingAudioPlayer-C2TfYsO8.js

@@ -0,0 +1,293 @@
+var c = Object.defineProperty;
+var g = (h, t, e) => t in h ? c(h, t, { enumerable: !0, configurable: !0, writable: !0, value: e }) : h[t] = e;
+var s = (h, t, e) => g(h, typeof t != "symbol" ? t + "" : t, e);
+import { l as r } from "./index-DwhR9l52.js";
+class k {
+  constructor(t) {
+    // AudioContext is managed internally
+    s(this, "audioContext", null);
+    s(this, "sampleRate");
+    s(this, "channelCount");
+    s(this, "debug");
+    // Session-level state
+    s(this, "sessionId");
+    s(this, "sessionStartTime", 0);
+    // AudioContext time when session started
+    s(this, "pausedTimeOffset", 0);
+    // Accumulated paused time
+    s(this, "pausedAt", 0);
+    // Time when paused
+    s(this, "scheduledTime", 0);
+    // Next chunk schedule time in AudioContext time
+    // Playback state
+    s(this, "isPlaying", !1);
+    s(this, "isPaused", !1);
+    // Audio buffer queue
+    s(this, "audioChunks", []);
+    s(this, "scheduledChunks", 0);
+    // Number of chunks already scheduled
+    s(this, "activeSources", []);
+    // Event callbacks
+    s(this, "onEndedCallback");
+    this.sessionId = `session_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`, this.sampleRate = (t == null ? void 0 : t.sampleRate) ?? 24e3, this.channelCount = (t == null ? void 0 : t.channelCount) ?? 1, this.debug = (t == null ? void 0 : t.debug) ?? !1;
+  }
+  /**
+   * Initialize audio context (create and ensure it's ready)
+   */
+  async initialize() {
+    this.audioContext || (this.audioContext = new AudioContext({
+      sampleRate: this.sampleRate
+    }), this.audioContext.state === "suspended" && await this.audioContext.resume(), this.log("AudioContext initialized", {
+      sessionId: this.sessionId,
+      sampleRate: this.audioContext.sampleRate,
+      state: this.audioContext.state
+    }));
+  }
+  /**
+   * Add audio chunk (16-bit PCM)
+   */
+  addChunk(t, e = !1) {
+    if (!this.audioContext) {
+      r.error("AudioContext not initialized");
+      return;
+    }
+    this.audioChunks.push({ data: t, isLast: e }), this.log(`Added chunk ${this.audioChunks.length}`, {
+      size: t.length,
+      totalChunks: this.audioChunks.length,
+      isLast: e,
+      isPlaying: this.isPlaying,
+      scheduledChunks: this.scheduledChunks
+    }), !this.isPlaying && this.audioChunks.length > 0 ? (this.log("[StreamingAudioPlayer] Auto-starting playback from addChunk"), this.startPlayback()) : this.isPlaying ? (this.log("[StreamingAudioPlayer] Already playing, scheduling next chunk"), this.scheduleNextChunk()) : this.log("[StreamingAudioPlayer] Not playing and no chunks, waiting for more chunks");
+  }
+  /**
+   * Start new session (stop current and start fresh)
+   */
+  async startNewSession(t) {
+    this.stop(), this.sessionId = `session_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`, this.audioChunks = [], this.scheduledChunks = 0, this.pausedTimeOffset = 0, this.pausedAt = 0, this.log("Starting new session", {
+      chunks: t.length
+    });
+    for (const e of t)
+      this.addChunk(e.data, e.isLast);
+  }
+  /**
+   * Start playback
+   */
+  startPlayback() {
+    if (!this.audioContext) {
+      this.log("[StreamingAudioPlayer] Cannot start playback: AudioContext not initialized");
+      return;
+    }
+    if (this.isPlaying) {
+      this.log("[StreamingAudioPlayer] Cannot start playback: Already playing");
+      return;
+    }
+    this.isPlaying = !0, this.sessionStartTime = this.audioContext.currentTime, this.scheduledTime = this.sessionStartTime, this.log("[StreamingAudioPlayer] Starting playback", {
+      sessionStartTime: this.sessionStartTime,
+      bufferedChunks: this.audioChunks.length,
+      scheduledChunks: this.scheduledChunks,
+      activeSources: this.activeSources.length
+    }), this.scheduleAllChunks();
+  }
+  /**
+   * Schedule all pending chunks
+   */
+  scheduleAllChunks() {
+    for (; this.scheduledChunks < this.audioChunks.length; )
+      this.scheduleNextChunk();
+  }
+  /**
+   * Schedule next audio chunk
+   */
+  scheduleNextChunk() {
+    if (!this.audioContext) {
+      this.log("[StreamingAudioPlayer] Cannot schedule chunk: AudioContext not initialized");
+      return;
+    }
+    if (!this.isPlaying) {
+      this.log("[StreamingAudioPlayer] Cannot schedule chunk: Not playing");
+      return;
+    }
+    const t = this.scheduledChunks;
+    if (t >= this.audioChunks.length) {
+      this.log(`[StreamingAudioPlayer] No more chunks to schedule (chunkIndex: ${t}, totalChunks: ${this.audioChunks.length})`);
+      return;
+    }
+    const e = this.audioChunks[t];
+    if (e.data.length === 0 && !e.isLast) {
+      this.scheduledChunks++;
+      return;
+    }
+    const l = e.data, d = e.isLast, n = this.pcmToAudioBuffer(l);
+    if (!n) {
+      r.error("Failed to create AudioBuffer from PCM data");
+      return;
+    }
+    try {
+      const i = this.audioContext.createBufferSource();
+      i.buffer = n, i.connect(this.audioContext.destination), i.start(this.scheduledTime), this.activeSources.push(i), i.onended = () => {
+        const u = this.activeSources.indexOf(i);
+        u >= 0 && this.activeSources.splice(u, 1), d && this.activeSources.length === 0 && (this.log("Last audio chunk ended, marking playback as ended"), this.markEnded());
+      }, this.scheduledTime += n.duration, this.scheduledChunks++, this.log(`[StreamingAudioPlayer] Scheduled chunk ${t + 1}/${this.audioChunks.length}`, {
+        startTime: this.scheduledTime - n.duration,
+        duration: n.duration,
+        nextScheduleTime: this.scheduledTime,
+        isLast: d,
+        activeSources: this.activeSources.length
+      });
+    } catch (i) {
+      r.errorWithError("Failed to schedule audio chunk:", i);
+    }
+  }
+  /**
+   * Convert PCM data to AudioBuffer
+   * Input: 16-bit PCM (int16), Output: AudioBuffer (float32 [-1, 1])
+   */
+  pcmToAudioBuffer(t) {
+    if (!this.audioContext)
+      return null;
+    if (t.length === 0) {
+      const u = Math.floor(this.sampleRate * 0.01), a = this.audioContext.createBuffer(
+        this.channelCount,
+        u,
+        this.sampleRate
+      );
+      for (let o = 0; o < this.channelCount; o++)
+        a.getChannelData(o).fill(0);
+      return a;
+    }
+    const e = new Uint8Array(t), l = new Int16Array(e.buffer, 0, e.length / 2), d = l.length / this.channelCount, n = this.audioContext.createBuffer(
+      this.channelCount,
+      d,
+      this.sampleRate
+    );
+    for (let i = 0; i < this.channelCount; i++) {
+      const u = n.getChannelData(i);
+      for (let a = 0; a < d; a++) {
+        const o = a * this.channelCount + i;
+        u[a] = l[o] / 32768;
+      }
+    }
+    return n;
+  }
+  /**
+   * Get current playback time (seconds)
+   */
+  getCurrentTime() {
+    if (!this.audioContext || !this.isPlaying)
+      return 0;
+    if (this.isPaused)
+      return this.pausedAt;
+    const e = this.audioContext.currentTime - this.sessionStartTime - this.pausedTimeOffset;
+    return Math.max(0, e);
+  }
+  /**
+   * Pause playback
+   */
+  pause() {
+    !this.isPlaying || this.isPaused || (this.isPaused = !0, this.pausedAt = this.getCurrentTime(), this.log("Playback paused", {
+      pausedAt: this.pausedAt
+    }));
+  }
+  /**
+   * Resume playback
+   */
+  async resume() {
+    if (!this.isPaused || !this.audioContext)
+      return;
+    const t = this.audioContext.currentTime - (this.sessionStartTime + this.pausedAt);
+    this.pausedTimeOffset += t, this.isPaused = !1, this.log("Playback resumed", {
+      pauseDuration: t,
+      totalPausedOffset: this.pausedTimeOffset
+    });
+  }
+  /**
+   * Stop playback
+   */
+  stop() {
+    if (this.audioContext) {
+      this.log("[StreamingAudioPlayer] Stopping playback", {
+        isPlaying: this.isPlaying,
+        audioChunks: this.audioChunks.length,
+        scheduledChunks: this.scheduledChunks,
+        activeSources: this.activeSources.length
+      }), this.isPlaying = !1, this.isPaused = !1, this.sessionStartTime = 0, this.scheduledTime = 0;
+      for (const t of this.activeSources) {
+        t.onended = null;
+        try {
+          t.stop(0);
+        } catch {
+        }
+        try {
+          t.disconnect();
+        } catch {
+        }
+      }
+      this.activeSources = [], this.audioChunks = [], this.scheduledChunks = 0, this.log("[StreamingAudioPlayer] Playback stopped, state reset");
+    }
+  }
+  /**
+   * Mark playback as ended
+   */
+  markEnded() {
+    var t;
+    this.log("Playback ended"), this.isPlaying = !1, (t = this.onEndedCallback) == null || t.call(this);
+  }
+  /**
+   * Set ended callback
+   */
+  onEnded(t) {
+    this.onEndedCallback = t;
+  }
+  /**
+   * Check if playing
+   */
+  isPlayingNow() {
+    return this.isPlaying && !this.isPaused;
+  }
+  /**
+   * Get total duration of buffered audio
+   */
+  getBufferedDuration() {
+    if (!this.audioContext)
+      return 0;
+    let t = 0;
+    for (const e of this.audioChunks)
+      t += e.data.length / 2 / this.channelCount;
+    return t / this.sampleRate;
+  }
+  /**
+   * Get remaining duration (buffered - played) in seconds
+   */
+  getRemainingDuration() {
+    const t = this.getBufferedDuration(), e = this.getCurrentTime();
+    return Math.max(0, t - e);
+  }
+  /**
+   * Dispose and cleanup
+   */
+  dispose() {
+    this.stop(), this.audioContext && (this.audioContext.close(), this.audioContext = null), this.audioChunks = [], this.scheduledChunks = 0, this.sessionStartTime = 0, this.pausedTimeOffset = 0, this.pausedAt = 0, this.scheduledTime = 0, this.onEndedCallback = void 0, this.log("StreamingAudioPlayer disposed");
+  }
+  /**
+   * Flush buffered audio
+   * - hard: stops all playing sources and clears all chunks
+   * - soft (default): clears UNSCHEDULED chunks only
+   */
+  flush(t) {
+    if ((t == null ? void 0 : t.hard) === !0) {
+      this.stop(), this.audioChunks = [], this.scheduledChunks = 0, this.sessionStartTime = 0, this.pausedAt = 0, this.scheduledTime = 0, this.log("Flushed (hard)");
+      return;
+    }
+    this.scheduledChunks < this.audioChunks.length && this.audioChunks.splice(this.scheduledChunks), this.log("Flushed (soft)", { remainingScheduled: this.scheduledChunks });
+  }
+  /**
+   * Debug logging
+   */
+  log(t, e) {
+    this.debug && r.log(`[StreamingAudioPlayer] ${t}`, e || "");
+  }
+}
+export {
+  k as StreamingAudioPlayer
+};
+//# sourceMappingURL=StreamingAudioPlayer-C2TfYsO8.js.map