talking-head-studio 0.4.11 → 0.4.12

package/README.md

@@ -1,6 +1,6 @@
 # talking-head-studio
 
-**Open-source avatar platform for Web, React Native, Unity, and Unreal. Any GLB model. Full lip-sync — with or without blend shapes.**
+**Make any GLB model talk — on the web and on React Native — with phoneme-accurate, audio-aligned lip-sync. With or without blend shapes.**
 
 [![npm version](https://img.shields.io/npm/v/talking-head-studio.svg)](https://www.npmjs.com/package/talking-head-studio)
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
@@ -8,65 +8,123 @@
 
 ---
 
-## What this is
+## The point: lip-sync that's driven by the audio, not guessed
 
-A drop-in avatar runtime and platform SDK built to be a self-hostable replacement for Ready Player Me. The core problem it solves: **any arbitrary 3D model should be able to talk, emote, and respond to a voice pipeline** — regardless of whether the artist baked in blend shapes, visemes, or any face rig at all.
+Most avatar libraries flap a jaw open in proportion to audio loudness. That reads as
+"mouth moving," not "speaking." talking-head-studio is built around a different model: a
+**viseme schedule** — a timed list of mouth shapes derived from the actual synthesized
+speech — drives morph targets on the model.
 
-The library ships a renderer (web iframe + React Native wgpu), a backend-agnostic face control contract, and a growing set of adapters that map TTS/audio/AI output onto whatever rendering mechanism the model actually supports.
+```
+TTS server  ──▶  AgentVisemePayload          ──▶  scheduleVisemes()  ──▶  morph drive
+(word-aligned     { cues: [{ viseme, startMs,      (this library,         (Three.js
+ phonemes)          endMs }], durationMs }          web + native)          morph targets)
+```
 
----
+The wire format is `AgentVisemePayload`: per-phoneme cues using the 9-shape Rhubarb
+vocabulary (`A`–`H`, `X`), each with a start/end time in milliseconds. The library maps
+those onto Oculus viseme morphs and schedules them against the audio clock, so the mouth
+hits each shape *when that sound is actually heard*.
+
+This pairs directly with a TTS server that emits viseme timings from real word alignment
+(we built [Qwen3-TTS](https://github.com/sitebay/Qwen3-TTS) for exactly this — it serves
+`AgentVisemePayload` over an SSE endpoint). But the format is open: emit cues from any
+source and the renderer consumes them identically.
 
-## Lip-sync tiers (any model works)
+### Four lip-sync tiers — every model works
 
-| Model type | Lip-sync method | Quality |
+The model decides the fidelity; you don't have to pre-process anything.
+
+| Your model has… | Method | Quality |
 |---|---|---|
-| GLB with Oculus viseme morphs | Direct morph drive via `MorphTargetBackend` | Excellent |
-| GLB with ARKit blend shapes | `remapArkitToOculus()` → morph drive | Good |
-| GLB with only `jawOpen` / `mouthOpen` | Amplitude fallback | Acceptable |
-| Any other GLB | Gaussian splat backend *(roadmap)* | Excellent |
+| Oculus viseme morphs | Direct morph drive (`MorphTargetBackend`) | Excellent |
+| ARKit blend shapes (52 AUs) | `remapArkitToOculus()` → morph drive | Good |
+| Only `jawOpen` / `mouthOpen` | Amplitude fallback | Acceptable |
+| No face rig at all | Gaussian splat backend *(roadmap — not yet built)* | Excellent |
 
-The last row is the goal: **scan any model into a Gaussian representation, generate per-viseme deltas via FLAME-based transfer, and drive it from the same `FaceControl` contract everything else uses.** No blend shapes required. No artist work required.
+If a model has no viseme morphs, scheduled cues still fall back to the jaw/amplitude path
+automatically — you never get a frozen face.
 
 ---
 
-## Architecture
+## Two renderers, one contract
 
-```
-TTS / audio / face tracking
-        ↓
-  AgentVisemePayload          ← canonical wire format for lip-sync schedules
-        ↓
-  FaceControl                 ← pose (HeadPose) + expression (ExpressionState) + gaze (EyeGaze)
-        ↓
-  AvatarBackend  ←────────────── swap without changing anything upstream
-    ├── MorphTargetBackend    ← Three.js morph targets (GLB with blend shapes)
-    ├── GaussianBackend       ← [roadmap] Gaussian splat + FLAME delta transfer
-    └── (your backend)        ← implement AvatarBackend, plug in
-        ↓
-  Renderer
-    ├── Web iframe            ← TalkingHead.web.tsx (any React app)
-    ├── React Native wgpu     ← WgpuAvatar (native GPU, no WebView latency)
-    └── Unity / Unreal        ← [roadmap] SDK plugins consuming same contracts
-```
+The same `AgentVisemePayload` / `FaceControl` contract drives both render paths, so you
+write your voice pipeline once:
+
+- **Web** — an isolated `<iframe>` running [met4citizen TalkingHead](https://github.com/met4citizen/TalkingHead)
+  as the rig (`TalkingHead.web.tsx`). Drop it into any React / Next / Vite app.
+- **React Native** — a native WebGPU renderer (`WgpuAvatar`, via `react-native-wgpu` +
+  react-three-fiber). No WebView, no postMessage latency, morphs driven on the GPU.
 
-Everything above `AvatarBackend` is renderer-agnostic. Everything above `FaceControl` is model-agnostic.
+Capabilities differ slightly between the two — see the [capability matrix](#runtime-capability-matrix).
 
 ---
 
-## Installation
+## Install
 
 ```bash
-# React Native / Expo
+# React Native / Expo WebView path
 npm install talking-head-studio react-native-webview
 
+# React Native / Expo native WebGPU path
+npx expo install react-native-wgpu @react-three/fiber three three-stdlib expo-asset
+
 # Web (React, Next.js, Vite)
 npm install talking-head-studio
 ```
 
+`three`, `@react-three/fiber`, and the platform packages are peer dependencies — bring your
+own versions. `react-native-webview` is only required for the WebView renderer. Native
+WebGPU uses `react-native-wgpu` and must run in a native build, not Expo Go.
+
+### React Native / Expo WebGPU setup
+
+Native WebGPU needs the React Native new architecture and the WebGPU build of Three.js.
+The example app in `example/` has the full working config; these are the important parts:
+
+```jsonc
+// app.json
+{
+  "expo": {
+    "newArchEnabled": true,
+    "plugins": ["expo-asset"]
+  }
+}
+```
+
+```js
+// metro.config.js
+const path = require('path');
+const { getDefaultConfig } = require('expo/metro-config');
+
+const config = getDefaultConfig(__dirname);
+const nodeModules = path.resolve(__dirname, 'node_modules');
+const threeWebgpu = path.resolve(nodeModules, 'three/build/three.webgpu.js');
+
+config.resolver.assetExts.push('glb');
+config.resolver.extraNodeModules = {
+  three: threeWebgpu,
+};
+
+module.exports = config;
+```
+
+Build and launch a native app so `WebGPUModule` is linked:
+
+```bash
+npx expo prebuild --platform android --no-install
+npx expo run:android
+```
+
+Expo Go cannot load the native WebGPU module.
+
 ---
 
 ## Quick start
 
+### Web / React Native component
+
 ```tsx
 import { useRef } from 'react';
 import { TalkingHead, type TalkingHeadRef } from 'talking-head-studio';
@@ -80,99 +138,180 @@ export default function Avatar() {
       avatarUrl="https://example.com/your-model.glb"
       mood="happy"
       cameraView="upper"
-      hairColor="#1a1a2e"
-      skinColor="#e0a370"
-      accessories={[{
-        id: 'sunglasses',
-        url: 'https://example.com/sunglasses.glb',
-        bone: 'Head',
-        position: [0, 0.08, 0.12],
-        rotation: [0, 0, 0],
-        scale: 1.0,
-      }]}
       style={{ width: 400, height: 600 }}
-      onReady={() => console.log('ready')}
+      onReady={() => {
+        // Drive the mouth from a viseme schedule (e.g. from your TTS server)
+        ref.current?.scheduleVisemes({
+          cues: [
+            { viseme: 'A', startMs: 0, endMs: 90 },
+            { viseme: 'E', startMs: 90, endMs: 170 },
+            { viseme: 'X', startMs: 170, endMs: 220 },
+          ],
+          durationMs: 220,
+          audioStartedAtMs: Date.now(),
+        });
+      }}
     />
   );
 }
 ```
 
+### Native WebGPU (React Native, no WebView)
+
+```tsx
+import { WgpuAvatar, type WgpuAvatarRef } from 'talking-head-studio/wgpu';
+
+const ref = useRef<WgpuAvatarRef>(null);
+
+<WgpuAvatar
+  ref={ref}
+  avatarUrl="https://example.com/your-model.glb"
+  mood="neutral"
+  style={{ flex: 1 }}
+/>;
+// ref.current?.scheduleVisemes(payload) — same contract as the web component
+```
+
 ---
 
-## FaceControl — the core contract
+## TalkingHead component — props & ref
 
-The `FaceControl` type is the single value that flows between your voice pipeline and any avatar backend. If you're building a custom backend or integrating with a game engine, this is what you implement against.
+### Props
+
+| Prop | Type | Default | Description |
+|------|------|---------|-------------|
+| `avatarUrl` | `string` | required | Any `.glb`. Rigged or not. |
+| `authToken` | `string \| null` | `null` | Bearer token for authenticated GLB URLs. |
+| `mood` | `TalkingHeadMood` | `'neutral'` | `neutral \| happy \| sad \| angry \| fear \| disgust \| love \| sleep \| excited \| thinking \| concerned \| surprised` |
+| `cameraView` | `'head' \| 'upper' \| 'full'` | `'upper'` | Framing preset. |
+| `cameraDistance` | `number` | `-0.5` | Zoom offset. Negative = closer. |
+| `hairColor` | `string` | — | Hex color. Applied to materials named `hair`, `fur`. |
+| `skinColor` | `string` | — | Applied to `skin`, `body`, `face`. |
+| `eyeColor` | `string` | — | Applied to `eye`, `iris`. |
+| `accessories` | `TalkingHeadAccessory[]` | `[]` | Bone-attached GLB items. |
+| `onReady` | `() => void` | — | Fired when fully loaded. |
+| `onError` | `(msg: string) => void` | — | Fired on load failure. |
+| `style` | `ViewStyle / CSSProperties` | — | Container style. |
+
+### Ref methods
 
 ```ts
-import type { FaceControl, ExpressionState, HeadPose, EyeGaze } from 'talking-head-studio';
+// Lip-sync
+ref.current?.scheduleVisemes(payload); // AgentVisemePayload → full timed lip-sync schedule
+ref.current?.clearVisemes();
+ref.current?.sendAmplitude(0.7);       // amplitude 0..1 → jaw (fallback / no schedule)
 
-type HeadPose = {
-  yaw:   number; // -1..1, left..right
-  pitch: number; // -1..1, down..up
-  roll:  number; // -1..1, tilt
-};
+// Expression & appearance
+ref.current?.setMood('excited');
+ref.current?.setHairColor('#ff0000');
+ref.current?.setSkinColor('#8d5524');
+ref.current?.setEyeColor('#2e86de');
+ref.current?.setAccessories([...]);
 
-type EyeGaze = {
-  x: number; // -1..1, left..right
-  y: number; // -1..1, down..up
-};
+// Body — procedural motions, gestures, poses, animation clips
+ref.current?.dispatchMotion('groove');                       // looping procedural motion
+ref.current?.stopMotion();
+ref.current?.playGesture('thumbup');                         // upstream hand gesture
+ref.current?.playPose('oneknee');                            // upstream pose template
+ref.current?.playAnimation('/animations/wave.glb', { dur: 2 });
+ref.current?.lookAt(120, 80, 500);                           // turn toward viewport coords
+```
+
+The motion vocabulary (`groove`, `wave`, `nod`, `idle`, `attack`, `defend`, `celebrate`,
+plus every upstream gesture/pose name) is exported as typed constants —
+`MOTION_KEYS`, `TALKINGHEAD_GESTURES`, `TALKINGHEAD_POSES`, and the `isMotionKey()` guard —
+from both the package root and `talking-head-studio/contract`.
+
+### Runtime capability matrix
+
+Both renderers share one API; where native can't match the WebView's upstream rig, it
+falls back to a procedural approximation rather than failing. This table is the honest gap
+list.
+
+| Feature | Web (iframe) | Native (WGPU) | Notes |
+|---|:---:|:---:|---|
+| Viseme schedules (`scheduleVisemes`) | ✅ | ✅ | Both consume `AgentVisemePayload`. |
+| Amplitude jaw fallback (`sendAmplitude`) | ✅ | ⚠️ | Web drives jaw from amplitude; native exposes the method for API parity. |
+| Core procedural motions (`groove`, `attack`, `defend`) | ✅ | ✅ | Shared `MOTION_DEFS` source of truth. |
+| Gesture names (`thumbup`, `shrug`, …) | ✅ | ⚠️ | Web delegates to TalkingHead; native uses procedural approximations. |
+| Pose names (`oneknee`, `kneel`, `sitting`, …) | ✅ | ⚠️ | Web delegates to TalkingHead; native uses static procedural poses. |
+| Full mood vocabulary | ✅ | ✅ | All 8 upstream moods + friendly aliases. |
+| External animation clips (`playAnimation`) | ✅ | ⚠️ | Web delegates to TalkingHead; native plays GLB clips via `AnimationMixer`. |
+| Gaze (`lookAt`) | ✅ | ❌ | Native eye/head-gaze bridge is future work. |
+| Listening / mic-reactive mouth | ⚠️ | ❌ | Web can route host-provided audio; native bridge not implemented. |
+
+---
+
+## Self-hosting the runtime assets
+
+By default the web iframe pulls the TalkingHead rig, three.js, and the HeadAudio model
+from public CDNs (jsDelivr, gstatic). To run fully self-hosted — no external CDN — vendor
+those files and point the renderer at your own origin:
+
+```ts
+import { buildAvatarHtml } from 'talking-head-studio/html';
+
+const html = buildAvatarHtml({
+  avatarUrl: 'https://your-cdn/model.glb',
+  vendorBaseUrl: 'https://your-cdn/vendor', // serves three.module.js, talkinghead.mjs, etc.
+  // ...
+});
+```
+
+`vendorBaseUrl` replaces every CDN reference; `dracoDecoderUrl` overrides the DRACO decoder
+location independently.
+
+---
+
+## FaceControl — the lower-level contract
+
+If you're writing a custom backend or a game-engine integration, `FaceControl` is the
+single value that flows between a voice pipeline and any avatar backend.
+
+```ts
+import type { FaceControl, ExpressionState, HeadPose, EyeGaze } from 'talking-head-studio';
+
+type HeadPose = { yaw: number; pitch: number; roll: number };  // each -1..1
+type EyeGaze = { x: number; y: number };                       // each -1..1
 
 type ExpressionState = {
-  jawOpen:         number; // 0..1
-  mouthSmile:      number;
-  mouthFunnel:     number;
-  mouthPucker:     number;
-  mouthWide:       number;
-  upperLipRaise:   number;
-  lowerLipDepress: number;
-  cheekRaise:      number;
-  blinkLeft:       number;
-  blinkRight:      number;
-  browInnerUp:     number;
-  browDownLeft:    number;
-  browDownRight:   number;
-  eyeGazeLeft:     EyeGaze;
-  eyeGazeRight:    EyeGaze;
-};
+  jawOpen: number; mouthSmile: number; mouthFunnel: number; mouthPucker: number;
+  mouthWide: number; upperLipRaise: number; lowerLipDepress: number; cheekRaise: number;
+  blinkLeft: number; blinkRight: number; browInnerUp: number;
+  browDownLeft: number; browDownRight: number;
+  eyeGazeLeft: EyeGaze; eyeGazeRight: EyeGaze;
+}; // all weights 0..1 unless noted
 ```
 
-### Driving FaceControl from a viseme schedule
+Drive it from a viseme schedule:
 
 ```ts
 import { useFaceControlsFromVisemes } from 'talking-head-studio';
 
-// schedule: AgentVisemePayload from your TTS backend
-const faceControl = useFaceControlsFromVisemes(schedule);
-// → { pose: { yaw:0, pitch:0, roll:0 }, expr: { jawOpen: 0.7, ... } }
+const faceControl = useFaceControlsFromVisemes(schedule); // rAF-sampled FaceControl
 ```
 
-### Implementing a custom backend
+Or implement a backend against it:
 
 ```ts
 import type { AvatarBackend, AvatarRenderTarget, FaceControl } from 'talking-head-studio';
 
-class MyGaussianBackend implements AvatarBackend {
-  initialize() { /* load splat data, FLAME weights */ }
-  attach(target: AvatarRenderTarget) { /* bind to canvas/surface */ }
-  setControl(control: FaceControl) { /* map ExpressionState → splat coefficients */ }
-  renderFrame() { /* rasterize */ }
-  dispose() { /* cleanup */ }
+class MyBackend implements AvatarBackend {
+  initialize() {}
+  attach(target: AvatarRenderTarget) {}
+  setControl(control: FaceControl) {}
+  renderFrame() {}
+  dispose() {}
 }
 ```
 
----
-
-## MorphTargetBackend — Three.js GLB adapter
+### MorphTargetBackend — the built-in Three.js adapter
 
-The first concrete `AvatarBackend` implementation. Give it any loaded Three.js scene and it will find morph targets, build a lookup cache, and drive them from `FaceControl`.
+The concrete `AvatarBackend` for GLB-with-morphs. Hand it a loaded scene; it discovers
+morph targets, builds a lookup cache, and drives them from `FaceControl`.
 
 ```ts
-import * as THREE from 'three';
-import { GLTFLoader } from 'three/examples/jsm/loaders/GLTFLoader';
-import { MorphTargetBackend } from 'talking-head-studio';
-
-const loader = new GLTFLoader();
-const gltf = await loader.loadAsync('/avatar.glb');
+import { MorphTargetBackend, createNeutralExpression } from 'talking-head-studio';
 
 const backend = new MorphTargetBackend(gltf.scene, {
   mood: 'neutral',
@@ -184,79 +323,27 @@ const backend = new MorphTargetBackend(gltf.scene, {
   },
 });
 
-// Each frame:
 backend.setControl(faceControl);
 backend.renderFrame();
-
-// Debug: what morphs does this model actually have?
-console.log(backend.availableChannels);
-// → { visemes: ['aa','PP','oh',...], expressions: ['jawOpen','blinkLeft',...], gaze: ['lookLeft','lookUp'] }
+console.log(backend.availableChannels); // what this model actually supports
 ```
 
----
-
-## ARKit → Oculus remap
-
-Models with ARKit blend shapes (52 facial action units) but no Oculus viseme morphs can be remapped analytically — no ML, no FLAME, no artist work.
+### ARKit → Oculus remap (no ML, no artist work)
 
 ```ts
 import { remapArkitToOculus, getArkitWeightsForViseme } from 'talking-head-studio';
 
-// Runtime: face tracking data → Oculus viseme weights
-const oculusWeights = remapArkitToOculus({
-  jawOpen: 0.7,
-  mouthLowerDownLeft: 0.4,
-  mouthLowerDownRight: 0.4,
-});
-// → { aa: 0.68, PP: 0.03, oh: 0.12, ... }
-
-// Bake-time: get the ARKit recipe for a specific viseme
-const recipe = getArkitWeightsForViseme('ou');
-// → { mouthPucker: 0.9, mouthRollLower: 0.3 }
+remapArkitToOculus({ jawOpen: 0.7, mouthLowerDownLeft: 0.4 }); // → { aa: 0.68, oh: 0.12, ... }
+getArkitWeightsForViseme('ou');                                // → { mouthPucker: 0.9, ... }
 ```
 
-The full `ARKIT_TO_OCULUS` coefficient table is exported so you can build your own bake pipeline.
-
----
-
-## TalkingHead component — props & ref
-
-### Props
-
-| Prop | Type | Default | Description |
-|------|------|---------|-------------|
-| `avatarUrl` | `string` | required | Any `.glb`. Rigged or not. |
-| `authToken` | `string \| null` | `null` | Bearer token for authenticated GLB URLs. |
-| `mood` | `TalkingHeadMood` | `'neutral'` | `neutral \| happy \| sad \| angry \| excited \| thinking \| concerned \| surprised` |
-| `cameraView` | `'head' \| 'upper' \| 'full'` | `'upper'` | Framing preset. |
-| `cameraDistance` | `number` | `-0.5` | Zoom offset. Negative = closer. |
-| `hairColor` | `string` | — | Hex color. Applied to materials named `hair`, `fur`. |
-| `skinColor` | `string` | — | Applied to `skin`, `body`, `face`. |
-| `eyeColor` | `string` | — | Applied to `eye`, `iris`. |
-| `accessories` | `TalkingHeadAccessory[]` | `[]` | Bone-attached GLB items. |
-| `onReady` | `() => void` | — | Fired when fully loaded. |
-| `onError` | `(msg: string) => void` | — | Fired on load failure. |
-| `style` | `ViewStyle / CSSProperties` | — | Container style. |
-
-### Ref methods
-
-```ts
-ref.current?.sendAmplitude(0.7);       // amplitude 0..1 → jaw
-ref.current?.scheduleVisemes(payload); // AgentVisemePayload → full lip-sync schedule
-ref.current?.clearVisemes();
-ref.current?.setMood('excited');
-ref.current?.setHairColor('#ff0000');
-ref.current?.setSkinColor('#8d5524');
-ref.current?.setEyeColor('#2e86de');
-ref.current?.setAccessories([...]);
-ref.current?.dispatchMotion('nod');
-```
+The full `ARKIT_TO_OCULUS` coefficient table is exported for building your own bake pipeline.
 
 ---
 
 ## Accessories
 
-Any GLB attached to any skeleton bone. Placement is editable at runtime via the 3D editor.
+Any GLB attached to any skeleton bone, placeable at runtime.
 
 ```ts
 interface TalkingHeadAccessory {
@@ -269,59 +356,56 @@ interface TalkingHeadAccessory {
 }
 ```
 
-Common Mixamo bones: `Head, Neck, Spine, Spine1, Spine2, LeftHand, RightHand, LeftFoot, RightFoot, Hips`
-
-The 3D editor (`talking-head-studio/editor`) provides a gizmo for live placement with front/top/side views. LLM-assisted placement is available via the companion backend.
+Common Mixamo bones: `Head, Neck, Spine, Spine1, Spine2, LeftHand, RightHand, LeftFoot, RightFoot, Hips`.
+The 3D editor (`talking-head-studio/editor`, web only) provides a gizmo for live placement.
 
 ---
 
-## Packages
+## Subpath exports
 
-| Path | Description |
+| Import | Description |
 |------|-------------|
-| `talking-head-studio` | Live avatar renderer + FaceControl contracts |
-| `talking-head-studio/editor` | R3F-based 3D editor with gizmo (web only) |
+| `talking-head-studio` | Avatar component + `FaceControl` contracts + motion constants |
+| `talking-head-studio/contract` | Stable type-only entrypoint — visemes, FaceControl, backends, motion |
+| `talking-head-studio/html` | `buildAvatarHtml()` for self-hosted / custom iframe embedding |
+| `talking-head-studio/wgpu` | React Native WebGPU renderer (`WgpuAvatar`) |
+| `talking-head-studio/editor` | R3F 3D editor with placement gizmo (web only) |
 | `talking-head-studio/appearance` | Material color system for any GLB |
 | `talking-head-studio/voice` | Audio recording + WAV conversion hooks |
 | `talking-head-studio/sketchfab` | Sketchfab search + download hooks |
 | `talking-head-studio/api` | Studio API client (avatar CRUD, voice profiles) |
 | `talking-head-studio/wardrobe` | Accessory + outfit state management |
-| `talking-head-studio/wgpu` | React Native wgpu renderer |
-| `packages/avatar-creator` | Embeddable avatar creator widget |
-| `packages/agent-avatar` | LiveKit agent + MCP integration |
+
+Workspace packages (`packages/avatar-creator`, `packages/agent-avatar`) ship an embeddable
+creator widget and a LiveKit + MCP agent integration.
 
 ---
 
 ## Roadmap
 
-### Now — shipped
-- `FaceControl` canonical face control space (pose + expression + gaze)
-- `AvatarBackend` interface — swap renderers without changing upstream code
-- `MorphTargetBackend` — Three.js GLB adapter with morph target discovery and mood layering
-- ARKit → Oculus analytical remap (`remapArkitToOculus`, full coefficient table)
-- `useFaceControlsFromVisemes` — rAF-sampled hook from `AgentVisemePayload`
-- `AgentVisemePayload` canonical TTS → lip-sync wire format
-- `AvatarGlbParams` — typed API contract for quality/compression/morph group selection
-- `CalibrationProfile` — per-avatar range remapping and gaze limits
-- Platform type stubs: SDK (web/Unity/Unreal), marketplace catalog, avatar GLB API
-- `packages/avatar-creator` — embeddable creator widget with preset catalog
-- `packages/agent-avatar` — LiveKit agent + MCP tool integration
-
-### Next
-- **GLB schema walker** — scan any loaded GLB and report: morph target coverage, skeleton bones, LODs, viseme tier. Prerequisite for the validator and import pipeline.
-- **`GET /avatars/{id}.glb` with `AvatarGlbParams`** — extend the companion backend to serve quality/compression/morph-group variants on the existing endpoint.
-- **Creator postMessage bridge** — let partners embed the avatar creator in an iframe and receive avatar IDs back, like RPM's WebView creator.
-
-### Medium term
-- **`GaussianBackend`** — Gaussian splat renderer implementing `AvatarBackend`. Takes any model, scans it, drives expression via FLAME-based per-viseme delta transfer. No artist work, no blend shapes required. This is the zero-prerequisite lip-sync path.
-- **FLAME viseme transfer pipeline** (Python, companion backend) — fit FLAME to a face screenshot, generate Oculus viseme deltas, bake back into the GLB as morph targets. Background task on upload for any avatar missing viseme morphs.
-- **Unity SDK** — C# plugin implementing the `AvatarBackend` contract. Blueprint-friendly API for loading GLBs, driving morphs, consuming `AgentVisemePayload`.
-- **Unreal plugin** — UE5 plugin with Blueprint-accessible `UAvatarDescriptor` and a sample Quickstart map.
-
-### Longer term
-- Avatar marketplace — `CatalogItem`, `AvatarAsset`, `RarityLevel` types are already defined. Backend + web store + in-creator purchasing.
-- RPM migration tools — import existing RPM avatars where technically possible.
-- SLA + deprecation policy — for teams that need a reliability guarantee as they move off RPM.
+> **Status legend:** ✅ shipped · 🔜 in progress · 🧪 designed, not yet built
+
+**Shipped today**
+- ✅ `FaceControl` face-control space (pose + expression + gaze) and `AvatarBackend` interface
+- ✅ `MorphTargetBackend` — GLB morph discovery + mood layering
+- ✅ ARKit → Oculus analytical remap with full coefficient table
+- ✅ `AgentVisemePayload` viseme schedule format + `scheduleVisemes` on both renderers
+- ✅ Shared procedural motion engine (web + native WGPU), gestures, poses, animation clips
+- ✅ Self-hosting via `buildAvatarHtml({ vendorBaseUrl })`
+- ✅ `packages/avatar-creator`, `packages/agent-avatar`
+
+**In progress**
+- 🔜 Native (WGPU) gaze bridge (`lookAt`) and mic-reactive listening
+- 🔜 GLB schema walker — report morph coverage, bones, LODs, viseme tier for any model
+
+**Designed, not yet built**
+- 🧪 `GaussianBackend` — Gaussian-splat renderer + FLAME per-viseme delta transfer, so a
+  model with *no* face rig still gets excellent lip-sync. This is the zero-prerequisite path.
+- 🧪 FLAME viseme transfer pipeline (companion backend) — bake Oculus visemes into a GLB
+  that lacks them
+- 🧪 Unity / Unreal SDKs implementing the same `AvatarBackend` contract
+- 🧪 Avatar marketplace + RPM import tooling (`CatalogItem` / `AvatarAsset` types exist;
+  backend and store do not)
 
 ---
 
@@ -331,14 +415,16 @@ The 3D editor (`talking-head-studio/editor`) provides a gizmo for live placement
 git clone https://github.com/sitebay/talking-head-studio.git
 cd talking-head-studio
 npm install
-npm run typecheck   # must be clean (excluding known expo-audio peer dep warnings)
+npm run typecheck   # must be clean
 npm test
 ```
 
-The repo is a monorepo with `packages/*` as npm workspaces. The main library is the root package.
+Monorepo with `packages/*` as npm workspaces; the main library is the root package. The
+publish gate (`prepublishOnly`) runs lint, typecheck, tests, and metadata checks.
 
 ---
 
-## License
+## Credits & license
 
-MIT
+Built on [met4citizen/TalkingHead](https://github.com/met4citizen/TalkingHead) (rig +
+gestures/poses on the web path) and [Three.js](https://threejs.org). MIT licensed.

package/dist/TalkingHead.d.ts

@@ -1,7 +1,8 @@
 import React from 'react';
 import { type StyleProp, type ViewStyle } from 'react-native';
-export type TalkingHeadMood = 'neutral' | 'happy' | 'sad' | 'angry' | 'excited' | 'thinking' | 'concerned' | 'surprised';
+export type TalkingHeadMood = 'neutral' | 'happy' | 'sad' | 'angry' | 'fear' | 'disgust' | 'love' | 'sleep' | 'excited' | 'thinking' | 'concerned' | 'surprised';
 import type { AgentVisemePayload, OculusViseme, VisemeCue } from './core/avatar/visemes';
+import type { MotionKey, TalkingHeadGesture, TalkingHeadPose } from './core/avatar/motion';
 export type TalkingHeadLoadingStage = 'booting' | 'fetching_model' | 'loading_avatar' | 'loading_fallback' | 'ready';
 export interface TalkingHeadLoadingState {
     stage: TalkingHeadLoadingStage;
@@ -88,8 +89,32 @@ export interface TalkingHeadRef {
     setSkinColor: (color: string) => void;
     setEyeColor: (color: string) => void;
     setAccessories: (accessories: TalkingHeadAccessory[]) => void;
-    /** Dispatch a named motion/gesture to the avatar (e.g. 'wave_right', 'dance_idle'). */
-    dispatchMotion: (name: string) => void;
+    /** Play a procedural motion (e.g. 'attack', 'defend', 'groove'). */
+    dispatchMotion(name: MotionKey): void;
+    dispatchMotion(name: string): void;
+    /** Stop the current procedural motion and return to rest. */
+    stopMotion: () => void;
+    /** Play an upstream TalkingHead hand gesture (e.g. 'thumbup'). */
+    playGesture: (name: TalkingHeadGesture | string, opts?: {
+        dur?: number;
+        mirror?: boolean;
+        ms?: number;
+    }) => void;
+    /** Stop the current gesture, easing out over `ms`. */
+    stopGesture: (ms?: number) => void;
+    /** Strike a pose — a built-in template name (e.g. 'oneknee') or a pose-file URL. */
+    playPose: (urlOrTemplate: TalkingHeadPose | string, dur?: number) => void;
+    /** Release the current pose and return to the default stance. */
+    stopPose: () => void;
+    /** Play a full body animation from a GLB/FBX URL (e.g. a combat move). */
+    playAnimation: (url: string, opts?: {
+        dur?: number;
+        index?: number;
+    }) => void;
+    /** Stop the current body animation. */
+    stopAnimation: () => void;
+    /** Turn head/eyes toward viewport coordinates (px), easing over `ms`. */
+    lookAt: (x: number, y: number, ms?: number) => void;
 }
 /** @deprecated Use AvatarPlayerRef */
 export type TalkingHeadRefAlias = TalkingHeadRef;