talking-head-studio 0.4.10 → 0.4.12

package/README.md

@@ -1,6 +1,6 @@
 # talking-head-studio
 
-**The missing UI layer for AI Agents. Drop-in, lip-syncing 3D avatars for Web, React, and React Native.**
+**Make any GLB model talk — on the web and on React Native — with phoneme-accurate, audio-aligned lip-sync. With or without blend shapes.**
 
 [![npm version](https://img.shields.io/npm/v/talking-head-studio.svg)](https://www.npmjs.com/package/talking-head-studio)
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
@@ -8,56 +8,122 @@
 
 ---
 
-## Why this?
+## The point: lip-sync that's driven by the audio, not guessed
 
-- **Zero-Jank React Native & Web:** True cross-platform rendering. React Native gets a blazing fast wgpu-accelerated native render loop, skipping WebView bridge latency entirely. React on web gets a robust `react-three-fiber` setup. Same API, same props.
-- **Universal GLB Compatibility:** Bring any GLB. Out-of-the-box support for standard ARKit blendshapes. Rigged models get full phoneme-based lip-sync. Non-rigged models get an amplitude-driven jaw animation fallback.
-- **Built for AI & Voice Pipelines:** Wire `sendAmplitude` or visemes directly to LiveKit, Web Audio, ElevenLabs, OpenAI Realtime, or any audio source.
-- **Always Alive:** Procedural idle animations (breathing, nodding, swaying) keep your avatar from feeling like a static doll.
-- **Dynamic Wardrobe & Accessories:** Swap hair, skin, and eye colors on the fly. Attach hats, glasses, or backpacks to any bone at runtime.
+Most avatar libraries flap a jaw open in proportion to audio loudness. That reads as
+"mouth moving," not "speaking." talking-head-studio is built around a different model: a
+**viseme schedule** — a timed list of mouth shapes derived from the actual synthesized
+speech — drives morph targets on the model.
 
----
+```
+TTS server  ──▶  AgentVisemePayload          ──▶  scheduleVisemes()  ──▶  morph drive
+(word-aligned     { cues: [{ viseme, startMs,      (this library,         (Three.js
+ phonemes)          endMs }], durationMs }          web + native)          morph targets)
+```
+
+The wire format is `AgentVisemePayload`: per-phoneme cues using the 9-shape Rhubarb
+vocabulary (`A`–`H`, `X`), each with a start/end time in milliseconds. The library maps
+those onto Oculus viseme morphs and schedules them against the audio clock, so the mouth
+hits each shape *when that sound is actually heard*.
+
+This pairs directly with a TTS server that emits viseme timings from real word alignment
+(we built [Qwen3-TTS](https://github.com/sitebay/Qwen3-TTS) for exactly this — it serves
+`AgentVisemePayload` over an SSE endpoint). But the format is open: emit cues from any
+source and the renderer consumes them identically.
+
+### Four lip-sync tiers — every model works
+
+The model decides the fidelity; you don't have to pre-process anything.
+
+| Your model has… | Method | Quality |
+|---|---|---|
+| Oculus viseme morphs | Direct morph drive (`MorphTargetBackend`) | Excellent |
+| ARKit blend shapes (52 AUs) | `remapArkitToOculus()` → morph drive | Good |
+| Only `jawOpen` / `mouthOpen` | Amplitude fallback | Acceptable |
+| No face rig at all | Gaussian splat backend *(roadmap — not yet built)* | Excellent |
 
-## Table of Contents
-
-- [Installation](#installation)
-- [Quick Start](#quick-start)
-- [Subpath Exports](#subpath-exports)
-- [Props](#props)
-- [Ref API](#ref-api)
-- [Accessories](#accessories)
-- [Color Customization](#color-customization)
-- [Voice Pipeline Integration](#voice-pipeline-integration)
-- [GLB Compatibility](#glb-compatibility)
-- [Plain React / Next.js](#plain-react--nextjs)
-- [MotionEngine (Upcoming)](#motionengine-upcoming)
-- [Contributing](#contributing)
-- [Credits](#credits)
-- [License](#license)
+If a model has no viseme morphs, scheduled cues still fall back to the jaw/amplitude path
+automatically — you never get a frozen face.
 
 ---
 
-## Installation
+## Two renderers, one contract
 
-### React Native / Expo
+The same `AgentVisemePayload` / `FaceControl` contract drives both render paths, so you
+write your voice pipeline once:
+
+- **Web** — an isolated `<iframe>` running [met4citizen TalkingHead](https://github.com/met4citizen/TalkingHead)
+  as the rig (`TalkingHead.web.tsx`). Drop it into any React / Next / Vite app.
+- **React Native** — a native WebGPU renderer (`WgpuAvatar`, via `react-native-wgpu` +
+  react-three-fiber). No WebView, no postMessage latency, morphs driven on the GPU.
+
+Capabilities differ slightly between the two — see the [capability matrix](#runtime-capability-matrix).
+
+---
+
+## Install
 
 ```bash
+# React Native / Expo WebView path
 npm install talking-head-studio react-native-webview
+
+# React Native / Expo native WebGPU path
+npx expo install react-native-wgpu @react-three/fiber three three-stdlib expo-asset
+
+# Web (React, Next.js, Vite)
+npm install talking-head-studio
 ```
 
-`react-native-webview` is a peer dependency. If you are using Expo, it is available as a built-in package.
+`three`, `@react-three/fiber`, and the platform packages are peer dependencies — bring your
+own versions. `react-native-webview` is only required for the WebView renderer. Native
+WebGPU uses `react-native-wgpu` and must run in a native build, not Expo Go.
 
-### Web only (React, Next.js, Vite)
+### React Native / Expo WebGPU setup
+
+Native WebGPU needs the React Native new architecture and the WebGPU build of Three.js.
+The example app in `example/` has the full working config; these are the important parts:
+
+```jsonc
+// app.json
+{
+  "expo": {
+    "newArchEnabled": true,
+    "plugins": ["expo-asset"]
+  }
+}
+```
+
+```js
+// metro.config.js
+const path = require('path');
+const { getDefaultConfig } = require('expo/metro-config');
+
+const config = getDefaultConfig(__dirname);
+const nodeModules = path.resolve(__dirname, 'node_modules');
+const threeWebgpu = path.resolve(nodeModules, 'three/build/three.webgpu.js');
+
+config.resolver.assetExts.push('glb');
+config.resolver.extraNodeModules = {
+  three: threeWebgpu,
+};
+
+module.exports = config;
+```
+
+Build and launch a native app so `WebGPUModule` is linked:
 
 ```bash
-npm install talking-head-studio
+npx expo prebuild --platform android --no-install
+npx expo run:android
 ```
 
-No `react-native` or WebView runtime dependency needed. The package ships a web entry point that renders via `<iframe srcdoc>` automatically when bundled for browser targets.
+Expo Go cannot load the native WebGPU module.
 
 ---
 
-## Quick Start
+## Quick start
+
+### Web / React Native component
 
 ```tsx
 import { useRef } from 'react';
@@ -72,397 +138,293 @@ export default function Avatar() {
       avatarUrl="https://example.com/your-model.glb"
       mood="happy"
       cameraView="upper"
-      hairColor="#1a1a2e"
-      skinColor="#e0a370"
-      accessories={[
-        {
-          id: 'sunglasses',
-          url: 'https://example.com/sunglasses.glb',
-          bone: 'Head',
-          position: [0, 0.08, 0.12],
-          rotation: [0, 0, 0],
-          scale: 1.0,
-        },
-      ]}
       style={{ width: 400, height: 600 }}
-      onReady={() => console.log('Avatar loaded')}
-      onError={(msg) => console.error('Load failed:', msg)}
+      onReady={() => {
+        // Drive the mouth from a viseme schedule (e.g. from your TTS server)
+        ref.current?.scheduleVisemes({
+          cues: [
+            { viseme: 'A', startMs: 0, endMs: 90 },
+            { viseme: 'E', startMs: 90, endMs: 170 },
+            { viseme: 'X', startMs: 170, endMs: 220 },
+          ],
+          durationMs: 220,
+          audioStartedAtMs: Date.now(),
+        });
+      }}
     />
   );
 }
 ```
 
----
-
-## Subpath Exports
-
-The package ships five independent entry points. Import only what you need — each subpath has its own optional peer dependencies.
-
-### `talking-head-studio` — Live talking avatar
-```tsx
-import { TalkingHead } from 'talking-head-studio';
-// Peer deps: react
-// Native-only peers: react-native (optional), react-native-webview (optional)
-```
-
-### `talking-head-studio/editor` — 3D editor with gizmo (web)
-R3F-based canvas with PivotControls gizmo for placing accessories on an avatar. Web only.
-```tsx
-import { AvatarCanvas } from 'talking-head-studio/editor';
-// Peer deps: @react-three/fiber, @react-three/drei, three
-```
+### Native WebGPU (React Native, no WebView)
 
-### `talking-head-studio/appearance` — Material color system
-Apply skin/hair/eye colors to any GLB avatar. Works in both the live view and the 3D editor.
 ```tsx
-import { applyAppearanceToObject3D, type AvatarAppearance } from 'talking-head-studio/appearance';
-// No extra peer deps
-```
+import { WgpuAvatar, type WgpuAvatarRef } from 'talking-head-studio/wgpu';
 
-### `talking-head-studio/voice` — Audio recording hooks
-Headless hooks for recording voice samples (WebM→WAV conversion included). Backend-agnostic — send audio wherever you want (Qwen3-TTS, ElevenLabs, Groq, etc).
-```tsx
-import { useAudioRecording, useAudioPlayer } from 'talking-head-studio/voice';
-// No extra peer deps (browser APIs only)
-```
+const ref = useRef<WgpuAvatarRef>(null);
 
-### `talking-head-studio/sketchfab` — Sketchfab search & download
-Headless hooks and utilities for searching and downloading GLB models from Sketchfab. Bring your own UI and API key.
-```tsx
-import { useSketchfabSearch, ACCESSORY_CATEGORIES, downloadModel } from 'talking-head-studio/sketchfab';
-// No extra peer deps
+<WgpuAvatar
+  ref={ref}
+  avatarUrl="https://example.com/your-model.glb"
+  mood="neutral"
+  style={{ flex: 1 }}
+/>;
+// ref.current?.scheduleVisemes(payload) — same contract as the web component
 ```
 
 ---
 
-## Props
+## TalkingHead component — props & ref
+
+### Props
 
 | Prop | Type | Default | Description |
 |------|------|---------|-------------|
-| `avatarUrl` | `string` | **required** | URL to any `.glb` model. Rigged or non-rigged. |
-| `authToken` | `string \| null` | `null` | Bearer token sent when fetching the model URL. CDN URLs are excluded automatically. |
-| `mood` | `TalkingHeadMood` | `'neutral'` | Avatar expression. See [Moods](#moods) below. |
-| `cameraView` | `'head' \| 'upper' \| 'full'` | `'upper'` | Camera framing preset. |
-| `cameraDistance` | `number` | `-0.5` | Camera zoom offset. Negative values zoom in. |
-| `hairColor` | `string` | -- | CSS color applied to materials whose name contains `hair` or `fur`. |
-| `skinColor` | `string` | -- | CSS color applied to materials whose name contains `skin`, `body`, or `face`. |
-| `eyeColor` | `string` | -- | CSS color applied to materials whose name contains `eye` or `iris`. |
-| `accessories` | `TalkingHeadAccessory[]` | `[]` | Array of GLB items to attach to bones. See [Accessories](#accessories). |
-| `onReady` | `() => void` | -- | Fires once the avatar and scene are fully loaded. |
-| `onError` | `(message: string) => void` | -- | Fires on load failure. |
-| `style` | `ViewStyle` | -- | Container style (works on both native and web). |
-
-### Moods
-
-The `mood` prop accepts one of:
-
-```
-neutral | happy | sad | angry | excited | thinking | concerned | surprised
-```
-
-Mood can be changed at any time via props or the ref API. On rigged models, mood maps to blend shape expressions. On non-rigged models, mood is a no-op.
-
----
+| `avatarUrl` | `string` | required | Any `.glb`. Rigged or not. |
+| `authToken` | `string \| null` | `null` | Bearer token for authenticated GLB URLs. |
+| `mood` | `TalkingHeadMood` | `'neutral'` | `neutral \| happy \| sad \| angry \| fear \| disgust \| love \| sleep \| excited \| thinking \| concerned \| surprised` |
+| `cameraView` | `'head' \| 'upper' \| 'full'` | `'upper'` | Framing preset. |
+| `cameraDistance` | `number` | `-0.5` | Zoom offset. Negative = closer. |
+| `hairColor` | `string` | — | Hex color. Applied to materials named `hair`, `fur`. |
+| `skinColor` | `string` | — | Applied to `skin`, `body`, `face`. |
+| `eyeColor` | `string` | — | Applied to `eye`, `iris`. |
+| `accessories` | `TalkingHeadAccessory[]` | `[]` | Bone-attached GLB items. |
+| `onReady` | `() => void` | — | Fired when fully loaded. |
+| `onError` | `(msg: string) => void` | — | Fired on load failure. |
+| `style` | `ViewStyle / CSSProperties` | — | Container style. |
+
+### Ref methods
 
-## Ref API
-
-Access runtime controls through a React ref. Every method is safe to call at any time -- calls made before the avatar is ready are silently dropped.
-
-```tsx
-const ref = useRef<TalkingHeadRef>(null);
-
-// Drive lip-sync from an audio amplitude value (0..1)
-ref.current?.sendAmplitude(0.7);
+```ts
+// Lip-sync
+ref.current?.scheduleVisemes(payload); // AgentVisemePayload → full timed lip-sync schedule
+ref.current?.clearVisemes();
+ref.current?.sendAmplitude(0.7);       // amplitude 0..1 → jaw (fallback / no schedule)
 
-// Change expression
+// Expression & appearance
 ref.current?.setMood('excited');
-
-// Change colors at runtime
 ref.current?.setHairColor('#ff0000');
 ref.current?.setSkinColor('#8d5524');
 ref.current?.setEyeColor('#2e86de');
-
-// Swap accessories without re-mounting the component
-ref.current?.setAccessories([
-  {
-    id: 'crown',
-    url: 'https://example.com/crown.glb',
-    bone: 'Head',
-    position: [0, 0.22, 0],
-    rotation: [0, 0, 0],
-    scale: 0.8,
-  },
-]);
+ref.current?.setAccessories([...]);
+
+// Body — procedural motions, gestures, poses, animation clips
+ref.current?.dispatchMotion('groove');                       // looping procedural motion
+ref.current?.stopMotion();
+ref.current?.playGesture('thumbup');                         // upstream hand gesture
+ref.current?.playPose('oneknee');                            // upstream pose template
+ref.current?.playAnimation('/animations/wave.glb', { dur: 2 });
+ref.current?.lookAt(120, 80, 500);                           // turn toward viewport coords
 ```
 
-### Ref Methods
-
-| Method | Signature | Description |
-|--------|-----------|-------------|
-| `sendAmplitude` | `(amplitude: number) => void` | Feed audio amplitude (0 to 1) for jaw animation. |
-| `setMood` | `(mood: TalkingHeadMood) => void` | Change avatar expression at runtime. |
-| `setHairColor` | `(color: string) => void` | Update hair material color. |
-| `setSkinColor` | `(color: string) => void` | Update skin material color. |
-| `setEyeColor` | `(color: string) => void` | Update eye/iris material color. |
-| `setAccessories` | `(accessories: TalkingHeadAccessory[]) => void` | Replace the entire accessory set. Handles loading, diffing, and cleanup automatically. |
+The motion vocabulary (`groove`, `wave`, `nod`, `idle`, `attack`, `defend`, `celebrate`,
+plus every upstream gesture/pose name) is exported as typed constants —
+`MOTION_KEYS`, `TALKINGHEAD_GESTURES`, `TALKINGHEAD_POSES`, and the `isMotionKey()` guard —
+from both the package root and `talking-head-studio/contract`.
+
+### Runtime capability matrix
+
+Both renderers share one API; where native can't match the WebView's upstream rig, it
+falls back to a procedural approximation rather than failing. This table is the honest gap
+list.
+
+| Feature | Web (iframe) | Native (WGPU) | Notes |
+|---|:---:|:---:|---|
+| Viseme schedules (`scheduleVisemes`) | ✅ | ✅ | Both consume `AgentVisemePayload`. |
+| Amplitude jaw fallback (`sendAmplitude`) | ✅ | ⚠️ | Web drives jaw from amplitude; native exposes the method for API parity. |
+| Core procedural motions (`groove`, `attack`, `defend`) | ✅ | ✅ | Shared `MOTION_DEFS` source of truth. |
+| Gesture names (`thumbup`, `shrug`, …) | ✅ | ⚠️ | Web delegates to TalkingHead; native uses procedural approximations. |
+| Pose names (`oneknee`, `kneel`, `sitting`, …) | ✅ | ⚠️ | Web delegates to TalkingHead; native uses static procedural poses. |
+| Full mood vocabulary | ✅ | ✅ | All 8 upstream moods + friendly aliases. |
+| External animation clips (`playAnimation`) | ✅ | ⚠️ | Web delegates to TalkingHead; native plays GLB clips via `AnimationMixer`. |
+| Gaze (`lookAt`) | ✅ | ❌ | Native eye/head-gaze bridge is future work. |
+| Listening / mic-reactive mouth | ⚠️ | ❌ | Web can route host-provided audio; native bridge not implemented. |
 
 ---
 
-## Accessories
-
-Attach any GLB model to any bone on the avatar skeleton. The system handles loading, disposal, and transform updates.
+## Self-hosting the runtime assets
 
-### Accessory shape
+By default the web iframe pulls the TalkingHead rig, three.js, and the HeadAudio model
+from public CDNs (jsDelivr, gstatic). To run fully self-hosted — no external CDN — vendor
+those files and point the renderer at your own origin:
 
 ```ts
-interface TalkingHeadAccessory {
-  id: string;                        // Unique identifier for diffing
-  url: string;                       // URL to a .glb file
-  bone: string;                      // Target bone name (e.g. "Head", "RightHand", "Spine")
-  position: [number, number, number]; // Offset from the bone origin
-  rotation: [number, number, number]; // Euler rotation in radians
-  scale: number;                      // Uniform scale factor
-}
-```
+import { buildAvatarHtml } from 'talking-head-studio/html';
 
-### Example: hat + glasses + backpack
-
-```tsx
-<TalkingHead
-  avatarUrl="https://example.com/avatar.glb"
-  accessories={[
-    {
-      id: 'cowboy-hat',
-      url: '/models/cowboy-hat.glb',
-      bone: 'Head',
-      position: [0, 0.18, 0],
-      rotation: [0, 0, 0],
-      scale: 1.2,
-    },
-    {
-      id: 'aviators',
-      url: '/models/aviator-glasses.glb',
-      bone: 'Head',
-      position: [0, 0.06, 0.11],
-      rotation: [0, 0, 0],
-      scale: 1.0,
-    },
-    {
-      id: 'backpack',
-      url: '/models/backpack.glb',
-      bone: 'Spine1',
-      position: [0, 0, -0.15],
-      rotation: [0, Math.PI, 0],
-      scale: 0.9,
-    },
-  ]}
-/>
+const html = buildAvatarHtml({
+  avatarUrl: 'https://your-cdn/model.glb',
+  vendorBaseUrl: 'https://your-cdn/vendor', // serves three.module.js, talkinghead.mjs, etc.
+  // ...
+});
 ```
 
-### Common bone names
+`vendorBaseUrl` replaces every CDN reference; `dracoDecoderUrl` overrides the DRACO decoder
+location independently.
 
-Mixamo-rigged models typically expose these bones:
-
-```
-Head, Neck, Spine, Spine1, Spine2,
-LeftShoulder, LeftArm, LeftForeArm, LeftHand,
-RightShoulder, RightArm, RightForeArm, RightHand,
-LeftUpLeg, LeftLeg, LeftFoot,
-RightUpLeg, RightLeg, RightFoot
-```
-
-Bone matching is flexible -- if an exact match is not found, the component tries a prefix match (useful for Sketchfab exports like `Head_5`). If no bone matches, the accessory falls back to the scene root.
+---
 
-### Runtime accessory swaps
+## FaceControl — the lower-level contract
 
-```tsx
-// Remove all accessories
-ref.current?.setAccessories([]);
+If you're writing a custom backend or a game-engine integration, `FaceControl` is the
+single value that flows between a voice pipeline and any avatar backend.
 
-// Swap glasses for a monocle
-ref.current?.setAccessories([
-  { id: 'monocle', url: '/models/monocle.glb', bone: 'Head', position: [0.03, 0.07, 0.11], rotation: [0, 0, 0], scale: 0.6 },
-]);
+```ts
+import type { FaceControl, ExpressionState, HeadPose, EyeGaze } from 'talking-head-studio';
+
+type HeadPose = { yaw: number; pitch: number; roll: number };  // each -1..1
+type EyeGaze = { x: number; y: number };                       // each -1..1
+
+type ExpressionState = {
+  jawOpen: number; mouthSmile: number; mouthFunnel: number; mouthPucker: number;
+  mouthWide: number; upperLipRaise: number; lowerLipDepress: number; cheekRaise: number;
+  blinkLeft: number; blinkRight: number; browInnerUp: number;
+  browDownLeft: number; browDownRight: number;
+  eyeGazeLeft: EyeGaze; eyeGazeRight: EyeGaze;
+}; // all weights 0..1 unless noted
 ```
 
-Accessories that were previously loaded but are absent from the new array are automatically disposed (geometry, materials, textures).
-
----
-
-## Color Customization
+Drive it from a viseme schedule:
 
-Colors can be set via props (applied on initial load) or via the ref API (applied at runtime without reloading the model).
-
-The system matches material names against known keywords:
+```ts
+import { useFaceControlsFromVisemes } from 'talking-head-studio';
 
-| Target | Material name keywords |
-|--------|----------------------|
-| Hair | `hair`, `fur` |
-| Skin | `skin`, `body`, `face` |
-| Eyes | `eye`, `iris` |
+const faceControl = useFaceControlsFromVisemes(schedule); // rAF-sampled FaceControl
+```
 
-```tsx
-// Via props
-<TalkingHead hairColor="#2d1b00" skinColor="#f0c8a0" eyeColor="#3d6b4f" />
+Or implement a backend against it:
 
-// Via ref (runtime)
-ref.current?.setHairColor('#ff4500');
-ref.current?.setSkinColor('#c68642');
-ref.current?.setEyeColor('#1abc9c');
+```ts
+import type { AvatarBackend, AvatarRenderTarget, FaceControl } from 'talking-head-studio';
+
+class MyBackend implements AvatarBackend {
+  initialize() {}
+  attach(target: AvatarRenderTarget) {}
+  setControl(control: FaceControl) {}
+  renderFrame() {}
+  dispose() {}
+}
 ```
 
-This works on both rigged and non-rigged models -- any GLB with appropriately named materials will respond to color changes.
+### MorphTargetBackend — the built-in Three.js adapter
 
----
-
-## Voice Pipeline Integration
+The concrete `AvatarBackend` for GLB-with-morphs. Hand it a loaded scene; it discovers
+morph targets, builds a lookup cache, and drives them from `FaceControl`.
 
-The component is designed to sit at the end of a voice pipeline. Feed it audio amplitude and it handles the rest.
+```ts
+import { MorphTargetBackend, createNeutralExpression } from 'talking-head-studio';
+
+const backend = new MorphTargetBackend(gltf.scene, {
+  mood: 'neutral',
+  expressionScale: 1.0,
+  calibration: {
+    neutral: { pose: { yaw: 0, pitch: 0, roll: 0 }, expr: createNeutralExpression() },
+    ranges: { jawOpen: { min: 0, max: 0.85 } }, // clamp jaw for this model
+    gazeLimits: { x: { min: -0.6, max: 0.6 } },
+  },
+});
 
-### Primary: HeadAudio phoneme lip-sync
+backend.setControl(faceControl);
+backend.renderFrame();
+console.log(backend.availableChannels); // what this model actually supports
+```
 
-On rigged models in browser contexts with Web Audio available, [HeadAudio](https://github.com/met4citizen/HeadAudio) provides phoneme-level lip-sync automatically. Audio elements in the page are intercepted and routed through the lip-sync engine -- no wiring required on your end.
+### ARKit → Oculus remap (no ML, no artist work)
 
-### Fallback: amplitude-driven jaw
+```ts
+import { remapArkitToOculus, getArkitWeightsForViseme } from 'talking-head-studio';
 
-When phoneme-level lip-sync is unavailable (React Native WebView, non-rigged models, or missing blend shapes), `sendAmplitude` drives jaw movement directly via morph targets.
+remapArkitToOculus({ jawOpen: 0.7, mouthLowerDownLeft: 0.4 }); // → { aa: 0.68, oh: 0.12, ... }
+getArkitWeightsForViseme('ou');                                // → { mouthPucker: 0.9, ... }
+```
 
-### LiveKit integration
+The full `ARKIT_TO_OCULUS` coefficient table is exported for building your own bake pipeline.
 
-```tsx
-import { useDataChannel } from '@livekit/components-react';
+---
 
-function AvatarWithLiveKit() {
-  const ref = useRef<TalkingHeadRef>(null);
+## Accessories
 
-  useDataChannel('agent_speaking', (data) => {
-    if (data.amplitude !== undefined) {
-      ref.current?.sendAmplitude(data.amplitude);
-    }
-  });
+Any GLB attached to any skeleton bone, placeable at runtime.
 
-  return <TalkingHead ref={ref} avatarUrl="..." />;
+```ts
+interface TalkingHeadAccessory {
+  id: string;
+  url: string;
+  bone: string;                       // 'Head' | 'Spine' | 'RightHand' | ...
+  position: [number, number, number];
+  rotation: [number, number, number]; // Euler, radians
+  scale: number;
 }
 ```
 
-### Web Audio analyser
-
-```tsx
-const audioCtx = new AudioContext();
-const analyser = audioCtx.createAnalyser();
-const buf = new Uint8Array(analyser.frequencyBinCount);
-
-// Connect your audio source to the analyser
-source.connect(analyser);
-
-// Poll amplitude and feed the avatar
-const interval = setInterval(() => {
-  analyser.getByteFrequencyData(buf);
-  const amplitude = buf.reduce((a, b) => a + b, 0) / buf.length / 255;
-  ref.current?.sendAmplitude(amplitude);
-}, 50);
-```
-
-### Any audio source
-
-The only contract is a number between 0 and 1, called at roughly 20 Hz. This works with ElevenLabs, OpenAI Realtime, Deepgram, Whisper, or any other TTS/STT pipeline.
+Common Mixamo bones: `Head, Neck, Spine, Spine1, Spine2, LeftHand, RightHand, LeftFoot, RightFoot, Hips`.
+The 3D editor (`talking-head-studio/editor`, web only) provides a gizmo for live placement.
 
 ---
 
-## GLB Compatibility
-
-### Rigged models (full feature set)
-
-For the complete experience -- phoneme lip-sync, expressions, moods, gestures -- your GLB should have:
-
-- A **Mixamo-compatible armature** (the component expects standard bone names)
-- **ARKit blend shapes** and/or **Oculus viseme blend shapes** for lip-sync
-- Standard Three.js-compatible GLB format
-
-Models from [Avaturn](https://avaturn.me/) or any Mixamo-rigged source work out of the box.
-
-### Non-rigged models (static fallback)
-
-Any valid GLB loads successfully. Non-rigged models get:
+## Subpath exports
 
-- Auto-framing and centering in the viewport
-- Orbit controls for rotation
-- Embedded animation playback (walk cycles, idle loops, etc.)
-- Amplitude-driven jaw via morph targets (if the model has `jawOpen`, `mouthOpen`, or `viseme_aa` blend shapes)
-- Color customization (if materials are named appropriately)
-- Accessory attachment (falls back to scene root if no bones exist)
+| Import | Description |
+|------|-------------|
+| `talking-head-studio` | Avatar component + `FaceControl` contracts + motion constants |
+| `talking-head-studio/contract` | Stable type-only entrypoint — visemes, FaceControl, backends, motion |
+| `talking-head-studio/html` | `buildAvatarHtml()` for self-hosted / custom iframe embedding |
+| `talking-head-studio/wgpu` | React Native WebGPU renderer (`WgpuAvatar`) |
+| `talking-head-studio/editor` | R3F 3D editor with placement gizmo (web only) |
+| `talking-head-studio/appearance` | Material color system for any GLB |
+| `talking-head-studio/voice` | Audio recording + WAV conversion hooks |
+| `talking-head-studio/sketchfab` | Sketchfab search + download hooks |
+| `talking-head-studio/api` | Studio API client (avatar CRUD, voice profiles) |
+| `talking-head-studio/wardrobe` | Accessory + outfit state management |
 
-### Upstream documentation
-
-For detailed model authoring guidance, see the [TalkingHead documentation](https://github.com/met4citizen/TalkingHead).
+Workspace packages (`packages/avatar-creator`, `packages/agent-avatar`) ship an embeddable
+creator widget and a LiveKit + MCP agent integration.
 
 ---
 
-## Plain React / Next.js
-
-This works on the web without `react-native` or `react-native-webview` installed at runtime.
-
-On web, the component renders an `<iframe>` with `srcdoc` containing the full Three.js scene. No WebView, no native modules, no build plugins.
-
-```tsx
-// Works in any React 18+ web app
-import { TalkingHead } from 'talking-head-studio';
-
-export default function Page() {
-  return (
-    <TalkingHead
-      avatarUrl="/models/avatar.glb"
-      mood="happy"
-      style={{ width: 600, height: 800 }}
-    />
-  );
-}
-```
-
-Metro and Expo use the native entry backed by `react-native-webview`. Standard web bundlers use the browser entry backed by a plain `<iframe>`. The API is identical.
+## Roadmap
 
----
+> **Status legend:** ✅ shipped · 🔜 in progress · 🧪 designed, not yet built
 
-## MotionEngine (Upcoming)
+**Shipped today**
+- ✅ `FaceControl` face-control space (pose + expression + gaze) and `AvatarBackend` interface
+- ✅ `MorphTargetBackend` — GLB morph discovery + mood layering
+- ✅ ARKit → Oculus analytical remap with full coefficient table
+- ✅ `AgentVisemePayload` viseme schedule format + `scheduleVisemes` on both renderers
+- ✅ Shared procedural motion engine (web + native WGPU), gestures, poses, animation clips
+- ✅ Self-hosting via `buildAvatarHtml({ vendorBaseUrl })`
+- ✅ `packages/avatar-creator`, `packages/agent-avatar`
 
-[MotionEngine](https://github.com/lhupyn/motion-engine) integration is in development. This will add real-time body tracking and gesture replay to the avatar, driven by webcam or motion capture data.
+**In progress**
+- 🔜 Native (WGPU) gaze bridge (`lookAt`) and mic-reactive listening
+- 🔜 GLB schema walker — report morph coverage, bones, LODs, viseme tier for any model
 
-Stay tuned.
+**Designed, not yet built**
+- 🧪 `GaussianBackend` — Gaussian-splat renderer + FLAME per-viseme delta transfer, so a
+  model with *no* face rig still gets excellent lip-sync. This is the zero-prerequisite path.
+- 🧪 FLAME viseme transfer pipeline (companion backend) — bake Oculus visemes into a GLB
+  that lacks them
+- 🧪 Unity / Unreal SDKs implementing the same `AvatarBackend` contract
+- 🧪 Avatar marketplace + RPM import tooling (`CatalogItem` / `AvatarAsset` types exist;
+  backend and store do not)
 
 ---
 
 ## Contributing
 
-Contributions are welcome. Please open an issue to discuss your idea before submitting a pull request.
-
 ```bash
 git clone https://github.com/sitebay/talking-head-studio.git
 cd talking-head-studio
 npm install
-npm run typecheck
+npm run typecheck   # must be clean
 npm test
 ```
 
----
-
-## Credits
-
-This project builds on excellent open-source work:
-
-- [met4citizen/TalkingHead](https://github.com/met4citizen/TalkingHead) -- The 3D avatar engine powering model loading, rigging, and expression systems.
-- [met4citizen/HeadAudio](https://github.com/met4citizen/HeadAudio) -- Phoneme-based lip-sync from audio streams using AudioWorklet.
-- [lhupyn/motion-engine](https://github.com/lhupyn/motion-engine) -- Real-time body motion tracking (upcoming integration).
-- [Three.js](https://threejs.org/) -- 3D rendering, loaded via CDN at runtime.
-
----
-
-## License
-
-MIT
- at runtime.
+Monorepo with `packages/*` as npm workspaces; the main library is the root package. The
+publish gate (`prepublishOnly`) runs lint, typecheck, tests, and metadata checks.
 
 ---
 
-## License
+## Credits & license
 
-MIT
+Built on [met4citizen/TalkingHead](https://github.com/met4citizen/TalkingHead) (rig +
+gestures/poses on the web path) and [Three.js](https://threejs.org). MIT licensed.