talking-head-studio 0.4.9 → 0.4.11

package/README.md

@@ -1,6 +1,6 @@
 # talking-head-studio
 
-**The missing UI layer for AI Agents. Drop-in, lip-syncing 3D avatars for Web, React, and React Native.**
+**Open-source avatar platform for Web, React Native, Unity, and Unreal. Any GLB model. Full lip-sync — with or without blend shapes.**
 
 [![npm version](https://img.shields.io/npm/v/talking-head-studio.svg)](https://www.npmjs.com/package/talking-head-studio)
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
@@ -8,56 +8,64 @@
 
 ---
 
-## Why this?
+## What this is
 
-- **Zero-Jank React Native & Web:** True cross-platform rendering. React Native gets a blazing fast wgpu-accelerated native render loop, skipping WebView bridge latency entirely. React on web gets a robust `react-three-fiber` setup. Same API, same props.
-- **Universal GLB Compatibility:** Bring any GLB. Out-of-the-box support for standard ARKit blendshapes. Rigged models get full phoneme-based lip-sync. Non-rigged models get an amplitude-driven jaw animation fallback.
-- **Built for AI & Voice Pipelines:** Wire `sendAmplitude` or visemes directly to LiveKit, Web Audio, ElevenLabs, OpenAI Realtime, or any audio source.
-- **Always Alive:** Procedural idle animations (breathing, nodding, swaying) keep your avatar from feeling like a static doll.
-- **Dynamic Wardrobe & Accessories:** Swap hair, skin, and eye colors on the fly. Attach hats, glasses, or backpacks to any bone at runtime.
+A drop-in avatar runtime and platform SDK built to be a self-hostable replacement for Ready Player Me. The core problem it solves: **any arbitrary 3D model should be able to talk, emote, and respond to a voice pipeline** — regardless of whether the artist baked in blend shapes, visemes, or any face rig at all.
+
+The library ships a renderer (web iframe + React Native wgpu), a backend-agnostic face control contract, and a growing set of adapters that map TTS/audio/AI output onto whatever rendering mechanism the model actually supports.
 
 ---
 
-## Table of Contents
-
-- [Installation](#installation)
-- [Quick Start](#quick-start)
-- [Subpath Exports](#subpath-exports)
-- [Props](#props)
-- [Ref API](#ref-api)
-- [Accessories](#accessories)
-- [Color Customization](#color-customization)
-- [Voice Pipeline Integration](#voice-pipeline-integration)
-- [GLB Compatibility](#glb-compatibility)
-- [Plain React / Next.js](#plain-react--nextjs)
-- [MotionEngine (Upcoming)](#motionengine-upcoming)
-- [Contributing](#contributing)
-- [Credits](#credits)
-- [License](#license)
+## Lip-sync tiers (any model works)
 
----
+| Model type | Lip-sync method | Quality |
+|---|---|---|
+| GLB with Oculus viseme morphs | Direct morph drive via `MorphTargetBackend` | Excellent |
+| GLB with ARKit blend shapes | `remapArkitToOculus()` → morph drive | Good |
+| GLB with only `jawOpen` / `mouthOpen` | Amplitude fallback | Acceptable |
+| Any other GLB | Gaussian splat backend *(roadmap)* | Excellent |
 
-## Installation
+The last row is the goal: **scan any model into a Gaussian representation, generate per-viseme deltas via FLAME-based transfer, and drive it from the same `FaceControl` contract everything else uses.** No blend shapes required. No artist work required.
 
-### React Native / Expo
+---
+
+## Architecture
 
-```bash
-npm install talking-head-studio react-native-webview
+```
+TTS / audio / face tracking
+        ↓
+  AgentVisemePayload          ← canonical wire format for lip-sync schedules
+        ↓
+  FaceControl                 ← pose (HeadPose) + expression (ExpressionState) + gaze (EyeGaze)
+        ↓
+  AvatarBackend  ←────────────── swap without changing anything upstream
+    ├── MorphTargetBackend    ← Three.js morph targets (GLB with blend shapes)
+    ├── GaussianBackend       ← [roadmap] Gaussian splat + FLAME delta transfer
+    └── (your backend)        ← implement AvatarBackend, plug in
+        ↓
+  Renderer
+    ├── Web iframe            ← TalkingHead.web.tsx (any React app)
+    ├── React Native wgpu     ← WgpuAvatar (native GPU, no WebView latency)
+    └── Unity / Unreal        ← [roadmap] SDK plugins consuming same contracts
 ```
 
-`react-native-webview` is a peer dependency. If you are using Expo, it is available as a built-in package.
+Everything above `AvatarBackend` is renderer-agnostic. Everything above `FaceControl` is model-agnostic.
 
-### Web only (React, Next.js, Vite)
+---
+
+## Installation
 
 ```bash
+# React Native / Expo
+npm install talking-head-studio react-native-webview
+
+# Web (React, Next.js, Vite)
 npm install talking-head-studio
 ```
 
-No `react-native` or WebView runtime dependency needed. The package ships a web entry point that renders via `<iframe srcdoc>` automatically when bundled for browser targets.
-
 ---
 
-## Quick Start
+## Quick start
 
 ```tsx
 import { useRef } from 'react';
@@ -74,19 +82,16 @@ export default function Avatar() {
       cameraView="upper"
       hairColor="#1a1a2e"
       skinColor="#e0a370"
-      accessories={[
-        {
-          id: 'sunglasses',
-          url: 'https://example.com/sunglasses.glb',
-          bone: 'Head',
-          position: [0, 0.08, 0.12],
-          rotation: [0, 0, 0],
-          scale: 1.0,
-        },
-      ]}
+      accessories={[{
+        id: 'sunglasses',
+        url: 'https://example.com/sunglasses.glb',
+        bone: 'Head',
+        position: [0, 0.08, 0.12],
+        rotation: [0, 0, 0],
+        scale: 1.0,
+      }]}
       style={{ width: 400, height: 600 }}
-      onReady={() => console.log('Avatar loaded')}
-      onError={(msg) => console.error('Load failed:', msg)}
+      onReady={() => console.log('ready')}
     />
   );
 }
@@ -94,372 +99,243 @@ export default function Avatar() {
 
 ---
 
-## Subpath Exports
-
-The package ships five independent entry points. Import only what you need — each subpath has its own optional peer dependencies.
+## FaceControl — the core contract
 
-### `talking-head-studio` — Live talking avatar
-```tsx
-import { TalkingHead } from 'talking-head-studio';
-// Peer deps: react
-// Native-only peers: react-native (optional), react-native-webview (optional)
-```
+The `FaceControl` type is the single value that flows between your voice pipeline and any avatar backend. If you're building a custom backend or integrating with a game engine, this is what you implement against.
 
-### `talking-head-studio/editor` — 3D editor with gizmo (web)
-R3F-based canvas with PivotControls gizmo for placing accessories on an avatar. Web only.
-```tsx
-import { AvatarCanvas } from 'talking-head-studio/editor';
-// Peer deps: @react-three/fiber, @react-three/drei, three
+```ts
+import type { FaceControl, ExpressionState, HeadPose, EyeGaze } from 'talking-head-studio';
+
+type HeadPose = {
+  yaw:   number; // -1..1, left..right
+  pitch: number; // -1..1, down..up
+  roll:  number; // -1..1, tilt
+};
+
+type EyeGaze = {
+  x: number; // -1..1, left..right
+  y: number; // -1..1, down..up
+};
+
+type ExpressionState = {
+  jawOpen:         number; // 0..1
+  mouthSmile:      number;
+  mouthFunnel:     number;
+  mouthPucker:     number;
+  mouthWide:       number;
+  upperLipRaise:   number;
+  lowerLipDepress: number;
+  cheekRaise:      number;
+  blinkLeft:       number;
+  blinkRight:      number;
+  browInnerUp:     number;
+  browDownLeft:    number;
+  browDownRight:   number;
+  eyeGazeLeft:     EyeGaze;
+  eyeGazeRight:    EyeGaze;
+};
 ```
 
-### `talking-head-studio/appearance` — Material color system
-Apply skin/hair/eye colors to any GLB avatar. Works in both the live view and the 3D editor.
-```tsx
-import { applyAppearanceToObject3D, type AvatarAppearance } from 'talking-head-studio/appearance';
-// No extra peer deps
-```
+### Driving FaceControl from a viseme schedule
 
-### `talking-head-studio/voice` — Audio recording hooks
-Headless hooks for recording voice samples (WebM→WAV conversion included). Backend-agnostic — send audio wherever you want (Qwen3-TTS, ElevenLabs, Groq, etc).
-```tsx
-import { useAudioRecording, useAudioPlayer } from 'talking-head-studio/voice';
-// No extra peer deps (browser APIs only)
-```
+```ts
+import { useFaceControlsFromVisemes } from 'talking-head-studio';
 
-### `talking-head-studio/sketchfab` — Sketchfab search & download
-Headless hooks and utilities for searching and downloading GLB models from Sketchfab. Bring your own UI and API key.
-```tsx
-import { useSketchfabSearch, ACCESSORY_CATEGORIES, downloadModel } from 'talking-head-studio/sketchfab';
-// No extra peer deps
+// schedule: AgentVisemePayload from your TTS backend
+const faceControl = useFaceControlsFromVisemes(schedule);
+// → { pose: { yaw:0, pitch:0, roll:0 }, expr: { jawOpen: 0.7, ... } }
 ```
 
----
-
-## Props
-
-| Prop | Type | Default | Description |
-|------|------|---------|-------------|
-| `avatarUrl` | `string` | **required** | URL to any `.glb` model. Rigged or non-rigged. |
-| `authToken` | `string \| null` | `null` | Bearer token sent when fetching the model URL. CDN URLs are excluded automatically. |
-| `mood` | `TalkingHeadMood` | `'neutral'` | Avatar expression. See [Moods](#moods) below. |
-| `cameraView` | `'head' \| 'upper' \| 'full'` | `'upper'` | Camera framing preset. |
-| `cameraDistance` | `number` | `-0.5` | Camera zoom offset. Negative values zoom in. |
-| `hairColor` | `string` | -- | CSS color applied to materials whose name contains `hair` or `fur`. |
-| `skinColor` | `string` | -- | CSS color applied to materials whose name contains `skin`, `body`, or `face`. |
-| `eyeColor` | `string` | -- | CSS color applied to materials whose name contains `eye` or `iris`. |
-| `accessories` | `TalkingHeadAccessory[]` | `[]` | Array of GLB items to attach to bones. See [Accessories](#accessories). |
-| `onReady` | `() => void` | -- | Fires once the avatar and scene are fully loaded. |
-| `onError` | `(message: string) => void` | -- | Fires on load failure. |
-| `style` | `ViewStyle` | -- | Container style (works on both native and web). |
-
-### Moods
-
-The `mood` prop accepts one of:
+### Implementing a custom backend
 
-```
-neutral | happy | sad | angry | excited | thinking | concerned | surprised
+```ts
+import type { AvatarBackend, AvatarRenderTarget, FaceControl } from 'talking-head-studio';
+
+class MyGaussianBackend implements AvatarBackend {
+  initialize() { /* load splat data, FLAME weights */ }
+  attach(target: AvatarRenderTarget) { /* bind to canvas/surface */ }
+  setControl(control: FaceControl) { /* map ExpressionState → splat coefficients */ }
+  renderFrame() { /* rasterize */ }
+  dispose() { /* cleanup */ }
+}
 ```
 
-Mood can be changed at any time via props or the ref API. On rigged models, mood maps to blend shape expressions. On non-rigged models, mood is a no-op.
-
 ---
 
-## Ref API
-
-Access runtime controls through a React ref. Every method is safe to call at any time -- calls made before the avatar is ready are silently dropped.
+## MorphTargetBackend — Three.js GLB adapter
 
-```tsx
-const ref = useRef<TalkingHeadRef>(null);
+The first concrete `AvatarBackend` implementation. Give it any loaded Three.js scene and it will find morph targets, build a lookup cache, and drive them from `FaceControl`.
 
-// Drive lip-sync from an audio amplitude value (0..1)
-ref.current?.sendAmplitude(0.7);
-
-// Change expression
-ref.current?.setMood('excited');
-
-// Change colors at runtime
-ref.current?.setHairColor('#ff0000');
-ref.current?.setSkinColor('#8d5524');
-ref.current?.setEyeColor('#2e86de');
-
-// Swap accessories without re-mounting the component
-ref.current?.setAccessories([
-  {
-    id: 'crown',
-    url: 'https://example.com/crown.glb',
-    bone: 'Head',
-    position: [0, 0.22, 0],
-    rotation: [0, 0, 0],
-    scale: 0.8,
+```ts
+import * as THREE from 'three';
+import { GLTFLoader } from 'three/examples/jsm/loaders/GLTFLoader';
+import { MorphTargetBackend } from 'talking-head-studio';
+
+const loader = new GLTFLoader();
+const gltf = await loader.loadAsync('/avatar.glb');
+
+const backend = new MorphTargetBackend(gltf.scene, {
+  mood: 'neutral',
+  expressionScale: 1.0,
+  calibration: {
+    neutral: { pose: { yaw: 0, pitch: 0, roll: 0 }, expr: createNeutralExpression() },
+    ranges: { jawOpen: { min: 0, max: 0.85 } }, // clamp jaw for this model
+    gazeLimits: { x: { min: -0.6, max: 0.6 } },
   },
-]);
-```
+});
 
-### Ref Methods
+// Each frame:
+backend.setControl(faceControl);
+backend.renderFrame();
 
-| Method | Signature | Description |
-|--------|-----------|-------------|
-| `sendAmplitude` | `(amplitude: number) => void` | Feed audio amplitude (0 to 1) for jaw animation. |
-| `setMood` | `(mood: TalkingHeadMood) => void` | Change avatar expression at runtime. |
-| `setHairColor` | `(color: string) => void` | Update hair material color. |
-| `setSkinColor` | `(color: string) => void` | Update skin material color. |
-| `setEyeColor` | `(color: string) => void` | Update eye/iris material color. |
-| `setAccessories` | `(accessories: TalkingHeadAccessory[]) => void` | Replace the entire accessory set. Handles loading, diffing, and cleanup automatically. |
+// Debug: what morphs does this model actually have?
+console.log(backend.availableChannels);
+// → { visemes: ['aa','PP','oh',...], expressions: ['jawOpen','blinkLeft',...], gaze: ['lookLeft','lookUp'] }
+```
 
 ---
 
-## Accessories
-
-Attach any GLB model to any bone on the avatar skeleton. The system handles loading, disposal, and transform updates.
+## ARKit → Oculus remap
 
-### Accessory shape
+Models with ARKit blend shapes (52 facial action units) but no Oculus viseme morphs can be remapped analytically — no ML, no FLAME, no artist work.
 
 ```ts
-interface TalkingHeadAccessory {
-  id: string;                        // Unique identifier for diffing
-  url: string;                       // URL to a .glb file
-  bone: string;                      // Target bone name (e.g. "Head", "RightHand", "Spine")
-  position: [number, number, number]; // Offset from the bone origin
-  rotation: [number, number, number]; // Euler rotation in radians
-  scale: number;                      // Uniform scale factor
-}
+import { remapArkitToOculus, getArkitWeightsForViseme } from 'talking-head-studio';
+
+// Runtime: face tracking data → Oculus viseme weights
+const oculusWeights = remapArkitToOculus({
+  jawOpen: 0.7,
+  mouthLowerDownLeft: 0.4,
+  mouthLowerDownRight: 0.4,
+});
+// → { aa: 0.68, PP: 0.03, oh: 0.12, ... }
+
+// Bake-time: get the ARKit recipe for a specific viseme
+const recipe = getArkitWeightsForViseme('ou');
+// → { mouthPucker: 0.9, mouthRollLower: 0.3 }
 ```
 
-### Example: hat + glasses + backpack
-
-```tsx
-<TalkingHead
-  avatarUrl="https://example.com/avatar.glb"
-  accessories={[
-    {
-      id: 'cowboy-hat',
-      url: '/models/cowboy-hat.glb',
-      bone: 'Head',
-      position: [0, 0.18, 0],
-      rotation: [0, 0, 0],
-      scale: 1.2,
-    },
-    {
-      id: 'aviators',
-      url: '/models/aviator-glasses.glb',
-      bone: 'Head',
-      position: [0, 0.06, 0.11],
-      rotation: [0, 0, 0],
-      scale: 1.0,
-    },
-    {
-      id: 'backpack',
-      url: '/models/backpack.glb',
-      bone: 'Spine1',
-      position: [0, 0, -0.15],
-      rotation: [0, Math.PI, 0],
-      scale: 0.9,
-    },
-  ]}
-/>
-```
-
-### Common bone names
-
-Mixamo-rigged models typically expose these bones:
-
-```
-Head, Neck, Spine, Spine1, Spine2,
-LeftShoulder, LeftArm, LeftForeArm, LeftHand,
-RightShoulder, RightArm, RightForeArm, RightHand,
-LeftUpLeg, LeftLeg, LeftFoot,
-RightUpLeg, RightLeg, RightFoot
-```
-
-Bone matching is flexible -- if an exact match is not found, the component tries a prefix match (useful for Sketchfab exports like `Head_5`). If no bone matches, the accessory falls back to the scene root.
-
-### Runtime accessory swaps
-
-```tsx
-// Remove all accessories
-ref.current?.setAccessories([]);
-
-// Swap glasses for a monocle
-ref.current?.setAccessories([
-  { id: 'monocle', url: '/models/monocle.glb', bone: 'Head', position: [0.03, 0.07, 0.11], rotation: [0, 0, 0], scale: 0.6 },
-]);
-```
-
-Accessories that were previously loaded but are absent from the new array are automatically disposed (geometry, materials, textures).
+The full `ARKIT_TO_OCULUS` coefficient table is exported so you can build your own bake pipeline.
 
 ---
 
-## Color Customization
-
-Colors can be set via props (applied on initial load) or via the ref API (applied at runtime without reloading the model).
+## TalkingHead component — props & ref
 
-The system matches material names against known keywords:
+### Props
 
-| Target | Material name keywords |
-|--------|----------------------|
-| Hair | `hair`, `fur` |
-| Skin | `skin`, `body`, `face` |
-| Eyes | `eye`, `iris` |
-
-```tsx
-// Via props
-<TalkingHead hairColor="#2d1b00" skinColor="#f0c8a0" eyeColor="#3d6b4f" />
+| Prop | Type | Default | Description |
+|------|------|---------|-------------|
+| `avatarUrl` | `string` | required | Any `.glb`. Rigged or not. |
+| `authToken` | `string \| null` | `null` | Bearer token for authenticated GLB URLs. |
+| `mood` | `TalkingHeadMood` | `'neutral'` | `neutral \| happy \| sad \| angry \| excited \| thinking \| concerned \| surprised` |
+| `cameraView` | `'head' \| 'upper' \| 'full'` | `'upper'` | Framing preset. |
+| `cameraDistance` | `number` | `-0.5` | Zoom offset. Negative = closer. |
+| `hairColor` | `string` | — | Hex color. Applied to materials named `hair`, `fur`. |
+| `skinColor` | `string` | — | Applied to `skin`, `body`, `face`. |
+| `eyeColor` | `string` | — | Applied to `eye`, `iris`. |
+| `accessories` | `TalkingHeadAccessory[]` | `[]` | Bone-attached GLB items. |
+| `onReady` | `() => void` | — | Fired when fully loaded. |
+| `onError` | `(msg: string) => void` | — | Fired on load failure. |
+| `style` | `ViewStyle / CSSProperties` | — | Container style. |
+
+### Ref methods
 
-// Via ref (runtime)
-ref.current?.setHairColor('#ff4500');
-ref.current?.setSkinColor('#c68642');
-ref.current?.setEyeColor('#1abc9c');
+```ts
+ref.current?.sendAmplitude(0.7);       // amplitude 0..1 → jaw
+ref.current?.scheduleVisemes(payload); // AgentVisemePayload → full lip-sync schedule
+ref.current?.clearVisemes();
+ref.current?.setMood('excited');
+ref.current?.setHairColor('#ff0000');
+ref.current?.setSkinColor('#8d5524');
+ref.current?.setEyeColor('#2e86de');
+ref.current?.setAccessories([...]);
+ref.current?.dispatchMotion('nod');
 ```
 
-This works on both rigged and non-rigged models -- any GLB with appropriately named materials will respond to color changes.
-
 ---
 
-## Voice Pipeline Integration
-
-The component is designed to sit at the end of a voice pipeline. Feed it audio amplitude and it handles the rest.
-
-### Primary: HeadAudio phoneme lip-sync
-
-On rigged models in browser contexts with Web Audio available, [HeadAudio](https://github.com/met4citizen/HeadAudio) provides phoneme-level lip-sync automatically. Audio elements in the page are intercepted and routed through the lip-sync engine -- no wiring required on your end.
-
-### Fallback: amplitude-driven jaw
-
-When phoneme-level lip-sync is unavailable (React Native WebView, non-rigged models, or missing blend shapes), `sendAmplitude` drives jaw movement directly via morph targets.
-
-### LiveKit integration
-
-```tsx
-import { useDataChannel } from '@livekit/components-react';
-
-function AvatarWithLiveKit() {
-  const ref = useRef<TalkingHeadRef>(null);
+## Accessories
 
-  useDataChannel('agent_speaking', (data) => {
-    if (data.amplitude !== undefined) {
-      ref.current?.sendAmplitude(data.amplitude);
-    }
-  });
+Any GLB attached to any skeleton bone. Placement is editable at runtime via the 3D editor.
 
-  return <TalkingHead ref={ref} avatarUrl="..." />;
+```ts
+interface TalkingHeadAccessory {
+  id: string;
+  url: string;
+  bone: string;                       // 'Head' | 'Spine' | 'RightHand' | ...
+  position: [number, number, number];
+  rotation: [number, number, number]; // Euler, radians
+  scale: number;
 }
 ```
 
-### Web Audio analyser
-
-```tsx
-const audioCtx = new AudioContext();
-const analyser = audioCtx.createAnalyser();
-const buf = new Uint8Array(analyser.frequencyBinCount);
-
-// Connect your audio source to the analyser
-source.connect(analyser);
-
-// Poll amplitude and feed the avatar
-const interval = setInterval(() => {
-  analyser.getByteFrequencyData(buf);
-  const amplitude = buf.reduce((a, b) => a + b, 0) / buf.length / 255;
-  ref.current?.sendAmplitude(amplitude);
-}, 50);
-```
-
-### Any audio source
+Common Mixamo bones: `Head, Neck, Spine, Spine1, Spine2, LeftHand, RightHand, LeftFoot, RightFoot, Hips`
 
-The only contract is a number between 0 and 1, called at roughly 20 Hz. This works with ElevenLabs, OpenAI Realtime, Deepgram, Whisper, or any other TTS/STT pipeline.
+The 3D editor (`talking-head-studio/editor`) provides a gizmo for live placement with front/top/side views. LLM-assisted placement is available via the companion backend.
 
 ---
 
-## GLB Compatibility
-
-### Rigged models (full feature set)
-
-For the complete experience -- phoneme lip-sync, expressions, moods, gestures -- your GLB should have:
-
-- A **Mixamo-compatible armature** (the component expects standard bone names)
-- **ARKit blend shapes** and/or **Oculus viseme blend shapes** for lip-sync
-- Standard Three.js-compatible GLB format
-
-Models from [Avaturn](https://avaturn.me/) or any Mixamo-rigged source work out of the box.
-
-### Non-rigged models (static fallback)
-
-Any valid GLB loads successfully. Non-rigged models get:
-
-- Auto-framing and centering in the viewport
-- Orbit controls for rotation
-- Embedded animation playback (walk cycles, idle loops, etc.)
-- Amplitude-driven jaw via morph targets (if the model has `jawOpen`, `mouthOpen`, or `viseme_aa` blend shapes)
-- Color customization (if materials are named appropriately)
-- Accessory attachment (falls back to scene root if no bones exist)
-
-### Upstream documentation
-
-For detailed model authoring guidance, see the [TalkingHead documentation](https://github.com/met4citizen/TalkingHead).
-
----
-
-## Plain React / Next.js
-
-This works on the web without `react-native` or `react-native-webview` installed at runtime.
-
-On web, the component renders an `<iframe>` with `srcdoc` containing the full Three.js scene. No WebView, no native modules, no build plugins.
-
-```tsx
-// Works in any React 18+ web app
-import { TalkingHead } from 'talking-head-studio';
-
-export default function Page() {
-  return (
-    <TalkingHead
-      avatarUrl="/models/avatar.glb"
-      mood="happy"
-      style={{ width: 600, height: 800 }}
-    />
-  );
-}
-```
-
-Metro and Expo use the native entry backed by `react-native-webview`. Standard web bundlers use the browser entry backed by a plain `<iframe>`. The API is identical.
+## Packages
+
+| Path | Description |
+|------|-------------|
+| `talking-head-studio` | Live avatar renderer + FaceControl contracts |
+| `talking-head-studio/editor` | R3F-based 3D editor with gizmo (web only) |
+| `talking-head-studio/appearance` | Material color system for any GLB |
+| `talking-head-studio/voice` | Audio recording + WAV conversion hooks |
+| `talking-head-studio/sketchfab` | Sketchfab search + download hooks |
+| `talking-head-studio/api` | Studio API client (avatar CRUD, voice profiles) |
+| `talking-head-studio/wardrobe` | Accessory + outfit state management |
+| `talking-head-studio/wgpu` | React Native wgpu renderer |
+| `packages/avatar-creator` | Embeddable avatar creator widget |
+| `packages/agent-avatar` | LiveKit agent + MCP integration |
 
 ---
 
-## MotionEngine (Upcoming)
-
-[MotionEngine](https://github.com/lhupyn/motion-engine) integration is in development. This will add real-time body tracking and gesture replay to the avatar, driven by webcam or motion capture data.
-
-Stay tuned.
+## Roadmap
+
+### Now — shipped
+- `FaceControl` canonical face control space (pose + expression + gaze)
+- `AvatarBackend` interface — swap renderers without changing upstream code
+- `MorphTargetBackend` — Three.js GLB adapter with morph target discovery and mood layering
+- ARKit → Oculus analytical remap (`remapArkitToOculus`, full coefficient table)
+- `useFaceControlsFromVisemes` — rAF-sampled hook from `AgentVisemePayload`
+- `AgentVisemePayload` canonical TTS → lip-sync wire format
+- `AvatarGlbParams` — typed API contract for quality/compression/morph group selection
+- `CalibrationProfile` — per-avatar range remapping and gaze limits
+- Platform type stubs: SDK (web/Unity/Unreal), marketplace catalog, avatar GLB API
+- `packages/avatar-creator` — embeddable creator widget with preset catalog
+- `packages/agent-avatar` — LiveKit agent + MCP tool integration
+
+### Next
+- **GLB schema walker** — scan any loaded GLB and report: morph target coverage, skeleton bones, LODs, viseme tier. Prerequisite for the validator and import pipeline.
+- **`GET /avatars/{id}.glb` with `AvatarGlbParams`** — extend the companion backend to serve quality/compression/morph-group variants on the existing endpoint.
+- **Creator postMessage bridge** — let partners embed the avatar creator in an iframe and receive avatar IDs back, like RPM's WebView creator.
+
+### Medium term
+- **`GaussianBackend`** — Gaussian splat renderer implementing `AvatarBackend`. Takes any model, scans it, drives expression via FLAME-based per-viseme delta transfer. No artist work, no blend shapes required. This is the zero-prerequisite lip-sync path.
+- **FLAME viseme transfer pipeline** (Python, companion backend) — fit FLAME to a face screenshot, generate Oculus viseme deltas, bake back into the GLB as morph targets. Background task on upload for any avatar missing viseme morphs.
+- **Unity SDK** — C# plugin implementing the `AvatarBackend` contract. Blueprint-friendly API for loading GLBs, driving morphs, consuming `AgentVisemePayload`.
+- **Unreal plugin** — UE5 plugin with Blueprint-accessible `UAvatarDescriptor` and a sample Quickstart map.
+
+### Longer term
+- Avatar marketplace — `CatalogItem`, `AvatarAsset`, `RarityLevel` types are already defined. Backend + web store + in-creator purchasing.
+- RPM migration tools — import existing RPM avatars where technically possible.
+- SLA + deprecation policy — for teams that need a reliability guarantee as they move off RPM.
 
 ---
 
 ## Contributing
 
-Contributions are welcome. Please open an issue to discuss your idea before submitting a pull request.
-
 ```bash
 git clone https://github.com/sitebay/talking-head-studio.git
 cd talking-head-studio
 npm install
-npm run typecheck
+npm run typecheck   # must be clean (excluding known expo-audio peer dep warnings)
 npm test
 ```
 
----
-
-## Credits
-
-This project builds on excellent open-source work:
-
-- [met4citizen/TalkingHead](https://github.com/met4citizen/TalkingHead) -- The 3D avatar engine powering model loading, rigging, and expression systems.
-- [met4citizen/HeadAudio](https://github.com/met4citizen/HeadAudio) -- Phoneme-based lip-sync from audio streams using AudioWorklet.
-- [lhupyn/motion-engine](https://github.com/lhupyn/motion-engine) -- Real-time body motion tracking (upcoming integration).
-- [Three.js](https://threejs.org/) -- 3D rendering, loaded via CDN at runtime.
-
----
-
-## License
-
-MIT
- at runtime.
+The repo is a monorepo with `packages/*` as npm workspaces. The main library is the root package.
 
 ---