@convai/embody 1.0.1-beta.0 → 1.1.0-beta.0

package/CHANGELOG.md

@@ -1,5 +1,43 @@
 # @convai/embody
 
+## 1.1.0-beta.0
+
+### Minor Changes
+
+- 62874a4: Characters at the studio films' quality under any app lighting, lip sync that stays on the voice, and a check of any model at load.
+
+  **Lip sync**
+
+  - The mouth follows the voice's own clock. Each reply anchors on the voice's first sound and the stream's first mouth movement (any mouth channel, so a reply opening on rounded or closed lips no longer runs a second early), with one consistent lead on every reply.
+  - A reply ends with one smooth close: no pop after the last word, and a stream held under the voice's tail is not mistaken for a stall.
+  - Voices that arrive in chunks with stalls between them (OpenAI voices) keep their lips on the sound: after a stall the mouth advances only over frames that were waiting, and frames that arrive after a dry spell start at the voice's next onset instead of a catch-up burst.
+  - No voice plays on a still mouth: when the stream runs dry mid-reply the voice level drives the mouth until frames return, eased in and out.
+  - The voice level is read from inbound-rtp stats when RTP carries none, and ignored until the voice is actually played out (autoplay-blocked pages no longer flutter the mouth).
+
+  **Rendering**
+
+  - Skin, hair, eyes and teeth take the app's lights: physically based skin with light wrap and pores, hair drawn as coverage with faint hair-coloured highlights and depth toward the scalp, living eyes with a measured iris and a catchlight, lit teeth shaded by the mouth, shadows on skin that read as shade.
+  - The scalp under the hair is shaded in the hair's colour; no band under the hairline, no bare parting, no grey patch at a high hairline.
+  - `PortraitLights` for `ConvaiCharacter`'s own canvas and `createPortraitRig` / `placePortraitRig` for plain three.js scenes: a portrait rig that follows the camera.
+  - A male record (the body manifest's `gender`) gets a male look: lashes at the texture's own alpha, a tighter and less pink skin response with more pore, less light wrap, and a `steady` resting mood. Every value is overridable (`SurfaceTuning.gender`, `lashOpacity`, `SkinTuning.roughness`/`sheen`/`sheenColor`/`specular`/`tint`, `mouth.expression`).
+  - Normals for morphed faces that ship no blend-shape normals (`morphNormals`, on by default): closed eyelids no longer shade black.
+  - Hair never moves into the body; hair physics was removed in favour of resting hair on the shoulders.
+
+  **Models and motion**
+
+  - `inspectModel(url, options)` and `character.report` check any GLB at load: morph naming and mapping (`morphMap`, auto for MetaHuman, Character Creator and ARKit), what the face cannot do, the skeleton, and every animation clip, each problem with the config or export change that fixes it. `diagnostics` prints it.
+  - Mixamo and Rigify clips are retargeted onto the body at load (`animations`).
+  - Gaze moves the eyes, then the neck, then the torso, added to the clip pose; each eye is aimed from the body's own eye shapes.
+  - Scale is measured once and the soles are grounded, not the ankle.
+
+## 1.0.1
+
+### Patch Changes
+
+- c779ffd: Fix: `ConvaiCharacter` and `CharacterModel` call `onReady` (and `onError`) once per character. An inline `onReady={(c) => …}` used to run again on every parent re-render, repeating whatever setup the app did there, such as adding a `speakingChange` listener.
+
+  Docs: two new guides, "Add your own animations to a character" (the `animations` option, `idleClip`, `crossfadeTo`, a talking clip on `speakingChange`, `register` at runtime) and "Use your own 3D model" (a resolver for a MetaHuman GLB you host, the rig requirements, Full and Lite files, `connect: false` previews). The `createCharacter` reference gains `modelVariant`, `character.model`, and sections for `AnimationController` and the `CharacterBundle` manifest; the React reference says which props reload the character. The README's Lite count no longer says 113.
+
 ## 1.0.1-beta.0
 
 ### Patch Changes

package/README.md

@@ -190,21 +190,25 @@ load-bearing, not incidental:
 
 | Stage | System | Why here |
 |---|---|---|
-| 0 normalize | unit normalizer | root scale must settle before anything reads world space |
-| 1 mixer | animation clips | drives the skeleton |
-| 2 gaze | `GazeSystem` | **overrides** the mixer's head rotation rather than blending with it |
+| 0 prepare | gaze restore | undoes last frame's gaze turn on bones the clip will not rewrite (scale is measured once at load) |
+| 1 mixer → ground | animation clips, then grounding | clips drive the skeleton, then that pose's soles are put on the floor |
+| 2 gaze | `GazeSystem` | eyes-then-neck-then-torso turns **added** to the clip's pose, so its sway and gestures stay |
 | 3 additive | `BreathingSystem` | added *after* the mixer so nothing accumulates |
-| 4 ik / 5 attach | *(reserved)* | foot lock and hair land here later |
+| 4 ik / 5 attach | *(reserved)* | |
 | 6 morph | lipsync → `EyeGazeSystem` → `BlinkSystem` | blink must run after lipsync to win the eyelid channels |
 
-- **Gaze** — head/neck solve about the rig's own axis basis, idle micro-drift,
-  occasional look-away shifts with dwell timing, turn banking, camera lock, and
-  a deviation pull-back that lets the clip's own head motion flow until it strays.
+- **Gaze** — eyes, then neck, then body: the eyes take small changes, the head
+  follows smoothly beyond a few degrees (or shifts quickly, eye-led, on a large
+  jump), and the torso comes round only when the neck has been twisted far for a
+  moment. Added to the clip's pose, with its fast head sway steadied while
+  fixating, and brief glances away while listening.
 - **Breathing** — asymmetric waveform, chest leading the waist, composed onto
   whatever the mixer wrote.
 - **Blink** — asymmetric close/hold/reopen envelope, resting droop that wanders
   rather than loops, partial and double blinks, lids following downward gaze.
-- **Eye gaze** — morph-only on `convai-mha-1`, which has no eye bones.
+- **Eye gaze** — morph-only on `convai-mha-1`, which has no eye bones. Each eye
+  is aimed at the gaze point from the body's own measured eye shapes, with
+  micro-saccades while holding eye contact.
 - **`prefers-reduced-motion`** damps every additive layer and suppresses
   look-away shifts, while leaving lipsync and clip playback untouched.
 

package/dist/character/createCharacter.d.ts

@@ -7,7 +7,7 @@ import { type CharacterResolver, type CharacterBundle, type Representation, type
 import { type ModelVariantPreference } from '../resolve/modelVariant.js';
 import { type SessionOptions } from '../connect/session.js';
 import type { ConvaiClient } from '@convai/web-sdk/core';
-import { type UnitNormalizer } from '../rig/units.js';
+import { type UnitNormalizer, type Grounding } from '../rig/units.js';
 import { AnimationController } from '../clips/animations.js';
 import { LipsyncSystem, type MouthTuning } from '../speech/lipsync.js';
 import type { CorrectiveMap } from '../speech/correctiveMap.js';
@@ -19,8 +19,10 @@ import { GazeSystem, type GazeConfig } from '../motion/gaze.js';
 import { type BreathingConfig } from '../motion/breathe.js';
 import { type EyeGazeConfig } from '../motion/eyeGaze.js';
 import { type FaceMaterialTuning } from '../render/faceMaterials.js';
+import { type SurfaceTuning } from '../render/surfaces.js';
 import { type BlinkConfig } from '../motion/blink.js';
 import { type ReducedMotion } from '../motion/reducedMotion.js';
+import { type CharacterReport } from '../inspect/report.js';
 /**
  * Where the Draco decoder is fetched from. Google's versioned CDN, which is
  * what three's own examples use and is served with permissive CORS.
@@ -92,8 +94,17 @@ export interface CreateCharacterOptions {
      * Full's blendshapes; `full` and `lite` force one. See `pickModelVariant`.
      */
     modelVariant?: ModelVariantPreference;
+    /** Standing height the body is scaled to, metres. Measured once, at load. Default 1.7. */
     targetHeight?: number;
+    /** World height of the floor the soles rest on. Default 0. */
     groundY?: number;
+    /**
+     * `feet` (default): every frame the lowest point of either sole rests on
+     * `groundY`, whatever the clip does. `clip`: the body is grounded once at
+     * load and the clip's own root motion decides its height after that — for
+     * jumps and anything else that should leave the floor.
+     */
+    grounding?: Grounding;
     mouth?: MouthTuning;
     /**
      * A corrective map fitted from this body's DNA (`compileCorrectiveMap`).
@@ -102,7 +113,12 @@ export interface CreateCharacterOptions {
     correctiveMap?: CorrectiveMap | null;
     /** Tier-2 customization: run once after load, before first frame. */
     decorators?: CharacterDecorator[];
-    /** Extra local clips (name → url), merged OVER manifest clips by name. */
+    /**
+     * Extra local clips (name → url), merged OVER manifest clips by name. A GLB,
+     * glTF or FBX file; a clip authored on a Mixamo, Unreal mannequin or Rigify
+     * skeleton is retargeted onto the body at load. FBX support is loaded only
+     * when an `.fbx` URL is requested.
+     */
     animations?: Record<string, string>;
     /**
      * Which clip to play on load. Defaults to a clip named "idle"
@@ -115,6 +131,11 @@ export interface CreateCharacterOptions {
     gaze?: GazeConfig;
     /** Stage 6 eyelid/blink config. */
     blink?: BlinkConfig;
+    /**
+     * Recompute normals where morphs move a face that ships no morph normals
+     * (closed lids shaded black with their open-eye normals). Default `true`.
+     */
+    morphNormals?: boolean;
     /** Stage 3 additive chest/spine breathing config. */
     breathing?: BreathingConfig;
     /** Stage 6 morph-only eye-look config. */
@@ -136,6 +157,38 @@ export interface CreateCharacterOptions {
      * `Eyeocclusion*`).
      */
     faceMaterials?: boolean | FaceMaterialTuning;
+    /**
+     * Skin and hair shading, so the body takes the app's lighting the way skin
+     * and hair do. On by default: skin becomes physically based (with light
+     * wrapping past the shadow edge and pore detail on WebGL), hair and lashes
+     * draw with soft, depth-sorted edges, the scalp under the hair is shaded,
+     * and textures are sampled anisotropically. Lighting stays the app's.
+     *
+     * Pass `false` to keep every skin and hair material exactly as shipped, or a
+     * `SurfaceTuning` to adjust or switch off parts of it.
+     */
+    surfaces?: boolean | SurfaceTuning;
+    /**
+     * Hair. Long hair rests on the shoulders instead of swinging with the head
+     * as one shell (skin weights, every renderer), except where that would push
+     * it into the body. `false` leaves hair as shipped. Default on.
+     */
+    hair?: boolean;
+    /**
+     * Face shapes under other names. Lipsync, blink and eye gaze address
+     * MetaHuman expression channels (`jawOpen`, `eyeBlinkL`, …); this maps
+     * channel → the model's own morph name. `'auto'` (default) finds them by
+     * name: renamed or prefixed MetaHuman shapes, Character Creator's extended
+     * set, and the ARKit 52 by meaning. The map in effect is in
+     * `handle.report.face.morphMap`, ready to pass back here to pin or correct.
+     * A map you pass is used as it is; `false` uses none.
+     */
+    morphMap?: 'auto' | false | Record<string, string>;
+    /**
+     * Print `handle.report` to the console when the model has errors or
+     * warnings: what it cannot do, and the config that fixes it. Default true.
+     */
+    diagnostics?: boolean;
 }
 export interface CharacterHandle {
     root: THREE.Object3D;
@@ -160,6 +213,11 @@ export interface CharacterHandle {
     client: ConvaiClient | null;
     /** Set by the app once its camera exists; gaze tracking no-ops without one. */
     camera?: THREE.Camera;
+    /**
+     * What this model can and cannot do with embody, read at load: face
+     * shapes found or mapped, skeleton, clips, and each problem with its fix.
+     */
+    report: CharacterReport;
     /**
      * Tick every frame. The lipsync queue comes from this character's own
      * session, so a single-character app passes nothing.
@@ -185,11 +243,13 @@ export interface CharacterHandle {
  * then blink — see the stage-6 comment below and `pipeline.ts`'s header.
  */
 export declare function buildPipeline(deps: {
-    units: Pick<UnitNormalizer, 'update'>;
+    units: Pick<UnitNormalizer, 'ground'>;
     animations: Pick<AnimationController, 'update'>;
     lipsync: Pick<LipsyncSystem, 'update'>;
     rig: ResolvedRig;
-}, opts: Pick<CreateCharacterOptions, 'motion' | 'gaze' | 'breathing' | 'eyeGaze' | 'blink'>): {
+    /** The body, for systems that read its meshes (morph normals). */
+    root?: THREE.Object3D;
+}, opts: Pick<CreateCharacterOptions, 'motion' | 'gaze' | 'breathing' | 'eyeGaze' | 'blink' | 'morphNormals'>): {
     pipeline: Pipeline;
     gaze: GazeSystem;
 };
@@ -202,7 +262,8 @@ export declare function buildPipeline(deps: {
  * `requested` is `CreateCharacterOptions.idleClip`: a name to insist on,
  * `false` to start paused, or undefined for the default. Clip names come from
  * the manifest and are not guaranteed, so the default prefers a clip actually
- * called idle and otherwise falls back to whatever the character shipped
+ * called idle, then one whose name contains it (`Idle_Motion`,
+ * `Armature|Idle`), and otherwise falls back to whatever the character shipped
  * first — anything is better than the bind pose.
  *
  * A requested name that the character does not have returns null rather than

package/dist/character/createCharacter.js

@@ -5,6 +5,7 @@ import { pickModelVariant } from '../resolve/modelVariant.js';
 import { createSession, adoptSession } from '../connect/session.js';
 import { createUnitNormalizer } from '../rig/units.js';
 import { AnimationController } from '../clips/animations.js';
+import { loadClipFile, gatherClips } from '../clips/sources.js';
 import { LipsyncSystem } from '../speech/lipsync.js';
 import { collectMaterials } from '../decorate/types.js';
 import { Pipeline } from './pipeline.js';
@@ -14,8 +15,14 @@ import { GazeSystem } from '../motion/gaze.js';
 import { BreathingSystem } from '../motion/breathe.js';
 import { EyeGazeSystem } from '../motion/eyeGaze.js';
 import { applyFaceMaterials } from '../render/faceMaterials.js';
+import { applySurfaces, shadeScalp, shadeHairDepth } from '../render/surfaces.js';
 import { BlinkSystem } from '../motion/blink.js';
+import { MorphNormals } from '../motion/morphNormals.js';
 import { createReducedMotion } from '../motion/reducedMotion.js';
+import { restHairOnShoulders } from '../motion/hair.js';
+import { VoiceClock } from '../speech/voiceClock.js';
+import { applyMorphMap, matchMorphs, morphNames } from '../inspect/morphMap.js';
+import { inspectCharacter, formatReport } from '../inspect/report.js';
 /**
  * Where the Draco decoder is fetched from. Google's versioned CDN, which is
  * what three's own examples use and is served with permissive CORS.
@@ -34,15 +41,19 @@ export const DEFAULT_DRACO_DECODER_PATH = 'https://www.gstatic.com/draco/version
  * then blink — see the stage-6 comment below and `pipeline.ts`'s header.
  */
 export function buildPipeline(deps, opts) {
-    const { units, animations, lipsync } = deps;
+    const { units, animations, lipsync, root } = deps;
     const pipeline = new Pipeline();
-    // Stage 0 — root scale must settle before anything reads world space.
-    pipeline.add({ stage: 0, update: () => void units.update() });
-    // Stage 1 — clips drive the skeleton.
+    // Stage 0 has no per-frame work: scale is measured once, at load, by
+    // createUnitNormalizer. Stage 1 — clips drive the skeleton, then the pose
+    // they produced is put on the ground (registration order is the contract:
+    // grounding must read this frame's pose, not last frame's).
     pipeline.add({ stage: 1, update: (dt) => animations.update(dt) });
+    pipeline.add({ stage: 1, update: () => units.ground() });
     const gaze = new GazeSystem(opts.gaze);
     if (opts.motion !== false) {
-        pipeline.add(gaze); // 2 — overrides mixer
+        // Stage 0 — undo last frame's turns on bones the mixer will not rewrite.
+        pipeline.add({ stage: 0, update: () => gaze.beforeMixer() });
+        pipeline.add(gaze); // 2 — added to the clip pose
         pipeline.add(new BreathingSystem(opts.breathing)); // 3 — additive
     }
     // Stage 6 — ORDER IS THE CONTRACT: lipsync, then eye gaze, then blink.
@@ -51,6 +62,9 @@ export function buildPipeline(deps, opts) {
         pipeline.add(new EyeGazeSystem(gaze, opts.eyeGaze));
         pipeline.add(new BlinkSystem(opts.blink, gaze));
     }
+    // Last in stage 6, after every morph writer: normals for what they moved.
+    if (opts.morphNormals !== false && root)
+        pipeline.add(new MorphNormals(root));
     return { pipeline, gaze };
 }
 /**
@@ -62,7 +76,8 @@ export function buildPipeline(deps, opts) {
  * `requested` is `CreateCharacterOptions.idleClip`: a name to insist on,
  * `false` to start paused, or undefined for the default. Clip names come from
  * the manifest and are not guaranteed, so the default prefers a clip actually
- * called idle and otherwise falls back to whatever the character shipped
+ * called idle, then one whose name contains it (`Idle_Motion`,
+ * `Armature|Idle`), and otherwise falls back to whatever the character shipped
  * first — anything is better than the bind pose.
  *
  * A requested name that the character does not have returns null rather than
@@ -73,7 +88,7 @@ export function pickIdleClip(names, requested) {
         return null;
     if (requested)
         return names.includes(requested) ? requested : null;
-    return names.find((n) => n.toLowerCase() === 'idle') ?? names[0] ?? null;
+    return names.find((n) => n.toLowerCase() === 'idle') ?? names.find((n) => /idle/i.test(n)) ?? names[0] ?? null;
 }
 /** Every material property that can hold a disposable `THREE.Texture`. */
 const TEXTURE_PROPS = [
@@ -109,6 +124,10 @@ export function disposeCharacter(deps) {
         if (!mesh.isMesh)
             return;
         mesh.geometry?.dispose();
+        // Shadow-only materials (surfaces gives hair its own); their maps are
+        // the drawing material's, freed below.
+        mesh.customDepthMaterial?.dispose();
+        mesh.customDistanceMaterial?.dispose();
         const materials = Array.isArray(mesh.material) ? mesh.material : [mesh.material];
         for (const mat of materials) {
             if (!mat)
@@ -184,12 +203,18 @@ export async function createCharacter(opts) {
     for (const [name, url] of Object.entries(opts.animations ?? {}))
         clipUrls.set(name, url);
     const uniqueUrls = [...new Set(clipUrls.values())];
-    let modelGltf, clipGltfs;
+    const clipErrors = new Map();
+    let modelGltf, clipFiles;
     try {
         ;
-        [modelGltf, ...clipGltfs] = await Promise.all([
+        [modelGltf, clipFiles] = await Promise.all([
             loader.loadAsync(modelUrl),
-            ...uniqueUrls.map((u) => loader.loadAsync(u)),
+            // A clip file that fails to load costs that clip, not the character:
+            // the report names it.
+            Promise.all(uniqueUrls.map((u) => loadClipFile(u, loader).catch((e) => {
+                clipErrors.set(u, e.message);
+                return null;
+            }))),
         ]);
     }
     finally {
@@ -205,15 +230,29 @@ export async function createCharacter(opts) {
     if (opts.faceMaterials !== false) {
         applyFaceMaterials(root, opts.faceMaterials === true || opts.faceMaterials === undefined ? {} : opts.faceMaterials, modelGltf.parser);
     }
-    // gather every clip whose name the manifest/overrides declared
-    const wanted = new Set(clipUrls.keys());
-    const clips = [];
-    for (const g of clipGltfs)
-        for (const a of g.animations)
-            if (wanted.has(a.name))
-                clips.push(a);
+    // The record's gender reaches the surfaces (male lashes are capped), unless
+    // the app set it.
+    const surfaces = opts.surfaces === false ? null : { gender: bundle.gender, ...(opts.surfaces === true || !opts.surfaces ? {} : opts.surfaces) };
+    if (surfaces)
+        applySurfaces(root, surfaces);
+    // Every clip the manifest/overrides declared. Clips from another skeleton
+    // (Mixamo, Rigify) are retargeted onto the body here, from its bind pose.
+    const files = new Map(uniqueUrls.flatMap((u, i) => (clipFiles[i] ? [[u, clipFiles[i]]] : [])));
+    const clipNotes = [];
+    let clips = gatherClips(root, files, clipUrls, {}, clipNotes);
+    for (const n of clipNotes)
+        if (n.status === 'not-loaded')
+            n.detail = clipErrors.get(n.url);
+    // A model file that brings its own animations, and nothing declared: use them.
+    let embedded = 0;
+    if (clipUrls.size === 0 && modelGltf.animations.length) {
+        clips = modelGltf.animations;
+        embedded = clips.length;
+        for (const c of clips)
+            clipNotes.push({ name: c.name, url: modelUrl, status: 'as-is' });
+    }
     const animations = new AnimationController(root);
-    animations.register(clips);
+    const registered = animations.register(clips);
     // Start idling. Without this a zero-config character stands in its bind
     // pose — arms out, visibly wrong — until the app happens to call play().
     // Making the caller do that is exactly the kind of required extra step
@@ -225,7 +264,16 @@ export async function createCharacter(opts) {
     const idle = pickIdleClip(animations.names, opts.idleClip);
     if (idle)
         animations.play(idle);
-    const lipsync = new LipsyncSystem(opts.mouth, opts.correctiveMap ?? null);
+    // Face shapes under other names, aliased before anything binds by name.
+    const match = matchMorphs(root);
+    const wantMap = opts.morphMap ?? 'auto';
+    const morphMap = wantMap === 'auto' ? match.map : wantMap === false ? {} : wantMap;
+    const known = morphNames(root);
+    const unknownMorphs = wantMap && wantMap !== 'auto' ? Object.entries(wantMap).filter(([, m]) => !known.has(m)).map(([c, m]) => `${c} → ${m}`) : [];
+    const applied = applyMorphMap(root, morphMap);
+    // A male record rests steady rather than in the light smile, unless the app chose a mood.
+    const mouth = opts.mouth?.expression === undefined && /^male$/i.test(bundle.gender ?? '') ? { ...opts.mouth, expression: 'steady' } : opts.mouth;
+    const lipsync = new LipsyncSystem(mouth, opts.correctiveMap ?? null);
     lipsync.bind(root);
     for (const d of opts.decorators ?? []) {
         d({ root, bundle, representation: rep, materials: collectMaterials(root) });
@@ -233,10 +281,29 @@ export async function createCharacter(opts) {
     const units = createUnitNormalizer(root, {
         targetHeight: opts.targetHeight,
         groundY: opts.groundY,
+        grounding: opts.grounding,
+    });
+    // Needs the body in metres and still in its bind pose: after the units
+    // pass, before the first update. (Resting hair checks itself against the
+    // body in metres.)
+    if (opts.hair !== false)
+        restHairOnShoulders(root);
+    if (surfaces && surfaces.scalp !== false && surfaces.hair !== false)
+        shadeScalp(root);
+    if (surfaces && surfaces.hair !== false)
+        shadeHairDepth(root);
+    const rig = resolveRig(root, CONVAI_MHA_1, { quiet: true });
+    const report = inspectCharacter({
+        root, rig, match,
+        morphMap: { map: Object.fromEntries(applied.map((c) => [c, morphMap[c] ?? morphMap[`CTRL_expressions_${c}`]])), source: wantMap === 'auto' ? 'auto' : wantMap === false ? 'off' : 'app', unknown: unknownMorphs },
+        clipNotes, registered, idle, embedded,
+        idleRequested: typeof opts.idleClip === 'string' ? opts.idleClip : undefined,
     });
-    const rig = resolveRig(root, CONVAI_MHA_1);
+    if (opts.diagnostics !== false && report.issues.some((i) => i.level !== 'info')) {
+        console.warn(formatReport(report, `model ${model.url.split(/[?#]/, 1)[0].split('/').pop()}`));
+    }
     const reduced = createReducedMotion();
-    const { pipeline, gaze } = buildPipeline({ units, animations, lipsync, rig }, { motion: opts.motion, gaze: opts.gaze, breathing: opts.breathing, eyeGaze: opts.eyeGaze, blink: opts.blink });
+    const { pipeline, gaze } = buildPipeline({ units, animations, lipsync, rig, root }, { motion: opts.motion, gaze: opts.gaze, breathing: opts.breathing, eyeGaze: opts.eyeGaze, blink: opts.blink });
     // Reused every frame — mutated in place rather than reallocated, so the
     // per-frame path makes zero steady-state allocations for the frame context.
     const frameCtx = {
@@ -255,8 +322,12 @@ export async function createCharacter(opts) {
         disposeCharacter({ root, mixer: animations.mixer, reducedMotion: reduced });
         throw sessionError;
     }
+    // The mouth follows the voice as it actually plays, read from the session's
+    // received audio (live: it changes on reconnect).
+    if (session)
+        lipsync.voice = new VoiceClock(() => session.voiceReceiver());
     const handle = {
-        root, bundle, representation: rep, model, animations, lipsync, rig, pipeline, gaze,
+        root, bundle, representation: rep, model, animations, lipsync, rig, pipeline, gaze, report,
         client: session?.client ?? null,
         camera: undefined,
         update(delta, queue) {

package/dist/character/pipeline.d.ts

@@ -3,13 +3,17 @@
  *
  * Order is load-bearing and each constraint below is inherited from the
  * prior art, not invented here:
- *   0 normalize — root scale must settle before anything reads world space
- *   1 mixer     — clips drive the skeleton
- *   2 gaze      — OVERRIDES the mixer's head rotation, never blends with it
+ *   0 prepare   — systems that ADD to the clip pose undo last frame's
+ *                 additions on bones the mixer will not rewrite (root scale is
+ *                 measured once at load, not here)
+ *   1 mixer     — clips drive the skeleton, then grounding puts that pose's
+ *                 soles on the floor (in that registration order)
+ *   2 gaze      — eyes-then-neck-then-torso turns, ADDED to the clip pose
  *   3 additive  — breathe/lean/gestures, added AFTER the mixer so nothing
  *                 accumulates (the mixer rewrites every frame)
- *   4 ik        — foot lock (reserved; no system in this slice)
- *   5 attach    — hair follows the final transform (reserved)
+ *   4 ik        — reserved (foot lock was measured and not needed: the clips'
+ *                 own root motion keeps planted feet within ~0.3 cm/s)
+ *   5 attach    — reserved
  *   6 morph     — lipsync, eye-look, blink. Blink must run after lipsync.
  */
 import type * as THREE from 'three';

package/dist/clips/animations.d.ts

@@ -7,6 +7,13 @@
  * which names those are via their own mapping. Loaded clips are track-filtered
  * against the bones this model actually has — embody's dev-time validation:
  * the clip GLBs carry richer rigs (fingers, hair) than some models.
+ *
+ * A clip also never moves, turns or scales the model's own container nodes
+ * (anything that is not a bone: the export's `root`, an armature group). Those
+ * transforms are the model's units and axes, not motion. Every published
+ * convai-mha-1 clip carries `root.scale = 0.01` while the bodies bind `root`
+ * at 1 with centimetre bones, so playing a clip used to shrink the body 100x,
+ * hidden only while the unit normaliser re-measured every frame.
  */
 import * as THREE from 'three';
 export interface ClipTransition {
@@ -24,7 +31,12 @@ export declare class AnimationController {
     private current;
     private pendingReturn;
     constructor(root: THREE.Object3D);
-    /** Register clips, filtering tracks to bones present on this model. */
+    /**
+     * Register clips, keeping only tracks that animate this model: transforms of
+     * its bones, and any property (morph weights) of other nodes it has.
+     * Transforms of non-bone container nodes are dropped silently — see the
+     * header. Tracks naming nodes the model lacks are dropped with a warning.
+     */
     register(clips: THREE.AnimationClip[]): {
         name: string;
         kept: number;

package/dist/clips/animations.js

@@ -7,6 +7,13 @@
  * which names those are via their own mapping. Loaded clips are track-filtered
  * against the bones this model actually has — embody's dev-time validation:
  * the clip GLBs carry richer rigs (fingers, hair) than some models.
+ *
+ * A clip also never moves, turns or scales the model's own container nodes
+ * (anything that is not a bone: the export's `root`, an armature group). Those
+ * transforms are the model's units and axes, not motion. Every published
+ * convai-mha-1 clip carries `root.scale = 0.01` while the bodies bind `root`
+ * at 1 with centimetre bones, so playing a clip used to shrink the body 100x,
+ * hidden only while the unit normaliser re-measured every frame.
  */
 import * as THREE from 'three';
 export class AnimationController {
@@ -24,17 +31,32 @@ export class AnimationController {
             }
         });
     }
-    /** Register clips, filtering tracks to bones present on this model. */
+    /**
+     * Register clips, keeping only tracks that animate this model: transforms of
+     * its bones, and any property (morph weights) of other nodes it has.
+     * Transforms of non-bone container nodes are dropped silently — see the
+     * header. Tracks naming nodes the model lacks are dropped with a warning.
+     */
     register(clips) {
-        const bones = new Set();
-        this.root.traverse((o) => bones.add(o.name));
+        const nodes = new Map();
+        this.root.traverse((o) => { if (!nodes.has(o.name))
+            nodes.set(o.name, o); });
         const report = [];
         for (const src of clips) {
-            const kept = src.tracks.filter((t) => bones.has(t.name.split('.')[0]));
+            let absent = 0;
+            const kept = src.tracks.filter((t) => {
+                const { nodeName, propertyName } = THREE.PropertyBinding.parseTrackName(t.name);
+                const node = nodes.get(nodeName);
+                if (!node)
+                    return void absent++, false;
+                const transform = propertyName === 'position' || propertyName === 'quaternion' || propertyName === 'scale';
+                return !transform || node.isBone === true;
+            });
             const dropped = src.tracks.length - kept.length;
-            if (dropped > 0)
-                console.warn(`[embody] clip "${src.name}": ${dropped}/${src.tracks.length} tracks target absent bones — dropped`);
+            if (absent > 0)
+                console.warn(`[embody] clip "${src.name}": ${absent}/${src.tracks.length} tracks target absent bones — dropped`);
             const clip = new THREE.AnimationClip(src.name, src.duration, kept);
+            clip.userData = src.userData;
             this.actions.set(src.name, this.mixer.clipAction(clip));
             report.push({ name: src.name, kept: kept.length, dropped });
         }