@convai/embody 0.4.1 → 0.5.0

package/CHANGELOG.md

@@ -1,5 +1,11 @@
 # @convai/embody
 
+## 0.5.0
+
+### Minor Changes
+
+- 249a96f: Optional corrective map for one-morph-per-control bodies. A table fitted from the body's MetaHuman DNA re-weights each lipsync frame toward the full rig (PSD correctives, joint arcs). Pass `correctiveMap` (from `compileCorrectiveMap`) to `createCharacter` or the React components; it is off by default and swaps live through the prop or `character.lipsync.correctiveMap`.
+
 ## 0.4.1
 
 ### Patch Changes

package/README.md

@@ -22,13 +22,18 @@ lipsync, gaze, breathing and blink. Layered on
 ## Install
 
 ```bash
-pnpm add @convai/embody three
+pnpm add @convai/embody @convai/web-sdk three
 # React consumers also need:
 pnpm add @react-three/fiber react
 ```
 
-`three` and `@convai/web-sdk` are peer dependencies. `react` and
-`@react-three/fiber` are optional peers — the `/core` path needs neither.
+`@convai/web-sdk` and `three` are peer dependencies. `react` and
+`@react-three/fiber` are optional peers — the root `@convai/embody` entry
+point needs neither. Node.js 18 or later.
+
+A character renders only if its avatar has a published web body, or a default
+body applies to it. Otherwise loading fails with *"Character has no 3D
+embodiment configured"* — a data problem, not a code one.
 
 ## Quick start
 
@@ -173,10 +178,33 @@ The defaults cover the normal case; each of these exists for a specific one.
 | `session` | Extra SDK options (voice, memory, transport, `endUserId`). Credentials and `characterId` are not repeated here. |
 | `baseUrl` | A different backend environment. Defaults per release channel. |
 | `queue` on `update()` | One client driving several bodies, when you are deciding which owns the current turn. |
+| `correctiveMap` | A table fitted from the body's MetaHuman DNA (`compileCorrectiveMap(json)`) that re-weights each lipsync frame toward what the full rig shows: PSD correctives and joint arcs that one-morph-per-control bodies lack. Off by default. Changing the prop, or `character.lipsync.correctiveMap`, swaps it live. In `pnpm dev`, the demo lists `*.map.json` tables from `../open-rig-dna/maps` (or `$EMBODY_CORRECTIVE_MAPS`) for A/B: **C** toggles, **F** frames the face. |
 
-## Docs
+## Documentation
 
-Design, plan and decision history live in [`docs/`](./docs):
+**User documentation** is published on docs.convai.com under *Web Plugins →
+Convai Embody*. Its source is [`docs/site/`](./docs/site) in this repository:
+
+| | |
+|---|---|
+| [Add a 3D character to a React app](./docs/site/convai-embody/add-a-3d-character-to-a-react-app.md) | install, render, verify, troubleshoot |
+| [Add a 3D character to a three.js scene](./docs/site/convai-embody/add-a-3d-character-to-a-three-js-scene.md) | the framework-free API |
+| [Add a chat widget to a character](./docs/site/convai-embody/add-a-chat-widget-to-a-character.md) | one connection for body and chat |
+| [React API reference](./docs/site/convai-embody/react-api-reference.md) | every prop and default |
+| [createCharacter reference](./docs/site/convai-embody/createcharacter-reference.md) | every option, the handle, the errors |
+| [How Convai Embody works](./docs/site/convai-embody/how-convai-embody-works.md) | resolution, the frame pipeline, environments |
+
+Edit those pages here, not in the documentation repository. **Docs are part of every change:** CI runs the same sync into a throwaway checkout of Convai-Documentation and lints every page with that repository's `check_page.py`, so a page that would fail the docs gate fails the PR. Section path, version variable and sidebar placement live in [`docs/site/docs.config.json`](./docs/site/docs.config.json); page order, including nested groups, in [`docs/site/sidebar.json`](./docs/site/sidebar.json). Every publish to
+npm `latest` runs [`docs-sync.yml`](./.github/workflows/docs-sync.yml), which
+opens a pull request against `Conv-AI/Convai-Documentation` (`staging`) with
+the pages, their sidebar entries, `last_reviewed`, and the `embody_version`
+GitBook variable. Merging that pull request publishes to the GitBook **Staging**
+Space for review; the pages reach docs.convai.com only when a human then merges
+`staging` into `main` in that repository. Run the workflow by hand for a
+docs-only fix. The pages follow that repository's authoring rules, recorded for
+this package in [`docs/site/pack.md`](./docs/site/pack.md).
+
+**Design, plan and decision history** live in [`docs/`](./docs):
 
 | | |
 |---|---|
@@ -194,13 +222,19 @@ The backend host is a source constant, `DEFAULT_BASE_URL` in
 
 | Branch | Host | npm tag |
 |---|---|---|
-| `stable-release` | `https://api.convai.com` | `latest` |
-| `beta-release` | `https://api.convai.com` | `beta` |
-| `stg-release` | `https://api-stg.convai.com` | `stg` |
+| `main` / `prod` | `https://api.convai.com` | `latest` |
+| `beta` | `https://api.convai.com` | `beta` |
+| `stg` | `https://api-stg.convai.com` | `stg` |
 
 Beta points at **production** — it is the first line of testing against real
-infrastructure, not a staging channel. Only `stg-release` targets staging,
-which has its own character database.
+infrastructure, not a staging channel. Only `stg` targets staging, which has
+its own character database. Character IDs belong to one environment: a staging
+ID against production returns a not-found error that reads like a permissions
+problem.
+
+Promotion has a silent failure mode: merging `stg` into `main` carries the
+staging `DEFAULT_BASE_URL` across with no conflict, because only `stg` ever
+edits that line. `pnpm check:hosts` is what catches it.
 
 `scripts/check-hosts.mjs` enforces this. Each channel build asserts the policy
 for the release it claims to be, so a staging artifact cannot be cut from
@@ -217,7 +251,7 @@ pnpm check:hosts   # auto-detects the current branch
 
 ```bash
 pnpm install
-pnpm test         # 183 tests
+pnpm test         # vitest
 pnpm typecheck
 pnpm check:tests  # every src/ module has a named test — see docs/testing-spec.md
 pnpm build        # emits dist/ with .d.ts

package/dist/character/createCharacter.d.ts

@@ -9,6 +9,7 @@ import type { ConvaiClient } from '@convai/web-sdk/core';
 import { type UnitNormalizer } from '../rig/units.js';
 import { AnimationController } from '../clips/animations.js';
 import { LipsyncSystem, type MouthTuning } from '../speech/lipsync.js';
+import type { CorrectiveMap } from '../speech/correctiveMap.js';
 import { type CharacterDecorator } from '../decorate/types.js';
 import type { BlendshapeQueue } from '@convai/web-sdk/core';
 import { Pipeline } from './pipeline.js';
@@ -87,6 +88,11 @@ export interface CreateCharacterOptions {
     targetHeight?: number;
     groundY?: number;
     mouth?: MouthTuning;
+    /**
+     * A corrective map fitted from this body's DNA (`compileCorrectiveMap`).
+     * Off by default. Swap it later through `handle.lipsync.correctiveMap`.
+     */
+    correctiveMap?: CorrectiveMap | null;
     /** Tier-2 customization: run once after load, before first frame. */
     decorators?: CharacterDecorator[];
     /** Extra local clips (name → url), merged OVER manifest clips by name. */

package/dist/character/createCharacter.js

@@ -223,7 +223,7 @@ export async function createCharacter(opts) {
     const idle = pickIdleClip(animations.names, opts.idleClip);
     if (idle)
         animations.play(idle);
-    const lipsync = new LipsyncSystem(opts.mouth);
+    const lipsync = new LipsyncSystem(opts.mouth, opts.correctiveMap ?? null);
     lipsync.bind(root);
     for (const d of opts.decorators ?? []) {
         d({ root, bundle, representation: rep, materials: collectMaterials(root) });

package/dist/index.d.ts

@@ -28,6 +28,8 @@ export { AnimationController } from './clips/animations.js';
 export type { ClipTransition } from './clips/animations.js';
 export { LipsyncSystem } from './speech/lipsync.js';
 export type { MouthTuning } from './speech/lipsync.js';
+export { compileCorrectiveMap, applyCorrectiveMap, CORRECTIVE_MAP_FORMAT } from './speech/correctiveMap.js';
+export type { CorrectiveMap, CorrectiveMapTable } from './speech/correctiveMap.js';
 export { createUnitNormalizer } from './rig/units.js';
 export { collectMaterials } from './decorate/types.js';
 export type { CharacterDecorator, DecoratorContext } from './decorate/types.js';

package/dist/index.js

@@ -25,6 +25,7 @@ export { httpResolver, pickRepresentation, DEFAULT_BASE_URL } from './resolve/re
 export { createCharacter } from './character/createCharacter.js';
 export { AnimationController } from './clips/animations.js';
 export { LipsyncSystem } from './speech/lipsync.js';
+export { compileCorrectiveMap, applyCorrectiveMap, CORRECTIVE_MAP_FORMAT } from './speech/correctiveMap.js';
 export { createUnitNormalizer } from './rig/units.js';
 export { collectMaterials } from './decorate/types.js';
 export { Pipeline } from './character/pipeline.js';

package/dist/react/useCharacter.js

@@ -44,6 +44,13 @@ export function useCharacter(opts) {
         if (character)
             character.camera = camera;
     }, [character, camera]);
+    // Live, unlike the props above: swapping the map must not reload the body,
+    // or an A/B on live speech would drop the turn it is comparing.
+    const correctiveMap = opts.correctiveMap ?? null;
+    useEffect(() => {
+        if (character)
+            character.lipsync.correctiveMap = correctiveMap;
+    }, [character, correctiveMap]);
     useFrame((_, delta) => character?.update(delta, queueRef.current));
     return {
         character,

package/dist/speech/correctiveMap.d.ts

@@ -0,0 +1,75 @@
+/**
+ * Corrective map — re-weights Convai's 251 face controls so that a body with
+ * one baked morph per control lands closer to what the full MetaHuman rig
+ * shows.
+ *
+ * RigLogic fires correctives (PSDs) when controls combine, and its face joints
+ * rotate, so the skin moves along arcs; a sum of straight-line morphs follows
+ * neither. A table fitted from the character's DNA (convai-embody-console's
+ * tools/corrective-mapping/fit_mapping.py) corrects each frame before the
+ * mouth layer sees it:
+ *
+ *     w = x + clamp(A · φ(x), ±limit)
+ *
+ * φ(x) is one activation per corrective (Π clamp01(x_i), weighted and capped
+ * at 1, as RigLogic computes it) and, in version 2 tables, each x_i² and each
+ * x_i · jawOpen. The table holds no geometry: feature definitions and
+ * coefficients only. Format and maths match the console's
+ * corrective-mapping.mjs, so a table fitted there loads here unchanged.
+ */
+export declare const CORRECTIVE_MAP_FORMAT = "convai-corrective-map";
+/** One corrective's inputs (channel indices) and per-input weights. */
+interface PsdDef {
+    inputs: number[];
+    weights: number[];
+}
+/** The JSON fit_mapping.py writes. */
+export type CorrectiveMapTable = {
+    format: typeof CORRECTIVE_MAP_FORMAT;
+    version: 1;
+    channels: number;
+    /** One row per corrective: its definition and the channels it corrects. */
+    rows: (PsdDef & {
+        a: [number, number][];
+    })[];
+} | {
+    format: typeof CORRECTIVE_MAP_FORMAT;
+    version: 2;
+    channels: number;
+    features: {
+        psd: PsdDef[];
+        squares?: boolean;
+        jaw?: number;
+    };
+    /** [feature index, [[channel, coefficient], …]] */
+    coefficients: [number, [number, number][]][];
+    limit?: number;
+};
+interface Column {
+    targets: Int16Array;
+    coeffs: Float32Array;
+}
+/** A validated table packed into typed arrays for the per-frame loop. */
+export interface CorrectiveMap {
+    readonly channels: number;
+    readonly psd: {
+        inputs: Int16Array;
+        weights: Float32Array;
+    }[];
+    readonly squares: boolean;
+    /** jawOpen's channel index, or -1 when the table has no jaw features. */
+    readonly jaw: number;
+    readonly columns: (Column | null)[];
+    readonly limit: number;
+    /** Scratch for the per-frame delta; one map serves one frame at a time. */
+    readonly delta: Float32Array;
+}
+/** Validate a table and pack it. Throws on anything malformed, at load rather than mid-speech. */
+export declare function compileCorrectiveMap(table: CorrectiveMapTable): CorrectiveMap;
+/**
+ * Map one frame into `out` (allocated when omitted) and return it. `frame` is
+ * not modified. A frame of the wrong width is returned unmapped: a table
+ * fitted for another channel layout must not scribble over this one.
+ */
+export declare function applyCorrectiveMap(frame: Float32Array, map: CorrectiveMap, out?: Float32Array): Float32Array;
+export {};

package/dist/speech/correctiveMap.js

@@ -0,0 +1,125 @@
+/**
+ * Corrective map — re-weights Convai's 251 face controls so that a body with
+ * one baked morph per control lands closer to what the full MetaHuman rig
+ * shows.
+ *
+ * RigLogic fires correctives (PSDs) when controls combine, and its face joints
+ * rotate, so the skin moves along arcs; a sum of straight-line morphs follows
+ * neither. A table fitted from the character's DNA (convai-embody-console's
+ * tools/corrective-mapping/fit_mapping.py) corrects each frame before the
+ * mouth layer sees it:
+ *
+ *     w = x + clamp(A · φ(x), ±limit)
+ *
+ * φ(x) is one activation per corrective (Π clamp01(x_i), weighted and capped
+ * at 1, as RigLogic computes it) and, in version 2 tables, each x_i² and each
+ * x_i · jawOpen. The table holds no geometry: feature definitions and
+ * coefficients only. Format and maths match the console's
+ * corrective-mapping.mjs, so a table fitted there loads here unchanged.
+ */
+export const CORRECTIVE_MAP_FORMAT = 'convai-corrective-map';
+const clamp01 = (v) => (v > 0 ? (v < 1 ? v : 1) : 0);
+function checkIndex(i, channels, what) {
+    if (!Number.isInteger(i) || i < 0 || i >= channels)
+        throw new Error(`${what} ${i} is outside 0..${channels - 1}.`);
+}
+function compilePsd(defs, channels) {
+    return defs.map((d) => {
+        d.inputs.forEach((i) => checkIndex(i, channels, 'A corrective input'));
+        return { inputs: Int16Array.from(d.inputs), weights: Float32Array.from(d.weights) };
+    });
+}
+function column(entries, channels) {
+    entries.forEach(([i]) => checkIndex(i, channels, 'A target channel'));
+    return { targets: Int16Array.from(entries.map(([i]) => i)), coeffs: Float32Array.from(entries.map(([, c]) => c)) };
+}
+/** Validate a table and pack it. Throws on anything malformed, at load rather than mid-speech. */
+export function compileCorrectiveMap(table) {
+    if (!table || table.format !== CORRECTIVE_MAP_FORMAT)
+        throw new Error(`Not a ${CORRECTIVE_MAP_FORMAT} table.`);
+    const channels = table.channels;
+    const delta = new Float32Array(channels);
+    if (table.version === 1) {
+        const psd = compilePsd(table.rows, channels);
+        const columns = table.rows.map((row) => column(row.a, channels));
+        return { channels, psd, squares: false, jaw: -1, columns, limit: Infinity, delta };
+    }
+    if (table.version === 2) {
+        const f = table.features;
+        const psd = compilePsd(f.psd, channels);
+        const jaw = f.jaw ?? -1;
+        if (jaw >= 0)
+            checkIndex(jaw, channels, 'The jaw channel');
+        const size = psd.length + (f.squares ? channels : 0) + (jaw >= 0 ? channels : 0);
+        const columns = Array.from({ length: size }, () => null);
+        for (const [feature, entries] of table.coefficients) {
+            checkIndex(feature, size, 'Feature');
+            columns[feature] = column(entries, channels);
+        }
+        return { channels, psd, squares: Boolean(f.squares), jaw, columns, limit: table.limit ?? Infinity, delta };
+    }
+    throw new Error(`Unsupported ${CORRECTIVE_MAP_FORMAT} version ${table.version}.`);
+}
+function addColumn(delta, col, value) {
+    if (!col)
+        return;
+    for (let k = 0; k < col.targets.length; k++)
+        delta[col.targets[k]] += value * col.coeffs[k];
+}
+/**
+ * Map one frame into `out` (allocated when omitted) and return it. `frame` is
+ * not modified. A frame of the wrong width is returned unmapped: a table
+ * fitted for another channel layout must not scribble over this one.
+ */
+export function applyCorrectiveMap(frame, map, out = new Float32Array(frame.length)) {
+    const n = map.channels;
+    if (frame.length !== n) {
+        out.set(frame);
+        return out;
+    }
+    const delta = map.delta;
+    delta.fill(0);
+    let f = 0;
+    for (const psd of map.psd) {
+        let prod = 1;
+        for (let k = 0; k < psd.inputs.length; k++) {
+            const v = frame[psd.inputs[k]];
+            if (!(v > 0)) {
+                prod = 0;
+                break;
+            }
+            prod *= v < 1 ? v : 1;
+        }
+        if (prod !== 0) {
+            let p = 0;
+            for (let k = 0; k < psd.weights.length; k++)
+                p += Math.min(1, psd.weights[k] * prod);
+            addColumn(delta, map.columns[f], p);
+        }
+        f++;
+    }
+    if (map.squares) {
+        for (let i = 0; i < n; i++) {
+            const v = clamp01(frame[i]);
+            if (v !== 0)
+                addColumn(delta, map.columns[f + i], v * v);
+        }
+        f += n;
+    }
+    if (map.jaw >= 0) {
+        const jaw = clamp01(frame[map.jaw]);
+        if (jaw !== 0) {
+            for (let i = 0; i < n; i++) {
+                const v = clamp01(frame[i]);
+                if (v !== 0)
+                    addColumn(delta, map.columns[f + i], v * jaw);
+            }
+        }
+    }
+    const limit = map.limit;
+    for (let i = 0; i < n; i++) {
+        const d = delta[i] > limit ? limit : delta[i] < -limit ? -limit : delta[i];
+        out[i] = frame[i] + d;
+    }
+    return out;
+}

package/dist/speech/lipsync.d.ts

@@ -15,6 +15,7 @@
  */
 import type * as THREE from 'three';
 import type { BlendshapeQueue } from '@convai/web-sdk/core';
+import { type CorrectiveMap } from './correctiveMap.js';
 /** Tier-1 customization: the mouth knobs, per character. Defaults = pagoda-act's tuned values. */
 export interface MouthTuning {
     /** Hard cap on jawOpen; peak excursions read exaggerated on camera. */
@@ -46,10 +47,18 @@ export declare class LipsyncSystem {
     energy: number;
     /** True while the bot owns the mouth; body layers read it. */
     speaking: boolean;
+    /**
+     * Corrective map for this body, or null to play the stream as sent. Applied
+     * to each frame as it comes off the queue, before the SDK's shaping and the
+     * mouth layer. Mutable so a viewer can A/B it on live speech.
+     */
+    correctiveMap: CorrectiveMap | null;
+    private mapped;
     private readonly ch;
-    constructor(tuning?: MouthTuning);
+    constructor(tuning?: MouthTuning, correctiveMap?: CorrectiveMap | null);
     /** Bind every morph-bearing mesh under root. Returns matched channel count. */
     bind(root: THREE.Object3D): number;
+    private correct;
     /** Tick each frame. Pass the queue only if this character owns the turn. */
     update(delta: number, q: BlendshapeQueue | undefined): void;
 }

package/dist/speech/lipsync.js

@@ -1,4 +1,5 @@
 import { createLipsyncProcessor, mhaChannelIndex, } from '@convai/web-sdk/lipsync-helpers';
+import { applyCorrectiveMap } from './correctiveMap.js';
 const DEFAULTS = {
     jawCap: 0.55,
     sealCeiling: 0.5,
@@ -10,7 +11,7 @@ const DEFAULTS = {
     gainOverrides: { mouthLowerLipDepressL: 0.28, mouthLowerLipDepressR: 0.28 },
 };
 export class LipsyncSystem {
-    constructor(tuning = {}) {
+    constructor(tuning = {}, correctiveMap = null) {
         this.bindings = [];
         this.accum = 0;
         this.lastFrame = null;
@@ -20,6 +21,8 @@ export class LipsyncSystem {
         this.energy = 0;
         /** True while the bot owns the mouth; body layers read it. */
         this.speaking = false;
+        this.mapped = null;
+        this.correctiveMap = correctiveMap;
         this.t = { ...DEFAULTS, ...tuning, gainOverrides: { ...DEFAULTS.gainOverrides, ...tuning.gainOverrides } };
         this.processor = createLipsyncProcessor('mha', {
             skipEyeChannels: true,
@@ -43,6 +46,14 @@ export class LipsyncSystem {
         this.bindings = this.processor.bindAll(root);
         return this.bindings.reduce((s, b) => s + b.matched, 0);
     }
+    correct(frame) {
+        const map = this.correctiveMap;
+        if (!map)
+            return frame;
+        if (this.mapped?.length !== frame.length)
+            this.mapped = new Float32Array(frame.length);
+        return applyCorrectiveMap(frame, map, this.mapped);
+    }
     /** Tick each frame. Pass the queue only if this character owns the turn. */
     update(delta, q) {
         if (!this.bindings.length)
@@ -74,7 +85,7 @@ export class LipsyncSystem {
             if (this.accum > 0.25)
                 this.accum = 0.25; // a starved queue banks no debt
             if (fresh)
-                this.lastFrame = fresh;
+                this.lastFrame = this.correct(fresh);
         }
         else {
             this.accum = 0;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@convai/embody",
-  "version": "0.4.1",
+  "version": "0.5.0",
   "description": "Give a Convai character a body: resolve 3D characters by characterId and drive them with lipsync and animation. Layered on @convai/web-sdk.",
   "license": "Apache-2.0",
   "author": "Convai <support@convai.com>",
@@ -99,7 +99,7 @@
     "typecheck": "tsc --noEmit",
     "test": "vitest run",
     "test:watch": "vitest",
-    "verify": "pnpm check:hosts && pnpm check:tests && pnpm typecheck && pnpm test && pnpm build",
+    "verify": "pnpm check:hosts && pnpm check:tests && pnpm typecheck && pnpm test && pnpm test:scripts && pnpm build",
     "changeset": "changeset",
     "version": "changeset version",
     "release": "pnpm build:prod && changeset publish",
@@ -109,6 +109,7 @@
     "build:stg": "pnpm clean && BRANCH=stg node scripts/check-hosts.mjs && pnpm build",
     "release:stg": "pnpm build:stg && changeset publish --tag stg",
     "docs": "typedoc",
-    "docs:watch": "typedoc --watch"
+    "docs:watch": "typedoc --watch",
+    "test:scripts": "node --test scripts/sync-docs.test.mjs"
   }
 }