@vincentt-xr/sdk 2.0.0-alpha.7 → 2.0.0-alpha.9

package/docs/api/external-face-mesh.mdx

@@ -0,0 +1,231 @@
+---
+title: "External Face Mesh"
+description: "Wear your own head or face model on the tracked face: it follows the head pose and the face's expression, with any topology and UV layout."
+sidebarTitle: "External Face Mesh"
+icon: "masks-theater"
+---
+
+`ExternalFaceMesh` puts your own face model on the tracked face. It follows the head pose and the
+face's expression in two ways at once:
+
+- **Landmarks:** the tracked landmarks bend the mesh, so the jaw opens and the lips and cheeks move.
+- **Morphs:** the model's own ARKit-named morph targets play the face's blendshapes. That gives an
+  authored blink and raised brows, and the eyeballs turn with the gaze.
+
+It is the foundation of the face-character pipeline
+([#194](https://github.com/vincentt-xr/sdk/issues/194)). Expression controllers, pinned accessories
+and face reprojection build on the same `FaceRig`.
+
+Import from `@vincentt-xr/sdk/face-effects`.
+
+## Quick start
+
+```tsx
+import { useLoader } from "@react-three/fiber";
+import { GLTFLoader } from "three/examples/jsm/loaders/GLTFLoader.js";
+import { FaceTracker } from "@vincentt-xr/sdk/tracking";
+import { ExternalFaceMesh } from "@vincentt-xr/sdk/face-effects";
+
+const Character = () => {
+  const head = useLoader(GLTFLoader, headUrl); // authored in canonical centimetres
+  return (
+    <FaceTracker>
+      <ExternalFaceMesh model={head.scene} />
+    </FaceTracker>
+  );
+};
+```
+
+It reads the parent `FaceTracker`'s face (or a `HeadBinding`'s) and never acquires the face model
+itself. The model is cloned with its geometry, so one loaded model can back several faces.
+
+## Authoring the model
+
+Author the model in the **canonical head frame**, the same frame as `CanonicalHeadFrame` and the v3
+head occluder `.blend` in `assets-src/head-occluder/`:
+
+- **Units and axes:** centimetres, +Y up, +Z out of the face toward the camera.
+- **Origin:** landmark 4, just above the nose tip, sits at `(0, -0.46, 7.59)`.
+
+The easiest way is to open the occluder `.blend` and fit your head over it. Its first 468 vertices
+are the tracked face landmarks, so the eyes, nose and mouth line up when your model sits on them.
+
+Any topology and any UV layout works. The face area should lie close to the canonical face surface:
+the closer it is, the more of the face follows the expression. The rest (skull, ears, neck,
+accessories) can go anywhere and stays rigid.
+
+## How it follows the face
+
+**What drives it** is set by `drive`:
+
+| `drive`            | What moves it                                                                                                    |
+| ------------------ | ---------------------------------------------------------------------------------------------------------------- |
+| `"auto"` (default) | `"morphs"` for a full ARKit set (mouth morphs too), `"hybrid"` with only eye or brow morphs, else `"landmarks"`. |
+| `"hybrid"`         | Landmarks for the jaw, lips and cheeks; the model's morphs for the eyes and brows.                               |
+| `"morphs"`         | Only the model's morph targets, driven by the blendshapes; otherwise just the head pose.                         |
+| `"landmarks"`      | Only the landmarks bend the mesh.                                                                                |
+| `"none"`           | The head pose only.                                                                                              |
+
+For a model with a full ARKit set, such as Face Cap's, the authored morphs look best, so `"auto"`
+picks `"morphs"`. Binding then plays no part, because nothing is bent.
+
+Why hybrid for eye and brow morphs: MediaPipe only tracks the edge of each eyelid, 16 points around each eye. Bending a
+sculpted lid with those points drags its edge down without rolling the lid over the eyeball. That
+leaves slits at the corners and creases the lid. An authored `eyeBlink` morph closes it cleanly, so
+hybrid masks the eye and brow landmarks (`FACE_REGION_MASKS`) and lets the morphs move those parts.
+
+**Morph names** map to MediaPipe's 51 blendshape channels (`FACE_MORPH_NAMES`, ARKit names). The
+match accepts:
+
+- ARKit names, e.g. `eyeBlinkLeft`;
+- Face Cap and Blender styles, e.g. `eyeBlink_L`, `eyeBlink.L`, `eyeBlink_left`;
+- any case;
+- exporter prefixes such as `blendShape1.`.
+
+`Left` is the model's own left (landmark 263's side, MediaPipe's left), as in ARKit. It is driven
+mesh to mesh, so the model's eye in front of the person's blinking eye blinks on every feed (on the
+mirrored webcam that is MediaPipe's right channel for the person's real left eye; GROUNDING "Left
+and right"). MediaPipe has no `tongueOut`.
+Morphs that no channel drives are left alone, so your own animation keeps them. The model's rest pose
+must have every morph at 0.
+
+**Binding** happens once, at load. Each vertex is bound to a point on the canonical face surface,
+which is MediaPipe's 853 face triangles. The mouth and eye openings are holes, so a lip never binds
+across the mouth. `binding` picks how:
+
+| `binding`          | Binds by                                                                             |
+| ------------------ | ------------------------------------------------------------------------------------ |
+| `"auto"` (default) | UV when the mesh's UVs are a canonical layout (most UV hits agree with 3D), else 3D. |
+| `"uv"`             | UV only, for a mesh whose face UVs follow `CANONICAL_FACE_UVS` (either v direction). |
+| `"surface"`        | The closest point in 3D. Any topology and layout.                                    |
+
+**Influence** depends on how far a vertex sits from the face surface. Within `falloff[0]` (0.8 cm)
+it follows fully; it fades to rigid at `falloff[1]` (2.5 cm). The back of the head, the ears and the
+neck therefore stay rigid.
+
+**Parts:** each mesh of the model follows in one of three ways (`parts`, by object name):
+
+- `"surface"`: every vertex follows its own point. Use it for skin.
+- `"follow"`: the whole mesh moves as one with the point under its centre, keeping its shape. Use
+  it for eyeballs, teeth or a nose ring. A follow part named like an eye (`eye…`) also turns with
+  the gaze (`eyeLookIn/Out/Up/Down`, up to `gazeDegrees` = `[25, 20]`).
+- `"rigid"`: head pose only. Use it for a helmet or hair.
+
+By default, a mesh whose median vertex sits more than 0.3 cm behind the face (eyeballs, teeth) is
+`"follow"`, and everything else is `"surface"`.
+
+**Neutral.** Only the expression moves the model, not the person's own face shape or resting
+expression. With `neutral: "auto"` (default), the rig averages the landmarks and the blendshape
+scores over the first `neutralMs` (600 ms) after the face is found, and subtracts that rest face. The
+blendshapes are rescaled, so a resting 0.1 reads 0 and a full blink still reads 1. It recaptures every time the face is found again, and when
+`neutralKey` changes: bind it to a "Recalibrate" button and ask the person to relax and look at the
+camera. With `neutral: "none"` the model also takes on the person's face shape (wider eyes, a longer
+jaw).
+
+## `ExternalFaceMesh`
+
+| Prop                  | Default       | What it does                                                                                |
+| --------------------- | ------------- | ------------------------------------------------------------------------------------------- |
+| `model`               | —             | The face model (e.g. a GLTF scene) in the canonical head frame.                             |
+| `drive`               | `"auto"`      | What moves it: `"auto"`, `"hybrid"`, `"morphs"`, `"landmarks"` or `"none"`.                 |
+| `binding`             | `"auto"`      | `"auto"`, `"uv"` or `"surface"`.                                                            |
+| `falloff`             | `[0.8, 2.5]`  | Full influence up to the first distance (cm), rigid beyond the second.                      |
+| `parts`               | auto          | Per mesh name: `"surface"`, `"follow"` or `"rigid"`.                                        |
+| `expression`          | `1`           | 0 = rigid, 1 = the tracked face, > 1 exaggerates.                                           |
+| `morphGain`           | —             | A gain per channel on its morph, e.g. `{ eyeBlinkLeft: 1.2 }`.                              |
+| `gazeDegrees`         | `[25, 20]`    | How far a full look turns an eyeball: sideways, up/down (degrees).                          |
+| `onBind`              | —             | Called once per model with every mesh's binding (readouts, debug views).                    |
+| `faceIndex`           | the tracker's | Which tracked face (when not inside a `FaceRig`; likewise the rows below).                  |
+| `smoothing`           | `0.5`         | Head pose EMA while still (`1` = none). Fast turns follow at once, so a shake never trails. |
+| `expressionSmoothing` | `0.6`         | Expression EMA, `1` = none. MediaPipe's depth is noisy; a little keeps the mesh calm.       |
+| `holdMs`              | `250`         | Keep the last pose and expression through shorter dropouts.                                 |
+| `neutral`             | `"auto"`      | `"auto"` subtracts the captured rest face; `"none"` keeps the person's face shape.          |
+| `neutralMs`           | `600`         | How long the neutral capture averages.                                                      |
+| `neutralKey`          | —             | A new value captures the neutral again.                                                     |
+| `leadMs`              | `0`           | Landmark lead between inferences (One Euro + velocity, at most this long). 0 = raw.         |
+| `onTrackingChange`    | —             | Called once on found (`true`) and once on lost (`false`).                                   |
+| `children`            | —             | More content in the canonical head frame.                                                   |
+
+## `FaceRig`
+
+`FaceRig` is the engine underneath, and every face-character component shares one. It is a group in
+the canonical head frame, like `CanonicalHeadFrame`, that also publishes the face's expression. Put
+several components inside one `FaceRig` to share it:
+
+```tsx
+<FaceTracker>
+  <FaceRig neutralKey={recalibrations}>
+    <ExternalFaceMesh model={head.scene} />
+    <ExternalFaceMesh model={beard.scene} />
+  </FaceRig>
+</FaceTracker>
+```
+
+`useFaceRig()` returns a ref to the live `FaceRigState`. Read it inside `useFrame`:
+
+- `tracking`: `"tracked"`, `"held"` or `"lost"`.
+- `pose`: `{ position, quaternion, scale }`, canonical cm → world.
+- `local`: the 468 landmarks in the head frame (cm).
+- `residual`: the expression, as each landmark's offset from the rest face (cm, 468 × 3).
+- `morphs`: the 51 blendshape channels (0..1, neutral removed).
+- `neutral`, `morphNeutral` and `neutralState`.
+
+The rig holds its last state through dropouts shorter than `holdMs`, hides when the face is lost,
+and snaps on reacquire.
+
+The pure pieces are exported for custom renderers:
+
+- `stepFaceRig(state, worldLandmarks, nowMs, options, scores)`, `createFaceRigState()` and
+  `recalibrateFaceRig(state)`.
+- `bindFaceMesh(positionsCm, uvs, { mode, falloff, uvTolerance })` and
+  `deformFaceMesh(binding, rest, residual, out, amount, toLocal)`.
+- `bindFaceModel(root, options)`, `planFaceDrive(meshes, mode)` and
+  `driveFaceModel(meshes, plan, rig, options)`.
+- `faceMorphChannel(name)`, `faceMorphRegion`, `FACE_GAZE_CHANNELS` and `FACE_REGION_MASKS`.
+- The canonical data: `CANONICAL_FACE_POSITIONS`, `CANONICAL_FACE_TRIANGLES` and
+  `CANONICAL_FACE_UVS`.
+
+## Example
+
+`/external-face-mesh` wears the three.js "facecap" head when it is present. The file goes in the
+git-ignored `example/src/assets/models/local/facecap-canonical.glb`; it has no redistribution
+licence, so it is never committed. That copy is prepared as follows:
+
+- aligned to the canonical frame;
+- ears raised to the anatomical height (brow to nose base);
+- every morph zeroed at rest (the original kept a frame of its animation).
+
+It carries all 52 ARKit morphs, of which 51 are tracked. Without it, the page uses the canonical
+head.
+
+The page has controls for:
+
+- drive, binding, expression and falloff;
+- a binding heat map (red follows, blue rigid);
+- keeping the face shape;
+- recalibrating the neutral;
+- the head occluder and two faces.
+
+## Manual visual test plan
+
+Open `/external-face-mesh` on the webcam.
+
+1. **Placement**: the model sits on your head, with its eyes, nose and mouth on yours, and stays
+   there as you turn and tilt.
+2. **Neutral**: look at the camera relaxed when it appears. At rest it keeps its own shape: eyes
+   not stretched, jaw not lengthened. Tick "keep the person's face shape" and it conforms to yours;
+   untick it.
+3. **Expression**: open your mouth (the jaw drops and the lips part, never stretching across), smile
+   (the mouth corners and cheeks move), raise your brows. With drive "none", nothing moves but the
+   pose.
+4. **Blink and gaze** (facecap, drive "auto" = morphs): blink one eye, then both. The lid rolls down over the
+   eye and closes, with no slit at the corners and no crease. Look left, right, up and down: the
+   eyeballs turn with you. Open your mouth and smile: facecap's own jawOpen and smile shapes play.
+   Switch drive to "landmarks" to see the difference.
+5. **Parts**: the eyes and teeth move as whole pieces, never squashed. The back of the head and the
+   ears stay rigid; the heat map shows red on the face fading to blue.
+6. **Recalibrate**: start with your mouth open, then relax and press Recalibrate. The rest pose is
+   corrected.
+7. **Dropouts**: a brief cover of the camera holds the head. Longer, and it hides, then snaps back
+   on reacquire.
+8. **Two faces**: on the two-people image, each face wears its own copy.

package/docs/api/face-actions.mdx

@@ -37,21 +37,23 @@ const isPouting = hasFaceAction(face, "mouthPout");
 
 ## Actions
 
-| Name                            | Meaning                      |
-| ------------------------------- | ---------------------------- |
-| `browRaise`                     | Either brow is raised        |
-| `eyeBlink`                      | Both eyes blink              |
-| `eyeBlinkEither`                | Either eye blinks            |
-| `eyeBlinkLeft`, `eyeBlinkRight` | Viewer-left/right eye blinks |
-| `mouthOpen`                     | Jaw opens                    |
-| `mouthPout`                     | Lips pucker                  |
-| `headUp`, `headDown`            | Pitch from neutral           |
-| `headTurnLeft`, `headTurnRight` | Yaw from neutral             |
-| `headTiltLeft`, `headTiltRight` | Roll from neutral            |
+| Name                            | Meaning                                                          |
+| ------------------------------- | ---------------------------------------------------------------- |
+| `browRaise`                     | Either brow is raised                                            |
+| `eyeBlink`                      | Both eyes blink                                                  |
+| `eyeBlinkEither`                | Either eye blinks                                                |
+| `eyeBlinkLeft`, `eyeBlinkRight` | The person's own left / right eye blinks                         |
+| `mouthOpen`                     | Jaw opens                                                        |
+| `mouthPout`                     | Lips pucker                                                      |
+| `headUp`, `headDown`            | Pitch from neutral                                               |
+| `headTurnLeft`, `headTurnRight` | Yaw from neutral, toward the person's own left / right           |
+| `headTiltLeft`, `headTiltRight` | Roll from neutral, toward the person's own left / right shoulder |
 
 Blendshape values are in `[0, 1]`. Head values are non-negative radians. Up and
-left follow the SDK convention and are positive before they are converted to an
-action magnitude.
+the person's own left follow the SDK convention and are positive before they
+are converted to an action magnitude. Left and right are the person's own side
+on every feed: the mirrored webcam, the rear camera, images and video
+(GROUNDING "Left and right").
 
 ## Neutral head pose
 
@@ -89,15 +91,17 @@ left, right, both, and either-eye movement cases. Use `mouthOpen`, `mouthPout`,
 and `browRaise` for the matching physical movements. These actions report
 observable pose/blendshape movement; they do not infer emotion.
 
-Left/right blink names use the mirrored viewer-facing convention used by effect
-editors. Raw `FaceInfo.blendshapes` retain MediaPipe's anatomical-side names.
+Left/right blink names are the person's own eyes on every feed. Raw
+`FaceInfo.blendshapes` keep MediaPipe's names (`eyeBlinkLeft` = MediaPipe's left
+eye: the person's own left on an unmirrored feed, their right on the mirrored
+webcam); the actions read the right one for `face.mirrored`.
 The lifecycle guarantees one `stay` frame between `begin` and `end`, including
 for a blink that drops below its release threshold on the next input frame.
 
 For image-based happy, sad, angry, fear, disgust, surprise, neutral, and unknown
 classification, use the separate `useFaceAttributeExpression` hook. The SDK
 does not fabricate these labels from movement thresholds. The verified
-`/facial-movement-expression` page exposes both independent surfaces.
+`/face-actions` page exposes both independent surfaces.
 
 Use `/face-actions` to view every value, calibrate neutral pose, adjust the
 selected action, and repeat the live verification checklist. All actions and

package/docs/api/face-attribute-expression.mdx

@@ -82,6 +82,6 @@ Without those headers, ONNX Runtime falls back to single-threaded WASM.
 Treat classifications as model estimates of visible facial expression—not
 ground-truth emotion, intent, identity, or mental state.
 
-Use the verified `/facial-movement-expression` page to inspect the live winning
+Use the verified `/face-actions` page to inspect the live winning
 class and all seven probabilities. Its real-camera model flow, including every
 expression class, was verified on 26 July 2026.

package/docs/api/face-binding-anchors.mdx

@@ -7,7 +7,9 @@ icon: "locate-fixed"
 
 Use `TrackingAnchor` to attach content to semantic face locations. Build editor
 dropdowns from `FACE_TARGETS`; do not copy landmark indices or invent target
-strings. Left and right use the tracked person's perspective.
+strings. Left and right are the tracked person's own side on every feed (GROUNDING "Left and
+right"): `face.leftEarlobe` is their real left ear lobe on the mirrored webcam (the screen's left)
+and on a video (the screen's right).
 
 ## Presets
 
@@ -33,7 +35,7 @@ does not provide anatomical earlobe points.
 ## Custom points
 
 For `face.custom`, pass `customAnchorPoint`. X and Y are clamped to `[-1, 1]`;
-the origin is face center, X runs from the person's left to right, Y runs from
+the origin is face center, X runs from the person's own left to right on every feed, Y runs from
 chin to forehead, and positive Z points toward the camera.
 
 ```tsx

package/docs/api/face-character.mdx

@@ -0,0 +1,157 @@
+---
+title: "Face Character (end to end)"
+description: "Every face-character piece together: a cute warp of your real face, a transform into a character, the character with your eyes and mouth, pins, a hat behind the head and hands in front."
+sidebarTitle: "Face Character"
+icon: "masks-theater"
+---
+
+This page puts the face-character pipeline ([#194](https://github.com/vincentt-xr/sdk/issues/194),
+#200) together in one lens, for up to two people at once. Each piece has its own page:
+
+| Piece                                                         | What it does here                                                                              |
+| ------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
+| [`FaceWarp3D`](/api/face-warp-3d)                             | "You": your real face made cute (a bigger-eyes shape key), photoreal                           |
+| [`FaceMorph`](/api/face-morph)                                | "Transform": your face swells into the character, your eyes flowing into its eyes              |
+| [`ExternalFaceMesh`](/api/external-face-mesh)                 | "Character": the character head, following your head                                           |
+| [`FaceExpressionController`](/api/face-expression-controller) | Plays the character's shape keys from your expression, plus blush, sparkles and a wink counter |
+| [`FaceReprojection`](/api/face-reprojection)                  | Your eyes and mouth on the character (`align` for its cartoon proportions)                     |
+| [`PinToFaceMesh`](/api/pin-to-face-mesh)                      | A bow and a star that ride the warp, the morph and the character's skin                        |
+| [`HeadAttachment`](/api/head-occluder) + occluder             | A party hat whose back goes behind your head                                                   |
+| `HandCutout`                                                  | Your hands stay in front of all of it                                                          |
+
+## The structure
+
+```tsx
+import {
+  ExternalFaceMesh,
+  FaceExpressionController,
+  FaceMorph,
+  FaceReprojection,
+  FaceRig,
+  FaceWarp3D,
+  HeadAttachment,
+  HeadBinding,
+  PinToFaceMesh,
+} from "@vincentt-xr/sdk/face-effects";
+import { HandCutout, HandTracker } from "@vincentt-xr/sdk/tracking";
+
+<HandTracker>
+  <HandCutout />
+</HandTracker>;
+
+{
+  [0, 1].map((faceIndex) => (
+    <HeadBinding key={faceIndex} faceIndex={faceIndex} numFaces={2} occluder={{ url: occluderUrl }}>
+      <FaceRig>
+        {phase === "you" && (
+          <FaceWarp3D deformation={cuteShapes} weights={{ cute: 0.7 }}>
+            <Pins /> {/* children ride the warp */}
+          </FaceWarp3D>
+        )}
+        {phase === "transform" && (
+          <FaceMorph shape={character} align={GHOST_ALIGN} weight={sweepRef} falloff={[4, 9]}>
+            <Pins /> {/* children ride the morph */}
+          </FaceMorph>
+        )}
+        {phase === "character" && (
+          <FaceExpressionController rules={ghostRules}>
+            <group ref={characterRef}>
+              <ExternalFaceMesh model={character} drive="none">
+                <FaceReprojection region="eyesAndMouth" align={GHOST_ALIGN} meshes={GHOST_WEARS} />
+              </ExternalFaceMesh>
+            </group>
+            <Pins onto={characterRef} /> {/* on the character's skin */}
+          </FaceExpressionController>
+        )}
+      </FaceRig>
+      <HeadAttachment point="crown" offset={{ position: [0, 0.6, 0.4] }}>
+        <PartyHat />
+      </HeadAttachment>
+    </HeadBinding>
+  ));
+}
+```
+
+### Stylised characters
+
+The SDK ghost head (`face-v1.glb`) has cartoon proportions: big eyes 3 cm lower than a real face's.
+Pass its own eye and mouth centres as `align` (`{ leftEye, rightEye, mouth }`). `FaceReprojection` and
+`FaceMorph` then bind it as if those sat on your face, so your eyes land on its eyes.
+
+Being rounder than a real face, it also needs a wider `FaceMorph` `falloff` (`[4, 9]`). Name the parts
+that wear your face with `meshes`. Eye parts are found by mesh or material name (`eye…`, `iris…`).
+
+### Expressions on a character's own shape keys
+
+The ghost's shape keys (`smile`, `wink`, `surprised`, `kiss`, `puffCheeks`, `frown`, `scrunch`) are
+played by `FaceExpressionController` rules from ARKit channels. That's why the `ExternalFaceMesh` uses
+`drive="none"`.
+
+The same controller also handles the extras:
+
+- the blush's opacity follows your smile;
+- sparkles show while your mouth is open;
+- each wink triggers a counter.
+
+### Draw order and the head occluder
+
+The occluder is your real head, skull included. It must hide what is behind your head, like the
+hat's back. It must never hide the face effects, including a character head smaller than your skull.
+
+Everything that is the face draws first, at `FACE_SURFACE_RENDER_ORDER` (-2), before the occluder
+(-1, on its meshes). Accessories come after (0). Set render orders on meshes, never on a `Group`:
+three sorts by the nearest group first.
+
+### Hands in front
+
+`HandCutout` draws your hands' silhouettes as near-plane depth before everything else. A hand over your
+face shows the real hand in front of the warp, the character, the pins and the hat.
+
+## Example
+
+`/face-character` has:
+
+- **A Transform button.**
+- **Controls:**
+  - the character (the ghost head, or facecap when the local file is present);
+  - "cute you";
+  - your eyes and mouth on the character;
+  - a toon material;
+  - the head occluder;
+  - hands in front;
+  - one or two people.
+- **Badges** for each face's tracking and a wink count.
+
+URL options:
+
+- `?phase=character` starts as the character;
+- `?hold=0.5` freezes the transform at that weight, for inspection;
+- `?faces=1`, `?occluder=0`, `?hands=0`, `?wear=0` and `?toon=1` set the controls.
+
+## Manual visual test plan
+
+Open `/face-character` on the webcam.
+
+1. **You**: your eyes are bigger and the face looks real. The bow sits on the side of your forehead
+   and the star on your cheek; both move with your face. The hat sits on your crown, and turning your
+   head hides its back behind your head.
+2. **Transform**: your face swells into the ghost over about 1.5 s. Your eyes flow into its eyes, and
+   the bow and star ride along. "Back to you" reverses it.
+3. **Character**:
+   - the whole round ghost is visible, with nothing cut away by the occluder;
+   - your eyes and mouth are on it;
+   - smile: it smiles and blushes;
+   - wink: the counter goes up;
+   - open your mouth: sparkles;
+   - puff your cheeks, kiss, frown and scrunch: it does too.
+4. **Occluder**: toggle the head occluder. The face effects never change; only the hat's back hides
+   or shows.
+5. **Hands**: put a hand over your face in every phase. The real hand stays in front of the warp, the
+   character, the pins and the hat. Turn the toggle off and they draw over your hand.
+6. **Two people**: with two faces in view, each gets its own lens. The badges follow each face.
+7. **Tracking loss**: cover your face. The lens holds briefly, then hides (the badge says "lost") and
+   comes back on reacquire.
+8. **Mirror**: switch between the webcam and the sample media. Everything stays on the right side of
+   the face.
+9. **facecap** (local): pick facecap. The transform and the character work on a realistic head, with
+   no `align` needed.

package/docs/api/face-effects.mdx

@@ -9,10 +9,19 @@ This section is split into focused pages:
 
 - [Face Mesh](/api/face-mesh) for the base 3D face mesh, canonical renderer,
   and face mesh material settings.
+- [Head Occluder](/api/head-occluder) for `HeadBinding`, the canonical-fit
+  depth-only head mesh and `CanonicalHeadFrame`, so 3D accessories pass behind
+  the tracked head.
 - [Face Painting](/api/face-painting) for the composite shader variant used to
   paint onto the live face mesh.
 - [Face Retouching](/api/face-retouching) for the legacy skin-retouch shader
   and its controls.
+- [Segmentation Overlay](/api/segmentation-overlay) for 2D semantic region
+  layers (lips, eyes, brows, face skin, person, background) without 3D
+  geometry, and when to prefer them over the canonical mesh.
+- [Lip Effect Textures](/guides/lip-effect-textures) for `LipEffect`: the
+  design-texture canvas contract, the generated lip UV template, mask modes,
+  and the custom material context.
 
 Use `@vincentt-xr/sdk/tracking` for trackers, `FaceInfo`, Face Actions, and the
 bare deforming `FaceMesh`. Use `@vincentt-xr/sdk/face-effects` for the modular

package/docs/api/face-expression-controller.mdx

@@ -0,0 +1,153 @@
+---
+title: "Face Expression Controller"
+description: "Turn the tracked expression into behaviour with typed rules: morph targets, material values, visibility, transforms, triggers and your own setters."
+sidebarTitle: "Face Expression Controller"
+icon: "sliders"
+---
+
+`FaceExpressionController` turns the tracked face's expression into behaviour. Each rule has three
+parts:
+
+- an **input**, such as a blendshape, a combination of them, a Face Action or your own function;
+- its **shaping**: range, curve, multiplier, clamp and smoothing;
+- its **outputs**: morph targets, material values, visibility, transforms, triggers or a setter.
+
+It is part of the face-character pipeline ([#194](https://github.com/vincentt-xr/sdk/issues/194)):
+#196 is the controller, and #197 is custom morph mapping. It runs on a
+[`FaceRig`](/api/external-face-mesh#facerig).
+
+Import from `@vincentt-xr/sdk/face-effects`.
+
+## Quick start
+
+```tsx
+import { ExternalFaceMesh, FaceExpressionController, FaceRig } from "@vincentt-xr/sdk/face-effects";
+import { FaceTracker } from "@vincentt-xr/sdk/tracking";
+
+<FaceTracker>
+  <FaceRig>
+    <FaceExpressionController
+      rules={[
+        { input: "jawOpen", morph: "jawOpen", multiplier: 1.4 },
+        { input: { action: "mouthOpen" }, target: "bubble", visible: { above: 0.45 } },
+        { input: { action: "eyeBlink" }, onStart: () => playBlinkSound() },
+      ]}
+    >
+      <ExternalFaceMesh model={head.scene} drive="none" />
+      <group name="bubble">…</group>
+    </FaceExpressionController>
+  </FaceRig>
+</FaceTracker>;
+```
+
+The outputs act on the controller's children, which sit in the canonical head frame (centimetres).
+Inside a `FaceRig` the controller shares that rig; otherwise it runs its own, so mount it in a
+`FaceTracker`.
+
+## Inputs
+
+| `input`                               | Reads                                                                                        |
+| ------------------------------------- | -------------------------------------------------------------------------------------------- |
+| `"jawOpen"`                           | A blendshape channel (ARKit names, `FACE_MORPH_NAMES`), with the person's neutral removed.   |
+| `{ channels: [...], combine: "avg" }` | Several channels: `"max"` (default), `"min"`, `"avg"` or `"sum"`.                            |
+| `{ action: "mouthOpen" }`             | A Face Action's value, as `useFaceAction` reads it (raw blendshapes, head turns in radians). |
+| `(frame) => number`                   | Your own function of `{ morphs, face, live }`.                                               |
+
+An unknown channel or action reads 0 and warns once, listing the valid names.
+
+## Shaping
+
+Every frame, a rule's value goes through these steps in order:
+
+```text
+input → range (→ 0..1) → curve → × multiplier → clamp → smoothing
+```
+
+| Field        | Default          | What it does                                                                              |
+| ------------ | ---------------- | ----------------------------------------------------------------------------------------- |
+| `range`      | `[0, 1]`         | The input span mapped onto 0..1: below the first value reads 0, above the second reads 1. |
+| `curve`      | `1`              | An exponent: above 1 eases in (subtle until strong), below 1 eases out.                   |
+| `multiplier` | `1`              | A scale after the curve: above 1 exaggerates.                                             |
+| `clamp`      | `[0, max(1, ×)]` | Limits after the multiplier.                                                              |
+| `smoothing`  | `0.6`            | A per-frame EMA factor, or `{ attack, release }` to rise and fall at different speeds.    |
+| `rest`       | `0`              | The value at rest: after a reset, and on face loss.                                       |
+| `onLost`     | `"rest"`         | On face loss, ease back to `rest`, or `"hold"` the last value.                            |
+
+## Outputs
+
+Outputs act on the object named `target` in the subtree; without a `target`, on the whole subtree.
+
+| Output              | Writes                                                                                                        |
+| ------------------- | ------------------------------------------------------------------------------------------------------------- |
+| `morph`             | Morph target name(s) on every mesh under it. An ARKit name also matches the Face Cap spelling (`eyeBlink_L`). |
+| `property`          | A material uniform (`uniforms[name].value`) or a numeric property (`opacity`, `emissiveIntensity`).           |
+| `visible`           | Shown while the value is ≥ `above` and/or ≤ `below` (default: ≥ 0.5).                                         |
+| `transform`         | An offset from the target's rest at value 1: `position`, `rotation` (degrees), `scale` (factor).              |
+| `set`               | Your own setter, called every frame with the value.                                                           |
+| `onStart` / `onEnd` | Edge callbacks, with `trigger: [on, off]` hysteresis (default `[0.5, 0.35]`).                                 |
+
+Several `transform` rules on one target add up, and the target returns to its rest pose every frame
+before they apply. Bindings re-resolve once a second, so children that load later (a GLTF behind
+`Suspense`) are picked up. A target, morph or property that is still missing after that warns once.
+
+**Morphs and `ExternalFaceMesh`.** When the controller drives a face model's morphs, set that
+model's `drive` to `"none"` or `"landmarks"`. Otherwise both write the morphs. For a simple gain on
+the automatic mapping, `ExternalFaceMesh`'s `morphGain` is enough.
+
+## Reset and face loss
+
+- **Reset:** a new `resetKey` puts every rule back at `rest` and ends active triggers, firing
+  `onEnd`.
+- **Face loss:** values ease back to `rest`, unless a rule sets `onLost: "hold"`.
+- **Neutral:** recalibrating the person's neutral is the rig's job (`FaceRig`'s `neutralKey`).
+
+## Headless use
+
+`useExpressionValues(rules)` runs the same rules without outputs. Read the values in `useFrame`:
+
+```tsx
+const expr = useExpressionValues([
+  { id: "smile", input: { channels: ["mouthSmileLeft", "mouthSmileRight"], combine: "avg" } },
+]);
+useFrame(() => {
+  shader.uniforms.uSmile.value = expr.get("smile");
+});
+```
+
+The engine is pure and exported:
+
+- `compileExpressionInput`, `createExpressionState` and
+  `stepExpressions(state, rules, readers, frame)`;
+- `shapeExpressionValue` and `resetExpressions`;
+- `expressionFrame(rig)`;
+- `createExpressionOutputs(root)`, which has `resolve(rules, warn)` and `apply(rules, state)`.
+
+## Example
+
+`/external-face-mesh?rules=1` runs on the facecap head when it is present locally (see
+[External Face Mesh](/api/external-face-mesh#example)). It shows:
+
+- **Custom morph mapping:** a cartoon mapping with an "exaggerate" slider. A toggle compares it with
+  `ExternalFaceMesh`'s automatic mapping.
+- **Props:** a hat that lifts and tilts with the brows, a bubble that appears when the mouth opens,
+  and a cheek blush that follows the smile.
+- **Triggers:** blink and mouth-open counters.
+- **Readout:** the live rule values.
+
+## Manual visual test plan
+
+Open `/external-face-mesh?rules=1` on the webcam and look at the camera, relaxed, when the head
+appears. That captures the neutral.
+
+1. **At rest**: every bar reads 0. The hat sits on the crown, there is no bubble and no blush.
+2. **Mouth**: open your mouth wide. With the custom mapping on, the model's jaw opens wider than
+   yours (exaggerate 1.4), and the bubble pops up. The mouth counter goes up once per opening, not
+   once per frame.
+3. **Blink**: each blink snaps the lids shut and reopens them softly, and the blink counter goes up
+   once.
+4. **Brows and smile**: raising your brows lifts and tilts the hat. Smiling brings up the blush,
+   up to 0.75 opacity.
+5. **Exaggerate**: at 0.5 everything is subdued; at 2 it is cartoonish but clamped.
+6. **Auto vs custom**: untick "custom morph mapping". `ExternalFaceMesh` drives the morphs one to
+   one, with no exaggeration and no soft reopen.
+7. **Face loss**: cover the camera. Everything eases back to rest, and the hat comes down.