@vincentt-xr/sdk 2.0.0-alpha.7 → 2.0.0-alpha.9

package/docs/api/face-mesh.mdx

@@ -68,8 +68,10 @@ texture, blend, or mask state changes.
 
 Material types and shader presets are quick SDK defaults. Use
 `materialOverride` when the face needs a custom Three.js material or shader.
-It accepts a material node or a factory that receives the current camera texture
-and geometry key. This prop wins over preset settings and is intentionally not
+It accepts a material node or a factory that receives the current camera texture,
+the geometry key, and `faceUv`: the shared live camera-UV `BufferAttribute`
+(`null` until the canonical mesh loads), also bound to every group's geometry as
+`attribute vec2 faceUv`. This prop wins over preset settings and is intentionally not
 part of `FaceMeshRendererSettings`, because materials, uniforms, and factories
 are not serializable scene data.
 
@@ -106,14 +108,13 @@ subtracts Y while adding X/Z. `DEFAULT_FACE_MESH_TRANSFORM.position.y` is
 backwards compatibility. New scene-object work should use `FaceMeshRenderer`.
 
 ```tsx
-import { CanonicalFaceMeshRenderer, FACE_WARP_3D_PRESET } from "@vincentt-xr/sdk/face-effects";
+import { CanonicalFaceMeshRenderer } from "@vincentt-xr/sdk/face-effects";
 
 <CanonicalFaceMeshRenderer
   materialType="shader"
   shaderPreset="originalFace"
   textureMap="none"
   alphaMap="canonical"
-  faceWarp3D={FACE_WARP_3D_PRESET}
 />;
 ```
 
@@ -124,8 +125,6 @@ import { CanonicalFaceMeshRenderer, FACE_WARP_3D_PRESET } from "@vincentt-xr/sdk
 
 ## FaceWarp3D
 
-`FACE_WARP_3D_PRESET`, `DEFAULT_FACE_WARP_3D_SETTINGS`, and
-`FaceWarp3DSettings` cover face, forehead, cheek, eye, nose, mouth, jaw, and
-chin proportions with normalized values around zero. Use `faceWarp3D` on
-`CanonicalFaceMeshRenderer` or `FaceMeshRenderer` settings instead of writing a
-second landmark deformation path.
+To reshape the face (bigger eyes, a longer nose), use [`FaceWarp3D`](/api/face-warp-3d). It applies
+blend shapes on the live face, with the camera image itself as the texture. It replaces the old
+proportion sliders and `CanonicalFaceMeshRenderer`'s `faceWarp3D` prop.

package/docs/api/face-morph.mdx

@@ -0,0 +1,117 @@
+---
+title: "Face Morph"
+description: "The person's face morphs into another shape (a sphere, a cube, a character), wearing their live camera face all the way."
+sidebarTitle: "Face Morph"
+icon: "wand-magic-sparkles"
+---
+
+`FaceMorph` morphs the person's face into another shape: a sphere, a cube, or a character. The shape
+wears their live camera face all the way, as in Snap Lens Studio's Face Morph.
+
+- `weight` 0 is the live face itself, so the image is unchanged.
+- `weight` 1 is the shape, which still moves with the person's expression.
+
+Animate the weight for the transformation.
+
+It is part of the face-character pipeline ([#194](https://github.com/vincentt-xr/sdk/issues/194)).
+Import from `@vincentt-xr/sdk/face-effects`.
+
+## Quick start
+
+```tsx
+import { FaceMorph } from "@vincentt-xr/sdk/face-effects";
+import { FaceTracker } from "@vincentt-xr/sdk/tracking";
+
+<FaceTracker>
+  <FaceMorph shape="sphere" weight={0.8} />
+</FaceTracker>;
+```
+
+## FaceMorph or FaceWarp3D?
+
+`FaceMorph` turns the face into another object (a ball, a box, a stylised character) that wears the camera
+face. To reshape the person's real face instead (bigger eyes, a longer nose, a cartoon's proportions,
+photoreal, with their own hair and skin), use [`FaceWarp3D`](/api/face-warp-3d). On a realistic head model,
+`FaceMorph` shows the model's own material on the skull and ears, while `FaceWarp3D` keeps the person's
+real hair. Use one face effect per face.
+
+## Shapes
+
+- **Built-in `"sphere"` and `"cube"`.** These are the face itself, with the eyes and mouth closed
+  over, subdivided and pushed out onto the shape. They fade out at the edge of the face.
+- **Your own model**, authored in the canonical head frame (centimetres, as for
+  [`ExternalFaceMesh`](/api/external-face-mesh)). Two things matter:
+
+  - its **base positions**: where it sits on a face. These bind it.
+  - a **shape key** named `"morph"` (`morphKey`): what it becomes.
+
+  In Blender, start from a mesh fitted to the face (the basis), sculpt the shape key, and export
+  glTF. A model without the key is its own base. For example, facecap morphs into its own shape.
+
+Every vertex is bound through its base pose to its closest point on the face, within `falloff` cm:
+
+- A vertex beyond that keeps to the shape, so a skull or the back of a head stays put.
+- Parts named `eye…` wear your eyes.
+- Teeth keep their own material.
+
+## Props
+
+| Prop                                | Default   | What it does                                                                                         |
+| ----------------------------------- | --------- | ---------------------------------------------------------------------------------------------------- |
+| `shape`                             | —         | `"sphere"`, `"cube"`, or a model (`THREE.Object3D`).                                                 |
+| `weight`                            | `1`       | 0 = the live face, 1 = the shape. Also a ref (`LiveValue`), read every frame.                        |
+| `expression`                        | `1`       | How strongly the shape takes the expression (Snap's Expression Multiplier). Also a ref.              |
+| `scale`                             | `1`       | Scale of the shape around the head (Snap's External Scale).                                          |
+| `morphKey`                          | `"morph"` | The shape key a model morphs into.                                                                   |
+| `faceTexture`                       | `1`       | How much of the camera face it wears: 0 shows the model's own material (a character's skin).         |
+| `region`                            | `"face"`  | `"face"`, `"eyes"`, `"mouth"` or `"eyesAndMouth"` (see [Face Reprojection](/api/face-reprojection)). |
+| `shading`                           | `"unlit"` | `"unlit"`: the camera's colours. `"lit"`: shaded by the scene lights.                                |
+| `color`                             | `#f1c6a8` | A built-in shape's colour where it shows no face.                                                    |
+| `subdivisions`                      | `2`       | Smoothness of a built-in shape.                                                                      |
+| `falloff`                           | `[1, 3]`  | How far from the face (cm, base pose) a vertex still morphs and wears it.                            |
+| `facing`, `fps`, `unwrapResolution` | —         | As for Face Reprojection.                                                                            |
+| rig props                           | —         | `faceIndex`, `smoothing`, `holdMs`, `neutral` … (when it runs its own `FaceRig`).                    |
+
+## How it works
+
+1. **Live face.** Every frame, the live face (`FaceRigState.local`) and its expression
+   (`residual`) are read on the filled canonical face. The eye and mouth centres are the
+   averages of their rings.
+2. **Blend.** Each vertex blends from its bound point on the live face to its shape position plus
+   the expression, by `weight` and by how close it is to the face.
+3. **Texture.** The face texture is the reprojector's unwrap, the camera face in the canonical
+   layout. Each vertex reads it through its `faceUv` / `faceCoverage` attributes, so no remap pass
+   is needed.
+
+**Pure pieces:** `createFaceMorphShape`, `createFaceMorphLive`, `writeFaceMorphLive` and
+`stepFaceMorph`.
+
+## Example
+
+`/face-morph` offers four shapes:
+
+- the sphere;
+- the cube;
+- a **Lens Studio-style head**, sculpted from Snap's face + skull + ears mesh (a "chibi" shape key
+  with a bigger skull, puffed cheeks, a small chin and bigger ears);
+- the **facecap** character.
+
+The last two are local, git-ignored files. The page has controls for weight (or a looping
+animation), expression, scale, face texture, region and shading.
+
+## Manual visual test plan
+
+Open `/face-morph` on the webcam.
+
+1. **Weight 0**: nothing changes. The mesh is exactly your face, with no edge or band.
+2. **Sphere / cube at 1**: your face is on a ball or a box, sharp and centred. The edges fade into
+   the camera image.
+3. **Animate**: the face swells into the shape and back, smoothly and without popping.
+4. **Expressions**: at weight 1, open your mouth, smile, blink and raise your brows. The shape does
+   too, and `expression` scales it.
+5. **Lens Studio head**: the skull, cheeks and ears grow as the weight rises. Your face stays on it
+   and the ears stay on the head.
+6. **facecap + face texture**: lower "face texture" and your face fades into facecap's skin. Your
+   eyes stay on its eyeballs until it is fully faded.
+7. **Head turn and dropout**: the shape turns with the head, holds through a short dropout, hides
+   when the face is lost, and comes back.

package/docs/api/face-painting.mdx

@@ -31,3 +31,10 @@ inspector controls to show for the active material state.
 Geometry visibility is grouped by FaceMesh part: left eye, right eye, mouth,
 and nose. Use `CANONICAL_FACE_MESH_GEOMETRY_SPECS` when an inspector needs the
 runtime group order instead of hardcoding another list.
+
+The `showLeftEye…` / `showRightEye…` settings are the person's own eyes on
+every feed (GROUNDING "Left and right"). The mesh's `"left-eye"` /
+`"right-eye"` groups are screen sides (`"left-eye"` is always the eye on the
+screen's left); the renderer maps the settings onto them with
+`resolveFaceMeshEyeVisibility(person, mirrored)`, which swaps them on
+unmirrored feeds.

package/docs/api/face-reprojection.mdx

@@ -0,0 +1,154 @@
+---
+title: "Face Reprojection"
+description: "The live camera face, worn by a face model in the model's own UV layout: the whole face, or only the eyes and mouth on a character."
+sidebarTitle: "Face Reprojection"
+icon: "masks-theater"
+---
+
+`FaceReprojection` puts the person's real face from the camera onto a face model
+([`ExternalFaceMesh`](/api/external-face-mesh)). The model can use any UV atlas.
+
+- **The whole face:** the person's face on the character's head.
+- **Only the eyes and / or mouth:** the classic talking-character effect.
+- **The eyes on the eyeballs** as well, if you choose.
+
+It is part of the face-character pipeline ([#194](https://github.com/vincentt-xr/sdk/issues/194),
+#199). Import from `@vincentt-xr/sdk/face-effects`.
+
+## Quick start
+
+```tsx
+import { ExternalFaceMesh, FaceReprojection } from "@vincentt-xr/sdk/face-effects";
+import { FaceTracker } from "@vincentt-xr/sdk/tracking";
+
+<FaceTracker>
+  <ExternalFaceMesh model={gltf.scene}>
+    <FaceReprojection region="eyesAndMouth" />
+  </ExternalFaceMesh>
+</FaceTracker>;
+```
+
+## How it works
+
+Two offscreen passes run per camera frame (at most `fps` a second):
+
+1. **Unwrap.** The camera face is laid out in the canonical face UV layout.
+   - Every landmark samples the camera image where it was tracked.
+   - The eye and mouth openings, which are holes in the canonical surface, are filled. So the eyes
+     and the inside of the mouth come along.
+   - Skin seen edge-on (a cheek in a profile view) fades out instead of smearing, and so does the
+     outer edge of the face.
+2. **Remap.** Every vertex of the model is bound once to its closest point on that filled canonical
+   face. The model is drawn in its own UV layout and reads the unwrap there. A few pixels of edge
+   padding stop seams at the edges of UV islands.
+
+The result is applied to the model's meshes through a patched clone of each material:
+
+- **unlit** (default): as the camera saw it;
+- **lit**: as the base colour, shaded by the scene lights.
+
+**Mirroring, aspect and resize** need nothing. The camera image and the landmarks share one space (a
+mirrored webcam is mirrored before tracking), and the unwrap follows the anatomy rather than the
+screen.
+
+## Props
+
+| Prop               | Default       | What it does                                                                                             |
+| ------------------ | ------------- | -------------------------------------------------------------------------------------------------------- |
+| `meshes`           | skin parts    | Which meshes wear it, by object name. They must share one atlas. Add the eyeballs for real eyes.         |
+| `region`           | `"face"`      | `"face"`, `"eyes"`, `"mouth"` or `"eyesAndMouth"` (soft areas around the openings, lids and lips in).    |
+| `amount`           | `1`           | How much of the camera face shows, 0..1.                                                                 |
+| `shading`          | `"unlit"`     | `"unlit"`: the camera's colours. `"lit"`: shaded by the scene lights.                                    |
+| `mask`             | —             | Red channel, in the model's UV layout (load it like the model's map; glTF textures use `flipY = false`). |
+| `canonicalMask`    | —             | Red channel, in the canonical face UV layout.                                                            |
+| `resolution`       | `1024`        | Output texture size, px (mipmapped).                                                                     |
+| `unwrapResolution` | `512`         | Canonical-layout texture size, px.                                                                       |
+| `padding`          | `2`           | Edge padding around UV islands, px (0–4).                                                                |
+| `fps`              | `30`          | The most updates a second (`0` = every new camera frame).                                                |
+| `facing`           | `[0.05, 0.3]` | Cosines: skin turned further than the first is hidden, fully kept from the second.                       |
+| `falloff`          | `[1, 3]`      | Distance from the canonical face (cm) over which a vertex stops taking the face.                         |
+| `apply`            | `true`        | Patch the meshes' materials. With `false`, only render; use `onTexture` with your own material.          |
+| `onTexture`        | —             | Receives `{ texture, unwrap }` once they exist.                                                          |
+
+**Eyes and lids.** Eyeballs (parts named `eye…`) read the eye openings. Skin and teeth read the face
+and the mouth opening, never the eyes. A lid's edge sits in the eye opening, and wearing the eye there
+would show when a blink morph closes the lid.
+
+**Face index and tracking.**
+
+- It uses the enclosing rig's face (`FaceRigState.faceIndex`, from `FaceRig` / `ExternalFaceMesh`
+  `faceIndex`).
+- It updates only on frames where that face is tracked, and keeps the last texture through a
+  dropout.
+- Both textures are freed on unmount.
+
+## Your own material or pipeline
+
+- **Your own material.** Pass `apply={false}` and use `onTexture`, or patch any built-in material
+  with `patchFaceReprojectionMaterial(material, createFaceReprojectionUniforms())`. The textures
+  hold premultiplied alpha.
+- **Without React.** `FaceReprojector` is the GPU engine:
+
+  - `setTargets([{ geometry, canonical, openings }])`;
+  - `render(renderer, cameraTexture, landmarks)` once per frame;
+  - read `texture` / `unwrapTexture`;
+  - `dispose()` when done.
+
+  It takes five more options for the unwrap, all off by default:
+
+  | Option          | Default              | Meaning                                                                                                                                                                               |
+  | --------------- | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+  | `uvs`           | `CANONICAL_FACE_UVS` | The unwrap's layout, one `[u, v]` per landmark. `FACE_ATLAS_UVS` unwraps into the face atlas; a `canonicalMask` must then be in that layout.                                          |
+  | `unwrapMipmaps` | `false`              | Mipmap the unwrap, for reading it smaller than it is.                                                                                                                                 |
+  | `mirrorFill`    | `false`              | Fill the far side of a turned face from the other side, once per unwrap; only where the mirror twin was seen clearly better. Points outside the image are filled from their twin too. |
+  | `inFrame`       | on with `mirrorFill` | Drop the parts of the face outside the image instead of stretching the edge pixels over them.                                                                                         |
+  | `unwrapPadding` | `0`                  | Padding of the unwrap, px (0–16). It pads alpha too: coverage grows at the face's edge and into unseen holes, so padded texels count as seen.                                         |
+
+  With `uvs`, `texture` (the remap into `setTargets`' own atlas) follows the layout, but the
+  canonical paths do not: `bindFaceReprojection(...).canonicalUv`, a `faceUv` attribute and
+  `patchFaceReprojectionMaterial(..., { attributes: true })` read the unwrap in the canonical
+  layout and would wear a scrambled face. Convert the binding first:
+  `writeFaceReprojectionLayoutUv(binding, FACE_ATLAS_UVS)` is the `faceUv` for that unwrap.
+
+- **Pure helpers.** `getFaceUnwrapMesh`, `getFaceUnwrapUvs` (the unwrap's UVs in a layout),
+  `writeFaceUnwrapFrame`, `writeFaceFacing` (how squarely each of the 471 unwrap vertices faces
+  the camera), `bindFaceReprojection`, `writeFaceReprojectionLayoutUv` and the region helpers.
+
+## Face swap and mirror
+
+To put a face on another live face instead of on a model, use [Face Swap](/api/face-swap):
+`FaceSwap` swaps two people's faces or puts a photo or painted face on every face, and
+`FaceMirror` mirrors your own face. They unwrap the face with the same `FaceReprojector`, into the
+face atlas (`FACE_ATLAS_UVS`) rather than the canonical layout, and draw it back on the tracked face
+in screen space. `/face-reprojection?mode=swap` and `?mode=mirror` show them.
+
+## Example
+
+`/face-reprojection` runs on the local facecap character (see
+[External Face Mesh](/api/external-face-mesh#example)). It has controls for region, shading,
+amount, facing, fps, resolution and face index, and an option to put your eyes on the eyeballs. It
+previews both textures in the top-right corner: the remap in the model's atlas, and the unwrap. Without the
+local model, only the unwrap shows.
+
+## Manual visual test plan
+
+Open `/face-reprojection` on the webcam with the local facecap model.
+
+1. **Whole face**: your face is on the character's skin, sharp and aligned, with no seam at the
+   edge. The ears, the scalp and the back of the head keep the character's own skin, and the edge
+   fades softly.
+2. **Head turn**: turn your head. The cheek turning away fades to the character's skin instead of
+   smearing. The near side stays sharp.
+3. **Expressions**: open your mouth (your teeth and tongue show in the character's mouth), smile and
+   raise your brows. Blink: the character blinks (its own morph) and the lids show your lids.
+4. **Eyes + mouth**: switch region to "eyes + mouth". Only your eyes and lips show, with soft edges,
+   on the character's grey skin. Tick "your eyes on the eyeballs" and your eyes move with your gaze.
+5. **Shading**: switch to "lit". The face takes the scene lighting. The amount slider fades it.
+6. **Mirror and source**: switch between the webcam and the sample videos and image. Each real eye
+   lands on the character's eye in front of it, on every source (mesh to mesh; GROUNDING "Left and
+   right").
+7. **Resolution / fps**: change the resolution (the previews resize) and the fps (lower updates
+   less often). Nothing leaks: switching back and forth keeps the frame rate.
+8. **Dropout**: cover the camera briefly. The face holds, then the character hides and comes back
+   with a fresh face.
+9. **Second face**: with two people, "second face" puts the second person's face on the character.

package/docs/api/face-retouching.mdx

@@ -52,22 +52,30 @@ space. They default to the full frame. The legacy compare demo uses
 
 For multiple people, render one `FaceRetouch` per person and pass `faceIndex`.
 `numFaces` defaults to `faceIndex + 1`, so `<FaceRetouch faceIndex={1} />`
-requests tracking for at least two faces. Multiple passes are transparent
-outside their own face mask, so they can be stacked over the same camera
-background.
+requests tracking for at least two faces. Each pass is see-through outside
+its own face mask, so they stack over the same camera background at the same
+draw order.
 
 ```tsx
 const faceCount = 2;
 
 {
   Array.from({ length: faceCount }, (_, faceIndex) => (
-    <FaceRetouch
-      key={faceIndex}
-      {...controls}
-      faceIndex={faceIndex}
-      numFaces={faceCount}
-      renderOrder={-1000 + faceIndex}
-    />
+    <FaceRetouch key={faceIndex} {...controls} faceIndex={faceIndex} numFaces={faceCount} />
   ));
 }
 ```
+
+## Draw order
+
+`FaceRetouch` draws in the opaque face layer at `FACE_RETOUCH_RENDER_ORDER`
+(-2.5, from `@vincentt-xr/sdk/face-effects`), with depth testing:
+
+- after the camera background (-1000) and `HandCutout` (-900), so a hand in
+  front of the face stays sharp instead of being smoothed;
+- before the face effects (`FaceWarp3D`, `FaceMorph`, characters at -2) and
+  makeup (-1.99 to -1.55), which draw over the retouched skin.
+
+Mount it at the top of the scene, not inside a tracker group (a group's
+render order would move it). Override `renderOrder` only to place a custom
+effect between the retouch and the face.