@vincentt-xr/sdk 2.0.0-alpha.6 → 2.0.0-alpha.8

package/docs/api/segmentation-overlay.mdx

@@ -0,0 +1,301 @@
+---
+title: "Segmentation Overlay (2D)"
+description: "Draw semantic segmentation regions (lips, eyes, brows, face skin, mouth, person, background, or any label an external model emits) as tinted 2D layers over the camera, without canonical 3D face geometry."
+sidebarTitle: "Segmentation Overlay"
+icon: "layer-group"
+---
+
+`SegmentationOverlay` renders one or more regions of a **semantic mask** as a
+screen-aligned 2D layer composited over the camera feed. Masks come from the
+SDK's own providers or from any external segmentation model (face parsing,
+matting), as plain data.
+
+Import doors: the component, both providers, settings and the `SemanticMask`
+type are in `@vincentt-xr/sdk/face-effects`; `SegmentationOverlay` and
+`usePersonMask` are also in `@vincentt-xr/sdk/tracking` for person /
+background work; the material object API, shaders, mask textures,
+rasterizer primitives and `computeMaskUvTransform` are
+`@vincentt-xr/sdk/low-level`.
+
+```tsx
+import {
+  SegmentationOverlay,
+  useFaceRegionMask,
+  usePersonMask,
+} from "@vincentt-xr/sdk/face-effects";
+
+const Lips = () => {
+  const faces = useFaceRegionMask(); // lips, mouth, eyes, brows, face_skin
+  return <SegmentationOverlay mask={faces} label="lips" color="#c0304f" blendMode="multiply" />;
+};
+
+const DimBackground = () => {
+  const person = usePersonMask(); // bg, person
+  return (
+    <SegmentationOverlay
+      mask={person}
+      label="bg"
+      color="#000000"
+      opacity={0.6}
+      blendMode="normal"
+    />
+  );
+};
+```
+
+## When to use 2D segmentation versus canonical 3D geometry
+
+| Need                                                                                                     | Use                                   |
+| -------------------------------------------------------------------------------------------------------- | ------------------------------------- |
+| A region tint, cut-out, or image stamped **in screen space**                                             | `SegmentationOverlay` (this page)     |
+| Regions with **no landmarks**: background and person (built in); hair, beard, ears with an external mask | `SegmentationOverlay` + a mask source |
+| Lipstick with highlights, gradients, UV-mapped designs that **follow the lip surface**                   | `LipEffect` (3D, canonical UVs)       |
+| Textures that must stay attached under head rotation, depth, occlusion                                   | `FaceMeshRenderer` / `LipEffect`      |
+
+The 2D path has no UV atlas and no per-vertex normals: a design texture is
+sampled in **screen** space (it does not rotate with the head), and lighting
+comes only from the camera pixels via the blend mode. It is cheaper (one
+full-screen quad, one small texture) and works for any label a mask carries.
+The 3D path needs the canonical mesh and landmark depth but gives stable
+texture mapping and real shading. Both can run at once.
+
+## The mask data model
+
+```ts
+type SemanticMask = {
+  width: number;
+  height: number;
+  indices: Uint8Array; // one class index per pixel, row 0 = top
+  labels: readonly string[]; // labels[index]; index 0 is conventionally "bg"
+  space?: "feed" | "source";
+  version: number; // bump on every change; consumers upload lazily
+};
+```
+
+Any model can feed this: write its per-pixel class ids into `indices` and
+name them in `labels`. `SEMANTIC_MASK_LABELS` lists what the SDK's own
+providers produce (`bg`, `brows`, `eyes`, `face_skin`, `lips`, `mouth`,
+`person`); any other string works, so a face-parsing model's `beard`, `ears`
+or `hair` render the same way. The SDK does not ship a model for those labels:
+landmarks carry no ear points and cannot tell whether facial hair is present,
+and the portrait segmenter only separates person from background. Helpers: `createSemanticMask`, `clearSemanticMask`, `fillPolygon`
+(normalized points, y down), `fillLandmarkPolygon`, `semanticMaskLabelIndex`.
+The mask object is meant to be mutated in place; bump `version` after writing.
+
+## Built-in providers
+
+- `useFaceRegionMask({ faceIndex = "all", regions, resolution = 256, maxUpdateFps, holdMs = 250, predictMs = 0 })`
+  rasterizes the face tracker's landmarks into `face_skin`, `brows`, `eyes`,
+  `lips` and `mouth` every frame (`FACE_REGION_POLYGONS` lists the landmark
+  loops; `rasterizeFaceRegions` is the pure rasterizer). `faceIndex: "all"`
+  stacks every tracked face into one mask; a number isolates one face.
+  Small or fast faces make the tracker drop out for a frame or two at a
+  time; `holdMs` keeps the last mask alive across such gaps (the same bridge
+  `useFaceInfos` applies), and `predictMs` moves the held landmarks along
+  their last smoothed velocity for up to that long so the mask keeps pace
+  with a moving subject instead of lagging behind it. Set `holdMs: 0` to
+  show raw tracker state. The bridge is `FaceTrackMemory`, usable with any
+  landmark set.
+- `usePersonMask({ maxUpdateFps = 15 })` exposes the portrait segmentation as
+  `bg` / `person`. It acquires the segmentation model itself. Each update is
+  a GPU→CPU readback, hence the default cap. The model downloads on first
+  use, so the mask stays at `version` 0 for a few seconds; an overlay draws
+  nothing until the first real mask arrives. The selfie segmenter is a
+  person-class model: every person in frame is `person`, so multi-person
+  frames work without configuration.
+- For `bg` and `person` prefer `blendMode="normal"` (a flat tint) or
+  `screen`; `multiply` darkens, which is right for lips on skin but nearly
+  invisible on a dark background.
+- `useHairMask({ modelUrl, quality, active })` runs MediaPipe's **Hair
+  Segmenter** and returns `{ mask, status, error, labels }` with labels
+  `bg` / `hair`. See [Hair segmentation](#hair-segmentation) below.
+- `useSemanticSegmentationMask({ preset, modelUrl, quality, active })` is
+  the generic form behind it; `preset: "multiclass"` runs the Selfie
+  Multiclass model (`bg`, `hair`, `body_skin`, `face_skin`, `clothes`,
+  `others`). One preset per session: the first acquirer decides.
+
+## Hair segmentation
+
+Hair is a dedicated region with its own model, because neither landmarks
+(no hair points) nor the portrait segmenter (person as one class) can
+produce it.
+
+**Model and runtime contract.** `XRModel.SEMANTIC_SEGMENTATION` runs a
+MediaPipe `ImageSegmenter` in `VIDEO` mode with `outputCategoryMask` on the
+GPU delegate. The `hair` preset is MediaPipe's Hair Segmenter
+(`hair_segmenter.tflite`, 512×512 input, classes `background` / `hair`). Its
+category mask carries the class index per pixel (0 background, 1 hair; the
+multiclass model likewise emits 0–5; both measured against the live models); the SDK's portrait selfie segmenter is
+different and emits 255 / 0, which `usePersonMask` normalises separately.
+By default the model loads from MediaPipe's published file, whose bytes are
+pinned in the SDK's integrity manifest, so the default is verified: a
+republished file fails loudly with an SRI mismatch instead of silently
+changing behaviour. The same hash is pinned under the SDK CDN path
+`/assets/xr/models/hair_segmenter.tflite`; once that file is uploaded, pass
+it as `modelUrl` (or the SDK will flip the default). `useHairMask()` with no
+options works:
+
+```tsx
+import { SegmentationOverlay, useHairMask } from "@vincentt-xr/sdk/face-effects";
+
+const HairTint = () => {
+  const hair = useHairMask();
+  if (hair.status === "error") console.warn(hair.error);
+  return (
+    <SegmentationOverlay
+      mask={hair.mask}
+      label="hair"
+      color="#7c3aed"
+      blendMode="overlay"
+      feather={2}
+    />
+  );
+};
+```
+
+**Lifecycle.** `status` is `loading` while the WASM and model download and
+the segmenter initialises, `ready` once inference runs, `error` if the asset
+fails verification or the segmenter throws (`error` carries the reason). An
+inference error is transient: the next successful frame returns the status
+to `ready`, so consumers always see the current state. `labels` is the
+preset's label list from the first render, before any data.
+The mask stays at `version` 0 until the first inference, so the overlay
+draws nothing during loading. The model is refcounted like every tracker:
+mount it once above scene switches, pause it with `active: false`, and it
+is disposed when the last consumer unmounts.
+
+**Coordinates and resolution.** The feed is stretched into a process canvas
+at `quality` × feed size (low 0.5, medium 0.75, high 1), so the mask is in
+feed space with the feed's aspect ratio and mirroring, exactly like the
+other providers; the overlay needs no coordinate setup. The mask is
+nearest-filtered at the process resolution; use `feather` for soft hair
+edges and `quality: "high"` when flyaway strands matter.
+
+**Performance.** Inference runs at the model's `targetFps` (15 by default,
+overridable through the session `perf` / pipeline options). Each inference
+reads the category mask back to the CPU, which is why the rate is capped;
+the hair model is heavier than the portrait segmenter, so prefer `medium`
+quality on phones. Only one semantic segmenter runs per session; use the
+`multiclass` preset when you need hair and clothes together.
+
+**Hair versus the face mesh and the portrait segmenter.** The face mesh is
+geometry: it follows landmarks, supports UV-mapped textures and lighting,
+and knows nothing about hair. The portrait segmenter answers "is this pixel
+a person", which includes hair, clothes and skin together. The hair
+segmenter answers "is this pixel hair", so it is the right source for hair
+tints, recolours, highlights, and for occluding or cutting out hair in a
+composite, all as 2D screen-space effects.
+
+Both return a stable object in **feed space**.
+
+## Coordinates, mirroring, aspect and resolution
+
+- **Feed space** is the session camera texture: what trackers see, already
+  cover-cropped to the viewport and mirrored for webcams. The overlay quad
+  covers the viewport exactly like `VideoBackground`, so a feed-space mask
+  needs no coordinate work. Row 0 of the mask is the top of the screen.
+- **Source space** (`space: "source"`) is the raw media frame. The overlay
+  cover-fits it to the feed (centred crop) using the mask's own aspect ratio
+  and the camera texture size, via `computeMaskUvTransform`. Set `mirrored`
+  when the feed is mirrored relative to that source (a webcam selfie view;
+  image and video sources are not mirrored).
+- **Resolution** is independent of the viewport: masks are nearest-filtered
+  class indices, so edges are as sharp as the mask. Raise `resolution` (long
+  edge) for crisper edges or use `feather` (in mask texels) to soften them.
+- Landmarks are normalized to the feed, so the face providers are resolution
+  and orientation agnostic; rotating the device re-crops the feed and the
+  mask follows.
+
+## Rendering
+
+Props are `Partial<SegmentationOverlaySettings>` plus `mask`, `texture`,
+`material`, `renderOrder` (default just above the video background) and
+`visible`.
+
+| Setting     | Meaning                                                                     |
+| ----------- | --------------------------------------------------------------------------- |
+| `label`     | Region(s) to show; comma-separate up to four (`"lips,mouth"`)               |
+| `invert`    | Show everything except the selected labels                                  |
+| `color`     | Tint; with `texture`, multiplied into the image                             |
+| `opacity`   | Layer opacity                                                               |
+| `blendMode` | `normal` (flat paint), `multiply` (keeps skin shading), `screen`, `overlay` |
+| `feather`   | Edge softening radius in mask texels (0 = hard class edges)                 |
+| `mirrored`  | Flip the mask horizontally                                                  |
+
+Output is a transparent layer, so other content composites normally above
+and below it. `texture` is sampled in screen UV (cover of the viewport);
+`resolveSegmentationOverlaySettings` / `patchSegmentationOverlaySettings`
+resolve partial settings, and `SegmentationOverlaySettingsPanel` from
+`@vincentt-xr/sdk/debug-ui` edits them live.
+
+### Custom materials
+
+`material` accepts a node, a `THREE.Material` instance (shareable), or a
+factory receiving:
+
+```ts
+type SegmentationOverlayMaterialContext = {
+  cameraTexture: THREE.Texture | null;
+  maskTexture: THREE.Texture; // red channel = class index / 255, nearest-filtered
+  maskTransform: THREE.Matrix3; // feed UV → mask UV
+  labelIndices: number[]; // the selected classes
+  mask: SemanticMask | null;
+  settings: SegmentationOverlaySettings;
+};
+```
+
+Sample membership as `abs(floor(texture2D(maskTexture, (maskTransform *
+vec3(uv, 1.0)).xy).r * 255.0 + 0.5) - index) < 0.5`. `labelIndices` is empty
+until the mask has data, and `maskTexture` is re-allocated when the mask
+changes size (the factory is re-run with the new one), so read both from the
+context on every call rather than caching them. The built-in shader is
+exported as `SEGMENTATION_OVERLAY_VERTEX_SHADER` /
+`SEGMENTATION_OVERLAY_FRAGMENT_SHADER`, and the material object API
+(`createSegmentationOverlayMaterial`, `updateSegmentationOverlayMaterial`,
+`useSegmentationOverlayMaterial`) lets you share one material.
+
+## Multiple faces
+
+`useFaceRegionMask({ faceIndex: "all" })` merges every face into one mask, so
+one overlay paints all lips. For per-face looks, create one provider per face
+(`faceIndex: 0`, `faceIndex: 1`) and one overlay each. Match the tracker's
+`numFaces`. `usePersonMask` is per scene, not per face.
+
+## Performance
+
+- One overlay is one full-screen quad and one small `R8` texture upload per
+  mask version. Combine labels (`label="lips,mouth"`) rather than stacking
+  overlays when they share a look.
+- Face masks default to a 256 px long edge; rasterizing a few landmark
+  polygons at that size costs well under a millisecond. Lower `resolution`
+  or set `maxUpdateFps` on the provider to trade sharpness for CPU.
+- `usePersonMask` reads the segmentation mask back from the GPU; keep its
+  `maxUpdateFps` low (15 by default) and share one provider between overlays.
+- The session's `perf={{ targetFps, camRes }}` still governs tracking cost.
+
+## Manual visual test plan
+
+Run `/segmentation-overlay` in the playground and check each row with the
+region picker, on a webcam and on an image/video source:
+
+| Mask / region                       | Source                                   | Expect                                                                                                                             |
+| ----------------------------------- | ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
+| `lips` (face provider)              | webcam, video                            | Tint hugs the outer lip line; mouth interior untinted; follows talking                                                             |
+| `mouth`                             | video, mouth open                        | Only the opening is painted                                                                                                        |
+| `eyes`, `brows`                     | video                                    | Both eyes / both brows painted; nothing on the lids' skin                                                                          |
+| `face_skin`                         | video                                    | Oval painted; eyes, brows, lips and mouth cut out                                                                                  |
+| `bg` (person provider)              | video                                    | Everything but the person painted; silhouette edge follows the mask                                                                |
+| `person`, `invert` on               | video                                    | Same as `bg`                                                                                                                       |
+| `mirrored` toggle                   | webcam                                   | Feed-space masks drift off the face when mirrored (expected); off again aligns                                                     |
+| portrait vs landscape pane          | resize the pane                          | Regions stay on the face after the feed re-crops                                                                                   |
+| `feather` 0 → 4                     | any                                      | Class edges go from stepped to soft                                                                                                |
+| `hold` 0 vs 250, `predict` 0 vs 150 | "Person Fullbody", 2–8 s                 | At 0 the regions blink while the far-away face is lost and re-found; with hold they stay on, and with predict they follow the walk |
+| `texture` on                        | any                                      | Image appears in screen space, clipped to the region                                                                               |
+| custom stripes material             | any                                      | Stripes clipped to the region, still following the face                                                                            |
+| two faces                           | "Two People Smiling"                     | `All faces`: both painted; `Face 0 only`: one                                                                                      |
+| `hair`, short dark hair             | "Person ThumbUp"                         | Hair painted to the hairline; no bleed onto forehead or background                                                                 |
+| `hair`, long hair, tied             | "Person Fullbody", "Person Shaking Head" | Painted mass follows head turns and shakes; strands at the edges soften with `feather`                                             |
+| `hair`, two people                  | "Two People Smiling"                     | Both heads painted; the mask is per pixel, not per face                                                                            |
+| `hair`, lighting                    | webcam, bright vs dim room               | Mask holds under both; dark hair on a dark background is the hard case                                                             |
+| `hair`, orientation                 | webcam, rotate device                    | Mask re-crops with the feed and stays on the hair                                                                                  |
+| `hair`, `status`                    | load with wrong `modelUrl`               | Status line reads `error` with the reason; nothing is painted                                                                      |

package/docs/api/session-lifecycle.mdx

@@ -0,0 +1,136 @@
+---
+title: "Capabilities & Session Lifecycle"
+description: "Check what the device can do before a session starts, read structured fallback reasons, and drive a hardened start / pause / resume / stop / reset lifecycle that survives camera loss and tab switches."
+sidebarTitle: "Capabilities & Lifecycle"
+icon: "heart-pulse"
+---
+
+Two things decide whether an AR experience works on a given phone: whether the
+platform can run it at all, and whether the session stays healthy once it
+runs. The SDK answers the first with a capability check that never opens the
+camera, and the second with a session state machine that releases resources
+exactly once and recovers from a lost camera.
+
+## Capability detection
+
+```tsx
+import { useXRCapabilities, resolveTrackerSupport, XRModel } from "@vincentt-xr/sdk";
+
+const { capabilities, status, refresh } = useXRCapabilities({
+  models: [XRModel.FACE_TRACKER], // optional: HEAD-probe each tracker's asset
+});
+if (status === "ready" && capabilities) {
+  const face = resolveTrackerSupport(capabilities, XRModel.FACE_TRACKER);
+  if (!face.supported) showFallback(face.reason); // { code, message, recoverable }
+  if (face.needsPermission) showCameraPrimer(); // the first getUserMedia will prompt
+}
+```
+
+`useXRCapabilities` needs no `<XRProvider>`, so it runs on a landing screen.
+`detectXRCapabilities(options)` is the same check as a plain promise.
+
+The report (`XRCapabilities`):
+
+| Field            | What it checks                                                                                                                                                                         |
+| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `camera`         | Secure context, `getUserMedia` presence, number of video inputs, Permissions API state. `status` is `supported`, `permission-prompt`, `permission-denied`, `unsupported` or `unknown`. |
+| `rendering`      | WebGL version (`0`/`1`/`2`), OffscreenCanvas, WebAssembly, WebGPU presence (informational).                                                                                            |
+| `models`         | For each `XRModel` in `models`: a `HEAD` request to its asset (status, reason, byte size). Unlisted models report `unknown`.                                                           |
+| `canRunTracking` | Rendering is supported and the camera is not unsupported.                                                                                                                              |
+
+`permission-denied` and `unsupported` are distinct: the first is fixable by the
+viewer (recoverable), the second is not. The Permissions API is consulted
+before the device list, so a browser that hides devices until permission is
+granted still reads as denied, not as "no camera".
+
+### Fallback reasons
+
+Every degraded state in the SDK is described by one `XRFallbackReason`:
+
+```ts
+type XRFallbackReason = { code: XRFallbackCode; message: string; recoverable: boolean };
+```
+
+| Code                                                                    | Source                     |
+| ----------------------------------------------------------------------- | -------------------------- |
+| `camera-permission-denied`, `camera-in-use`, `no-camera`                | camera check / session     |
+| `camera-insecure-context`, `camera-unsupported`                         | camera check               |
+| `camera-lost`                                                           | session (track ended)      |
+| `webgl-unavailable`, `offscreen-canvas-unavailable`, `wasm-unavailable` | rendering check            |
+| `model-unreachable`, `model-integrity`, `model-init-failed`             | model probe / `useXRModel` |
+
+Branch on `code`; `message` is display prose. `resolveTrackerSupport` returns
+the most fundamental blocking reason first (rendering, then camera, then the
+model).
+
+### Tracker consumers
+
+Inside a session, components that do not own a tracker read its status:
+
+```tsx
+import { useXRTrackerStatus } from "@vincentt-xr/sdk/low-level";
+
+const { status, fallback } = useXRTrackerStatus(XRModel.FACE_TRACKER);
+// status: "unavailable" | "loading" | "ready" | "blocked" | "error"
+```
+
+`blocked` means the camera failed (the session error maps to `fallback`);
+`error` means the model's own initialize failed (`useXRModelErrors()` lists
+them; `useXRModel().fallback` carries the same reason for the owner).
+
+## Session lifecycle
+
+```
+idle ──setMediaSource──> media-bound ──start──> running ──stop──> stopped
+                              ↑                 │    ↑               │
+                              │           pause │    │ resume         │
+                              │                 ▼    │               │
+                              │               paused ─┘               │
+                              └────────────reset──────────────────────┘
+```
+
+| Call               | Effect                                                                                                                                                          |
+| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `session.pause()`  | `running` → `paused`: feed loop stops, every model pauses, the video element pauses. The stream stays open. No-op elsewhere.                                    |
+| `session.resume()` | `paused` → `running` with no rebind. `start()` from `paused` does the same.                                                                                     |
+| `session.stop()`   | Releases the stream and tracks, keeps the models initialized. `start()` afterwards needs a `setMediaSource()` first; binding the same source again re-opens it. |
+| `session.reset()`  | `stop()` + release every model; back to `idle`. Clears errors.                                                                                                  |
+| provider unmount   | The facade is disposed: `reset()` plus listener removal. Every later call throws. Models and tracks are released exactly once.                                  |
+
+`useXRLifecycle()` (from `/low-level`) subscribes to the state; the example's
+log is built from it.
+
+### Camera loss and permission changes
+
+When a live video track ends (device unplugged, permission revoked, another
+app took it) or the camera permission flips to `denied`, the session releases
+the media once, suspends every model, publishes a session error with
+`reason: "camera-lost"` (or `"permission-denied"`) and parks in `stopped`.
+Recovery is the normal path: `setMediaSource()` then `start()`. A track that
+the session itself replaced during a source swap never triggers this.
+
+### Visibility
+
+A hidden tab throttles `requestAnimationFrame` and iOS Safari pauses the camera
+track. The session always retries `video.play()` when the tab returns. With
+`<XRProvider lifecycle={{ pauseWhenHidden: true }}>` it also pauses the whole
+session while hidden and resumes on return. A session the app paused itself
+is left paused.
+
+### Source swaps while paused
+
+`setMediaSource()` on a paused session binds the new media but stays paused;
+`resume()` starts the loop against it.
+
+## Manual validation
+
+Run `/session-health`:
+
+| Case              | Steps                     | Expect                                                              |
+| ----------------- | ------------------------- | ------------------------------------------------------------------- |
+| Capability report | page load, "re-check"     | camera / rendering / model rows with reasons; `canRunTracking`      |
+| Pause / resume    | buttons                   | lifecycle log `running → paused → running`; feed fps 0 while paused |
+| Stop / start      | stop, start               | `start` throws (needs a source); pick a media source to recover     |
+| Camera loss       | "simulate camera loss"    | `error camera-lost`, lifecycle `stopped`, trackers `blocked`        |
+| Recovery          | pick another media source | lifecycle `running`, error cleared                                  |
+| Route change      | navigate away and back    | no leaked stream; the page starts clean                             |

package/docs/api/trackers.mdx

@@ -28,13 +28,31 @@ import {
   `scaleWithFace`, and `followFaceRoll`
 - `HandTracker`: 21-joint skeleton; use `hand` or `handIndex` for selection
 - `GestureTracker`: named gesture recognition
+- `HolisticTracker`: one inference for face + hands + pose with
+  `HolisticTracker.Face` / `.Hand` / `.Body` sub-trackers; see
+  [Holistic Tracker](/api/holistic-tracker)
 - `BodyTracker`: 33 pose landmarks
 - `Segmentation`: portrait mask for background replacement
+- `useHairMask` / `useSemanticSegmentationMask` (hair or multiclass category
+  masks for 2D overlays; see [Segmentation Overlay](/api/segmentation-overlay))
 
 The example app route `/body` shows the recommended body-tracking pattern:
 mount `BodyTracker` once, then attach visible content with `TrackingAnchor`
 targets such as `body.head`, `body.leftShoulder`, and `body.rightWrist`.
 
+For a low-level hand, gesture, or body landmark, SDK tracker coordinates have a
+center origin, X right, Y up, and positive Z toward the viewer. X/Y occupy the
+usual `[-1, 1]` tracker range; Z is tracker-relative depth and is not a
+normalized screen axis. Convert a point to `ScreenImage`, `ScreenText`, or
+`ScreenShape` design pixels with `trackerPointToScreenPixels` from
+`@vincentt-xr/sdk`. Do not mirror, cover-map, or origin-flip it again.
+
+This is distinct from a raw MediaPipe face landmark: face landmarks use a
+top-left normalized image plane and are consumed by `FaceTracker`,
+`FaceMesh`, and `FaceMeshRenderer` internally. Attach 3D face content with
+`TrackingAnchor`; do not pass a raw face landmark to
+`trackerPointToScreenPixels`.
+
 Gesture names are exactly `closed_fist`, `open_palm`, `pointing_up`,
 `thumb_down`, `thumb_up`, `victory`, `i_love_you`, and `none`. Use
 `GestureTrigger` for one-shot callbacks and `useXRModelNode` from

package/docs/api/tracking-config.mdx

@@ -0,0 +1,108 @@
+---
+title: "Persistent Tracking Configuration"
+description: "A serializable, validated, versioned document for tracker selection and tuning (fps, camera, smoothing, prediction, assets, fallbacks), projected onto the provider, trackers and model options."
+sidebarTitle: "Tracking Config"
+icon: "sliders"
+---
+
+```tsx
+import { XRProvider, useXRTrackingConfig } from "@vincentt-xr/sdk";
+import { FaceTracker } from "@vincentt-xr/sdk/tracking";
+
+const App = () => {
+  // Above the provider: perf / lifecycle / quality come from the document.
+  const tracking = useXRTrackingConfig({
+    storageKey: "myapp.tracking",
+    initial: { trackers: { hand: { enabled: true } } },
+  });
+  const face = tracking.trackerProps("face");
+  // Wait for the persisted document before mounting the provider: a different
+  // perf / camera after hydration would re-initialize it (camera restart).
+  if (!tracking.hydrated) return null;
+  return (
+    <XRProvider {...tracking.providerProps}>
+      <XRScene>
+        {face.active && <FaceTracker numFaces={face.numFaces} targetFps={face.targetFps} />}
+      </XRScene>
+    </XRProvider>
+  );
+};
+
+tracking.update({ perf: { targetFps: 20 }, camera: { facingMode: "environment" } });
+tracking.reset();
+tracking.json; // the serialized document
+tracking.importJson(text);
+```
+
+## The document (`XRTrackingConfig`, version 1)
+
+| Section     | Fields                                                                                                                                                                                                                                                                                                                                                                     |
+| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `trackers`  | `face { enabled, numFaces 1..10, targetFps \| null, holdMs, predictMs, smoothing }`, `hand / gesture / body { enabled, targetFps \| null }`, `holistic { enabled, targetFps, hands, holdMs, predictMs, analysisMaxDimension, outputFaceBlendshapes, confidence 0..1, delegate }`, `segmentation { enabled, quality }`, `semanticSegmentation { enabled, preset, quality }` |
+| `perf`      | `targetFps` (0 = uncapped), `camRes`, `cameraZoom`, `analysisScale`                                                                                                                                                                                                                                                                                                        |
+| `lifecycle` | `pauseWhenHidden`                                                                                                                                                                                                                                                                                                                                                          |
+| `quality`   | an adaptive-quality partial config, or `false`                                                                                                                                                                                                                                                                                                                             |
+| `camera`    | `facingMode`, `deviceId \| null`, `mirrored: boolean \| "auto"`                                                                                                                                                                                                                                                                                                            |
+| `fallbacks` | `onUnsupported: "block" \| "degrade"`, `delegate: "auto" \| "GPU" \| "CPU"`                                                                                                                                                                                                                                                                                                |
+| `assets`    | `holisticModelUrl`, `semanticSegmentationModelUrl` (http(s) or `null` for the SDK default)                                                                                                                                                                                                                                                                                 |
+| `updatedAt` | wall-clock ms of the last change                                                                                                                                                                                                                                                                                                                                           |
+
+`targetFps: null` on a tracker means "inherit `perf.targetFps`".
+
+## What is excluded, on purpose
+
+Runtime resources never enter the document: media streams, textures, model
+instances, the session state, calibration measurements (they have their own
+store, `useXRCalibration`), quality decisions, and permission state.
+`serializeTrackingConfig` re-resolves the object, so anything attached at
+runtime is dropped on the way out, and `parseTrackingConfig` drops unknown
+fields on the way in.
+
+## Validation and defaults
+
+`resolveTrackingConfig(input, base?)` fills every field from `base`
+(`DEFAULT_TRACKING_CONFIG` unless given), type-checks and clamps each leaf,
+and reports dropped or clamped values as `warnings` instead of throwing.
+`DEFAULT_TRACKING_CONFIG` is the template's shipping default: face only,
+`perf { targetFps 24, camRes 720, cameraZoom 1.2 }`, adaptive quality on,
+front camera, `degrade` on unsupported devices.
+
+## Migration
+
+`parseTrackingConfig(json)` accepts any version: a document from an older
+(or missing) `version` is migrated field-by-field onto the current
+defaults, keeping the fields that still exist, with `migrated: true` and a
+warning. Corrupt JSON yields the defaults. `useXRTrackingConfig` re-saves a
+migrated document immediately, so the next load is clean. Bump
+`TRACKING_CONFIG_VERSION` and extend the parse path when the shape changes.
+
+## Projections
+
+- `toProviderProps(config)` → `{ perf, lifecycle, quality }` for `<XRProvider>`.
+- `toTrackerProps(config, kind)` → props for `<FaceTracker>` (`numFaces`, `targetFps`, `holdMs`, `predictMs`, `active`), `<HandTracker>`, `<BodyTracker>`, `<GestureTracker>`, `<HolisticTracker>` (incl. `modelUrl`, resolved `delegate`), `<Segmentation>`.
+- `toCameraRequest(config)` → `{ facingMode, deviceId?, mirrored? }` for `setMediaSource` / `switchCamera`.
+- `toModelOptions(config, model)` → acquire options for `useXRModel` / `registry.acquire`.
+- `enabledModels(config)` → the `XRModel[]` to probe with `useXRCapabilities` or preload.
+
+`fallbacks.delegate` replaces a tracker's `"auto"` delegate; `fallbacks.onUnsupported`
+is advisory for the app's own gate (see Capabilities).
+
+## Storage
+
+`loadTrackingConfig` / `saveTrackingConfig` / `clearTrackingConfig` use
+`localStorage` under `vincentt.xr.tracking-config` by default, with an
+injectable storage for tests or a server-backed store. The hook persists on
+every `update`, clears on `reset`, and exposes `hydrated`.
+
+## Manual validation
+
+Run `/tracking-config`:
+
+| Case     | Steps                                                                | Expect                                                  |
+| -------- | -------------------------------------------------------------------- | ------------------------------------------------------- |
+| Edit     | toggle hand, set face.numFaces 2                                     | diagnostics shows hand ready; JSON updates              |
+| Persist  | "reload page"                                                        | the same values come back; `persisted`                  |
+| Validate | numFaces 40 via "edit JSON" + import                                 | clamped to 10 with a warning                            |
+| Migrate  | import `{ "version": 0, "trackers": { "face": { "numFaces": 3 } } }` | numFaces 3, warning "migrated from version 0"           |
+| Camera   | facingMode environment (phone)                                       | rear camera opens straight; mirrored override respected |
+| Reset    | "reset"                                                              | defaults, storage cleared                               |

package/docs/docs.json

@@ -29,18 +29,35 @@
         "pages": [
           "api/runtime-3d",
           "api/screen-space-coordinates",
+          "api/coordinate-spaces",
           "api/screen-image",
           "api/screen-text",
           "api/screen-shape",
           "api/trackers",
+          "api/holistic-tracker",
           "api/face-info",
+          "api/iris-tracking",
           "api/face-actions",
+          "api/gestures",
+          "api/session-lifecycle",
+          "api/product-ar",
+          "api/product-details",
+          "api/tracking-config",
+          "compatibility",
+          "api/camera-permission",
+          "api/cross-camera",
+          "api/adaptive-quality",
+          "api/multi-subject",
+          "api/asset-lifecycle",
+          "api/diagnostics-calibration",
           "api/face-attribute-expression",
           "api/face-binding-anchors",
           "api/face-effects",
           "api/face-mesh",
           "api/face-painting",
           "api/face-retouching",
+          "api/segmentation-overlay",
+          "api/person-cutout",
           "api/capture-recording",
           "api/audio",
           "api/texture-sources",
@@ -60,10 +77,15 @@
         "icon": "plug",
         "pages": ["guides/harness-debug-ui"]
       },
+      {
+        "group": "Creator Guides",
+        "icon": "palette",
+        "pages": ["guides/lip-effect-textures"]
+      },
       {
         "group": "Examples",
         "icon": "code",
-        "pages": ["examples/scene-object-recipes"]
+        "pages": ["examples/scene-object-recipes", "examples/lip-shaders"]
       }
     ]
   }