@vincentt-xr/sdk 2.0.0-alpha.6 → 2.0.0-alpha.8

package/docs/api/gestures.mdx

@@ -0,0 +1,173 @@
+---
+title: "Gestures"
+description: "Reusable face, hand and body gesture detection with thresholds, hysteresis, confidence, cooldown and lifecycle events."
+sidebarTitle: "Gestures"
+icon: "hand-sparkles"
+---
+
+`useGesture` turns raw tracking into stable gesture events: pinch, grab,
+open palm, wave, hand raise, nod, head shake, mouth open, smile, blink,
+brow raise, arm raise and T-pose, plus your own. One engine handles the
+thresholds, hysteresis, hold, cooldown, confidence and tracking loss, so
+scenes stop re-implementing them.
+
+```tsx
+import { GestureListener, useGesture } from "@vincentt-xr/sdk/tracking";
+
+// Hook: React state that changes only when the gesture's state changes.
+const pinch = useGesture("pinch", { hand: "right", onBegin: () => grab(), onEnd: () => drop() });
+if (pinch.active) {
+  /* … */
+}
+
+// Declarative: mount and act on events.
+<GestureListener gesture="headNod" onBegin={() => confirm()} />;
+```
+
+Each gesture acquires the shared face, hand or body model it needs; no
+second detector starts. Mount gesture hooks once above scene switches, like
+any tracker.
+
+## Built-in gestures
+
+| Name                               | Kind | Measures                                                     | Default enter / exit / hold / cooldown |
+| ---------------------------------- | ---- | ------------------------------------------------------------ | -------------------------------------- |
+| `mouthOpen`                        | face | `jawOpen` blendshape                                         | 0.5 / 0.35 / 0 / 0                     |
+| `smile`                            | face | mean of `mouthSmileLeft` / `mouthSmileRight`                 | 0.4 / 0.25 / 0 / 0                     |
+| `blink`                            | face | both eyes closed (min of the `eyeBlink` blendshapes)         | 0.55 / 0.4 / 50 ms / 0                 |
+| `browRaise`                        | face | max of the brow-up blendshapes                               | 0.45 / 0.3 / 0 / 0                     |
+| `headNod`                          | face | pitch oscillation: 3 turning points ≥ 0.12 rad within 1.5 s  | 0.99 / 0.3 / 0 / 600 ms                |
+| `headShake`                        | face | yaw oscillation: 3 turning points ≥ 0.15 rad within 1.5 s    | 0.99 / 0.3 / 0 / 600 ms                |
+| `pinch`                            | hand | thumb–index tip gap relative to palm size (1 = touching)     | 0.75 / 0.5 / 0 / 0                     |
+| `grab`                             | hand | mean curl of index..pinky (1 = fist)                         | 0.7 / 0.45 / 0 / 0                     |
+| `openPalm`                         | hand | mean extension of index..pinky                               | 0.7 / 0.45 / 0 / 0                     |
+| `handRaised`                       | hand | wrist height in the frame (0 bottom, 1 top)                  | 0.7 / 0.6 / 0 / 0                      |
+| `oneFinger`                        | hand | index extended, middle..pinky curled (pointing)              | 0.65 / 0.45 / 0 / 0                    |
+| `peaceSign`                        | hand | index + middle extended, ring + pinky curled                 | 0.65 / 0.45 / 0 / 0                    |
+| `threeFingers`                     | hand | index + middle + ring extended, pinky curled                 | 0.65 / 0.45 / 0 / 0                    |
+| `fourFingers`                      | hand | index..pinky extended, thumb tucked                          | 0.65 / 0.45 / 0 / 0                    |
+| `fist`                             | hand | all four curled and the thumb tucked                         | 0.65 / 0.45 / 0 / 0                    |
+| `palm`                             | hand | all four extended and the thumb out                          | 0.65 / 0.45 / 0 / 0                    |
+| `thumbUp` / `thumbDown`            | hand | thumb out and pointing up / down, fingers curled             | 0.6 / 0.4 / 0 / 0                      |
+| `wave`                             | hand | wrist x oscillation: 3 turning points ≥ 0.08 within 1.2 s    | 0.99 / 0.3 / 0 / 600 ms                |
+| `armRaisedLeft` / `armRaisedRight` | body | wrist above its shoulder, in shoulder widths (person's side) | 0.3 / 0.15 / 0 / 0                     |
+| `armRaised` / `bothArmsRaised`     | body | either / both wrists above their shoulders                   | 0.3 / 0.15 / 0 / 0                     |
+| `tPose`                            | body | both arms level and outstretched by a shoulder width         | 0.6 / 0.4 / 150 ms / 0                 |
+
+Hand measures are scale-free (divided by the palm size) and body measures
+are in shoulder widths, so distance from the camera does not change the
+thresholds. `GESTURES[name].description` carries each line of the table.
+
+The finger-count signs score the _weakest_ required finger, so a pattern is
+1 only when every finger agrees: `peaceSign` needs index and middle extended
+_and_ ring and pinky curled. `grab` and `openPalm` are the softer, mean-curl
+versions of `fist` and `palm` that ignore the thumb; `measureFingerPattern`
+builds any other pattern (`{ fingers: [true, null, null, false], thumb: true }`
+uses `null` for "either").
+
+## The engine contract
+
+Every frame the definition measures a `{ value, confidence }` sample (or
+`null` when the tracker has no subject). The detector then applies, in order:
+
+1. **Confidence gate**: a sample below `minConfidence` (default 0.5), or no
+   sample, is tracking loss.
+2. **Loss grace**: an active gesture survives loss for `lostGraceMs`
+   (default 250, the SDK's hold window) with its value frozen, then ends
+   with `endReason: "lost"`. An inactive gesture just resets its hold timer.
+3. **Hysteresis**: inactive → `begin` when `value >= enterThreshold`;
+   active → `end` (`endReason: "released"`) when `value <= exitThreshold`.
+   The band between them is where the gesture is stable; the default exit is
+   0.15 below enter.
+4. **Hold**: the value must stay at or above enter for `holdMs` before
+   `begin`; a dip resets the clock.
+5. **Cooldown**: after `end`, no `begin` for `cooldownMs`.
+
+A `begin` is always followed by at least one `stay` frame, so a one-frame
+gesture still surfaces both edges to an effect that watches `justBegan` and
+`justEnded`.
+
+`GestureState`: `active`, `value`, `confidence`, `tracked`, `phase`
+(`begin` | `stay` | `end` | `none`), `justBegan`, `justEnded`, `endReason`
+(`released` | `lost` | null), `startedAtMs`, `durationMs`, `count`. The hook
+publishes a new state only when one of these changes, and `durationMs` is
+excluded from that comparison so an active gesture does not re-render every
+frame: it reads as of the last published change (an ending gesture carries
+its full length). Derive a live duration from `startedAtMs`.
+
+Confidence is presence-based for the face, hand and body trackers, which
+report no per-landmark score: 1 while the subject is tracked. A custom
+definition can supply a real score (the holistic result's `confidence`, for
+example).
+
+## Options
+
+`useGesture(gesture, options)` takes the detector options above plus
+`active`, `faceIndex` / `numFaces` (face), `hand` / `handIndex` (hand),
+`bodyIndex` (body), `targetFps`, and `onBegin` / `onEnd` / `onChange`
+callbacks, which fire from the frame loop with that frame's state.
+
+```tsx
+const nod = useGesture("headNod", { faceIndex: 1, numFaces: 2, cooldownMs: 1000 });
+const wave = useGesture("wave", { hand: "left", lostGraceMs: 400 });
+```
+
+Per-gesture defaults are the ones in the table; `resolveGestureDetectorOptions(overrides, defaults)`
+is what the hook applies.
+
+## Custom gestures
+
+```ts
+import { defineGesture, HAND_JOINTS } from "@vincentt-xr/sdk/tracking";
+
+// A "thumbs up": thumb tip well above the wrist while the other fingers curl.
+const thumbsUp = defineGesture({
+  name: "thumbsUp",
+  kind: "hand",
+  description: "Thumb up, fingers curled.",
+  defaults: { enterThreshold: 0.7, exitThreshold: 0.5 },
+  measure: (hand) => {
+    if (!hand || hand.length < 21) return null;
+    const lift = hand[HAND_JOINTS.thumbTip].y - hand[HAND_JOINTS.wrist].y;
+    return { value: Math.min(1, Math.max(0, lift / 0.3)), confidence: 1 };
+  },
+});
+
+const up = useGesture(thumbsUp);
+```
+
+Temporal gestures add `createMemory` and read it in `measure`; the built-in
+`OscillationTracker` (on `/low-level`) counts direction reversals for nods,
+shakes and waves.
+
+## Relation to the other gesture APIs
+
+- `<GestureTrigger>` fires once on a **named static hand pose** from the
+  gesture-recognizer model (`victory`, `thumb_up`, …). Use it for "show ✌️ to
+  start". `useGesture` measures **continuous and temporal** gestures from the
+  landmark trackers with thresholds you can tune.
+- `useFaceAction` is the face-only predecessor with the same hysteresis idea;
+  the face gestures here reuse its measurements, and add cooldown,
+  confidence, loss grace and the temporal nod / shake.
+
+## Tracking loss and confidence
+
+Losing the subject never leaves a gesture stuck: within `lostGraceMs` the
+state holds (value frozen, `tracked: false`), after it the gesture ends with
+`endReason: "lost"` so an effect can clean up differently from a deliberate
+release. A subject that reappears starts cold: hold timers restart and
+temporal memories reset, so a nod half-done before the dropout does not
+complete on return.
+
+## Manual validation
+
+Run `/gestures`:
+
+| Case          | Steps                                   | Expect                                                          |
+| ------------- | --------------------------------------- | --------------------------------------------------------------- |
+| Hysteresis    | hold a half pinch around the threshold  | Row stays active until the value drops below the exit threshold |
+| Hold          | set hold 300 ms on `pinch`, tap quickly | No begin for taps shorter than 300 ms                           |
+| Cooldown      | `headNod`, nod twice quickly            | One begin; the second counts after 600 ms                       |
+| Tracking loss | cover the hand mid-pinch                | Row shows "lost" then ends with `lost` after the grace window   |
+| Confidence    | set min confidence above 1              | Gesture never begins                                            |
+| Body          | "Person Fullbody" clip, raise an arm    | `armRaised` / side rows activate                                |

package/docs/api/holistic-tracker.mdx

@@ -0,0 +1,159 @@
+---
+title: "Holistic Tracker"
+description: "One MediaPipe inference for face, hands and pose, with unified results, dropout bridging and the same TrackingAnchor targets as the individual trackers."
+sidebarTitle: "Holistic Tracker"
+icon: "person-rays"
+---
+
+`<HolisticTracker>` runs MediaPipe's Holistic Landmarker: a single graph that
+returns face landmarks and blendshapes, left and right hand landmarks, and the
+33-point pose for **one person**. It is additive and opt-in; `FaceTracker`,
+`HandTracker` and `BodyTracker` are unchanged.
+
+```tsx
+import { HolisticTracker, TrackingAnchor } from "@vincentt-xr/sdk/tracking";
+
+<HolisticTracker holdMs={250}>
+  <HolisticTracker.Face>
+    <TrackingAnchor target="face.noseTip">{/* … */}</TrackingAnchor>
+  </HolisticTracker.Face>
+  <HolisticTracker.Hand hand="right">
+    <TrackingAnchor target="hand.indexTip">{/* … */}</TrackingAnchor>
+  </HolisticTracker.Hand>
+  <HolisticTracker.Body>
+    <TrackingAnchor target="body.leftShoulder">{/* … */}</TrackingAnchor>
+  </HolisticTracker.Body>
+</HolisticTracker>;
+```
+
+## Composition
+
+- `<HolisticTracker>` acquires `XRModel.HOLISTIC_TRACKER` through the shared,
+  refcounted registry. Every descendant and every other `<HolisticTracker>`
+  in the session shares that one model and one inference per frame.
+- `<HolisticTracker.Face>`, `<HolisticTracker.Hand hand="left" | "right">` and
+  `<HolisticTracker.Body>` are the stream groups. Each provides the same
+  tracker context as `FaceTracker` / `HandTracker` / `BodyTracker`, so
+  `TrackingAnchor` targets (`face.*`, `hand.*`, `body.*`, including
+  `body.neck` and `face.custom`) and `smoothing` work unchanged. The group is
+  hidden while its stream is absent, and its origin is the nose tip, the
+  wrist, or the head.
+- `useHolisticTracking()` returns `{ node, ready, status, error }`. Read
+  `node.getResult()` inside `useFrame`; the result object is stable and
+  mutated in place.
+
+## The unified result
+
+`HolisticResult` carries `version`, `timestampMs` and four streams that are
+`null` when neither detected nor held:
+
+| Stream                  | Coordinates                                                                                                                        | Extras                                                                               |
+| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
+| `face`                  | 478 `NormalizedLandmark`s in MediaPipe image space (0..1, top-left, Y down) — the same space as `FaceTracker`'s `detectionResults` | `blendshapes` name → score, `confidence`, `held`                                     |
+| `leftHand`, `rightHand` | 21 `TrackerPoint`s: center origin, X right, Y up, Z toward the viewer — the same as `HandTracker`'s `coordinates`                  | `handedness`, `worldLandmarks` (metres, MediaPipe world space), `confidence`, `held` |
+| `pose`                  | 33 `TrackerPoint`s — the same as `BodyTracker`'s `getLandmarks2D()`                                                                | `visibility[]`, `worldLandmarks`, `confidence`, `held`                               |
+
+Mirroring of coordinates follows the individual trackers: no source-dependent
+flip is applied to any stream. Hand **labels** are the person's anatomical
+side on every source: holistic assigns hands from the pose, whose left/right
+follow the image, so the SDK swaps them on the mirrored webcam feed.
+`leftHand` is the person's left hand (on the viewer's left in the selfie view),
+the same hand `<HandTracker hand="left">` selects, and `hands="left"` keeps it.
+The pose's left/right landmark indices are swapped the same way, so `pose`
+wrist 15 and `leftHand` are always the same physical hand. This differs from
+`BodyTracker`, whose indices follow the image.
+
+**Pre-release note:** the default model is MediaPipe's published
+`holistic_landmarker.task` on `storage.googleapis.com` (integrity pinned)
+until the SDK CDN copy is published. Pass `modelUrl` to self-host it. The face has no transformation matrix in the holistic graph, so there is
+no head rotation; use `<FaceTracker>` when children must roll with the head.
+
+### Confidence contract
+
+- Face and hands: `1` while detected (the holistic graph does not report
+  per-landmark visibility for them).
+- Pose: the mean landmark `visibility`.
+- While a stream is **held** after a dropout, `held` is `true` and
+  `confidence` decays linearly from the last live value to `0` at `holdMs`.
+
+### Dropouts
+
+Each stream runs through `FaceTrackMemory`, the SDK's one hold/predict bridge:
+a missing stream is replayed for `holdMs` (default 250), extrapolated along its
+last velocity for the first `predictMs` (default 0), then dropped. Set
+`holdMs={0}` for immediate disappearance. Do not add app-side holds.
+
+## Options
+
+| Prop                                                                                                                                               | Applies        | Notes                                                                                  |
+| -------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | -------------------------------------------------------------------------------------- |
+| `modelUrl`                                                                                                                                         | model creation | Defaults to MediaPipe's published `holistic_landmarker.task` (13.7 MB), SHA-384 pinned |
+| `minFaceDetectionConfidence`, `minFacePresenceConfidence`, `minPoseDetectionConfidence`, `minPosePresenceConfidence`, `minHandLandmarksConfidence` | model creation | MediaPipe thresholds, default 0.5                                                      |
+| `outputFaceBlendshapes`                                                                                                                            | model creation | Default `true`                                                                         |
+| `delegate`                                                                                                                                         | model creation | `"auto"` (default), `"GPU"`, `"CPU"`; see below                                        |
+| `targetFps`                                                                                                                                        | run time       | Overrides `perf.targetFps` for this model                                              |
+| `hands`                                                                                                                                            | live           | `"both"` (default), `"left"`, `"right"`, `"none"`; holistic tracks at most one of each |
+| `analysisMaxDimension`                                                                                                                             | live           | Longest side of the analysed frame, default 768                                        |
+| `holdMs`, `predictMs`                                                                                                                              | live           | Dropout bridge                                                                         |
+| `active`                                                                                                                                           | live           | Pause inference without unmounting                                                     |
+| `onStatusChange(status, error)`                                                                                                                    | —              | `loading` → `ready` / `error`; a transient inference error recovers on the next frame  |
+
+Model-creation options come from the first mounted consumer (like `numFaces`
+on `FaceTracker`); changing them re-acquires the model. Live options apply
+through `node.setOptions` and the last writer wins.
+
+## Delegate: GPU with an automatic CPU fallback
+
+MediaPipe's holistic face-blendshape stage has no WebGL implementation for
+one of its ops, so with `outputFaceBlendshapes` the GPU graph fails on its
+first frame (`UNIMPLEMENTED … No support of const`). With the default
+`delegate="auto"` the tracker rebuilds the graph on the CPU once: one
+`console.warn`, status dips to `loading` during the rebuild, no refetch, and
+`node.getDelegate()` reports `"CPU"` afterwards. CPU inference is several
+times slower (tens to hundreds of milliseconds per frame on a laptop). When
+blendshapes are not needed, `outputFaceBlendshapes={false}` keeps the whole
+graph on the GPU. `delegate="GPU"` reports the failure as `error` instead of
+retrying; `delegate="CPU"` starts on the CPU.
+
+## Coexistence with the individual trackers
+
+- Holistic is a **separate model**. Mounting `<FaceTracker>` next to
+  `<HolisticTracker>` runs two inferences; pick one per scene.
+- Inside a holistic scene, use the sub-trackers instead of `FaceTracker` /
+  `HandTracker` / `BodyTracker`. `GestureTracker` (named gestures) and
+  `Segmentation` are unrelated models and combine freely.
+- `FaceMesh`, face readers (`useFaceInfo`, `useFaceResults`), face effects and
+  head binding read `XRModel.FACE_TRACKER`; they do not consume holistic
+  results. For those, keep `<FaceTracker>`.
+
+## Combined signals
+
+`getHolisticSignals(result)` derives presence, `smile` / `mouthOpen`
+blendshape scores, raised hands (pose wrist above its shoulder), open palms
+(three of four fingers extended) and `greeting`: a raised open hand while
+smiling. `isHandRaised` and `isPalmOpen` are exported on their own.
+
+## Migration and performance
+
+- Replace `<FaceTracker>` + `<HandTracker>` + `<BodyTracker>` with one
+  `<HolisticTracker>` and three sub-trackers when a scene needs all three; the
+  anchors move over unchanged. Keep the individual trackers when only one
+  stream is needed: the holistic model is heavier than any single one.
+- Default inference cap follows `perf.targetFps` (the template ships 24).
+  Lower `analysisMaxDimension` (512) when only the face and hands matter and
+  the subject is close.
+- First load downloads 13.7 MB plus the shared MediaPipe engine.
+
+## Manual validation
+
+Run `/holistic-tracking` in the playground:
+
+| Case              | Source                               | Expect                                                                                    |
+| ----------------- | ------------------------------------ | ----------------------------------------------------------------------------------------- |
+| All streams       | "Person Fullbody"                    | Face, both hands and pose dots follow; HUD shows every stream present                     |
+| Hands option      | panel `hands: left`                  | Right hand dots disappear immediately; left keeps tracking                                |
+| Hold              | panel `holdMs: 0` vs 250             | Dots vanish instantly on a dropout vs. hold for a quarter second with decaying confidence |
+| Combined gesture  | webcam, raise an open hand and smile | The raised hand turns yellow and the HUD reports the greeting                             |
+| Status            | reload                               | `loading` → `ready`; a bad `modelUrl` reports `error` with the message                    |
+| Delegate fallback | blendshapes on, then off             | HUD shows `delegate CPU` after one warning; with blendshapes off it stays on `GPU`        |
+| Coexistence       | mount `<FaceTracker>` too            | Two models in the debugger; no crash, no shared state                                     |

package/docs/api/iris-tracking.mdx

@@ -0,0 +1,242 @@
+---
+title: "Iris Tracking"
+description: "Stable left/right iris data from the shared face tracker, and an iris effect that draws a shader or image material over each iris."
+sidebarTitle: "Iris Tracking"
+icon: "eye"
+---
+
+Iris tracking reads the ten refined iris landmarks the face tracker already
+produces. It starts no second detector: `useIrisTracking` acquires the same
+shared `XRModel.FACE_TRACKER` as `FaceTracker` and `useFaceInfo`, and
+`<IrisEffect>` renders over it.
+
+```tsx
+import { useIrisTracking } from "@vincentt-xr/sdk/tracking";
+import { IrisEffect } from "@vincentt-xr/sdk/face-effects";
+
+// Rendering: built-in procedural iris, or an image, or your own material.
+<IrisEffect mode="shader" color="#4fa3ff" opacity={0.85} smoothing={15} />;
+<IrisEffect mode="image" textureUrl="/iris.png" scale={1.1} />;
+
+// Data: read inside useFrame.
+const { result, status } = useIrisTracking({ faceIndex: 0, smoothing: 15 });
+useFrame(() => {
+  if (!result.detected) return;
+  const { center, radius, offset, confidence } = result.left;
+});
+```
+
+## Landmarks and sides
+
+| Side (person's) | Center | Ring (outer, top, inner, bottom) | Inner / outer corner | Upper / lower lid |
+| --------------- | ------ | -------------------------------- | -------------------- | ----------------- |
+| `left`          | 473    | 474, 475, 476, 477               | 362 / 263            | 386 / 374         |
+| `right`         | 468    | 469, 470, 471, 472               | 133 / 33             | 159 / 145         |
+
+Sides follow the face-mesh topology, so they are the person's anatomical
+sides on every source: mirrored webcam, straight video or image. The data
+needs the refined mesh (478 points); with fewer landmarks each eye reports
+`detected: false`. These indices are exported as `IRIS_LANDMARKS`. The
+allocation-free helpers behind the hook (`createIrisResult`,
+`updateIrisResult`, `computeIrisEye`, `IrisLead`) live on
+`@vincentt-xr/sdk/low-level` for pipelines that already hold their own
+landmarks.
+
+## Coordinate spaces
+
+- `center`, `ring`, `eyeCenter`: MediaPipe image space, 0..1 with a top-left
+  origin and Y down, the same space as `FaceInfo.landmarks` and
+  `detectionResults`. `z` is tracker-relative depth.
+- `trackerCenter`: SDK tracker space (center origin, X right, Y up, Z toward
+  the viewer), the same as hand and body points.
+- `radius`, `eyeWidth`: image-width units (half the horizontal ring diameter
+  and the corner-to-corner distance). To draw in world space, unproject the
+  center and a point `radius` to its right and take their distance, which is
+  what `<IrisEffect>` does.
+- `offset`: the iris center relative to the eye center, each axis divided by
+  half the eye width and clamped to [-1, 1]; +x toward the person's left, +y
+  up, on every source (the horizontal sign is corrected on the mirrored
+  webcam feed, so it never flips with the camera). **This is the iris position inside the eyelid opening, not a gaze
+  direction**: it ignores head pose and eyeball geometry. Use it for
+  pupil-follow and shader inputs, not for "where is the person looking".
+- `openness`: eyelid gap divided by eye width (about 0.25–0.35 open, 0 closed).
+
+Mirroring: the session mirrors the webcam feed and no other source, and the
+landmarks are already in that feed space, so positions need no flip. The
+`mirrored` prop on `<IrisEffect>` flips the material horizontally (image
+orientation, shader +x), nothing else.
+
+## Confidence and lifecycle
+
+MediaPipe reports no score for iris points and emits them even when the eye
+is closed, so confidence is derived: `eye.confidence` is `openness / 0.2`
+clamped to 0..1, scaled by the hold decay. `result.confidence` is 1 while the
+face is live and decays linearly to 0 over `holdMs` while `held`. Gate
+effects on these; the built-in materials fade with `irisConfidence`.
+
+`useIrisTracking` returns `status` (`loading` → `ready` / `error`) for the
+shared face model plus `result.detected` / `held` / `version` /
+`timestampMs` / `faceIndex`. The `result` object is stable and mutated in
+place; read it inside `useFrame`.
+
+## Smoothing, hold and face selection
+
+- Dropouts go through `FaceTrackMemory`, the SDK's one bridge: `holdMs` 250
+  and `predictMs` 0 by default, like every other face surface. A face that
+  stays missing longer than `holdMs` is dropped and `<IrisEffect>` hides at
+  once, so no stale eye effect lingers.
+- `smoothing` runs the same One Euro filter as `TrackingAnchor` on the
+  centers and radius (default 0 = raw; ~10 tames jitter at rest). It is a
+  low-pass whose floor cutoff is `smoothing × 0.1` Hz, so 15 attenuates and
+  lags a 2–3 Hz head shake: keep it low when exact following matters. It
+  resets on re-acquire, so the iris never slides in from an old position.
+- `targetFps` requests an inference cap for the shared face model (0 keeps
+  the session's `perf.targetFps`, 24 in the template). The first mounted face
+  consumer's value wins, so pass the same number everywhere the face model
+  is acquired, and note that a later change only takes effect once every
+  holder has released the model (a lone `<IrisEffect>` re-acquires and
+  re-initializes it; a session pipeline entry pins the first value for the
+  session). The example makes the effect the only holder so its panel value
+  is the real rate. **60 gives the best following, but use it with caution**: it runs
+  face inference 2.5× as often as the template default and will heat and
+  throttle phones. Ship it only when the eye effect is the point of the
+  experience and you have checked the target devices; otherwise keep the
+  session default (or 30) and let `leadMs` bridge the gap.
+- `leadMs` (default 80, 0 disables) bridges throttled inference to the render
+  rate: between samples the centers advance along their last
+  sample-to-sample velocity for at most `leadMs`, so a 24 fps face model
+  reads continuous at 60+ fps instead of hold-and-jump. When the lag itself
+  is the problem, raise the face model's `targetFps` first (the example
+  requests 60).
+- `faceIndex` / `numFaces` select the face exactly like `useFaceInfo`: the
+  index is the result-array position, not a persistent identity, and the
+  first mounted face consumer decides `numFaces`.
+
+## `<IrisEffect>` props and materials
+
+`<IrisEffect>` takes the flat `IrisEffectSettings` (`eye`: both | left |
+right, `mode`: shader | image, `color`, `textureUrl`, `tintImage` (multiply the
+image by `color`; off by default so images show as authored), `scale`, `opacity`,
+`targetFps`, `smoothing`, `leadMs`, `mirrored`, `faceIndex`, `holdMs`) plus `numFaces`, `active`,
+`predictMs`, `texture` (a pre-loaded three texture, beats `textureUrl`),
+`material`, `renderOrder` and `onTracking(handle)`.
+
+Each eye is a quad spanning the iris diameter, positioned on the camera's
+z=0 plane. Every material receives the same uniforms (`IrisUniforms`):
+
+| Uniform          | Type        | Meaning                            |
+| ---------------- | ----------- | ---------------------------------- |
+| `irisCenter`     | `vec2`      | Image-space center                 |
+| `irisRadius`     | `float`     | Image-width radius                 |
+| `irisOffset`     | `vec2`      | Position inside the eye, [-1, 1]   |
+| `irisConfidence` | `float`     | 0..1, fade with it                 |
+| `irisOpenness`   | `float`     | Eyelid gap / eye width             |
+| `time`           | `float`     | Seconds since mount                |
+| `color`          | `vec3`      | Tint                               |
+| `opacity`        | `float`     |                                    |
+| `mirror`         | `float`     | 1 when mirrored                    |
+| `tintImage`      | `float`     | 1 to multiply the image by `color` |
+| `map`            | `sampler2D` | The image in `image` mode          |
+
+A minimal custom material:
+
+```tsx
+import { IrisEffect, type IrisMaterialContext } from "@vincentt-xr/sdk/face-effects";
+
+const ring = ({ uniforms }: IrisMaterialContext) =>
+  new THREE.ShaderMaterial({
+    uniforms, // the shared, live uniforms
+    vertexShader: `varying vec2 vUv; void main(){ vUv = uv; gl_Position = projectionMatrix * modelViewMatrix * vec4(position, 1.0); }`,
+    fragmentShader: `
+      uniform float irisConfidence; uniform vec3 color; uniform float opacity; varying vec2 vUv;
+      void main(){
+        float r = length(vUv - 0.5) * 2.0;
+        float ring = smoothstep(0.6, 0.7, r) * (1.0 - smoothstep(0.9, 1.0, r));
+        gl_FragColor = vec4(color, ring * opacity * irisConfidence);
+        #include <colorspace_fragment>
+      }`,
+    transparent: true,
+    depthTest: false,
+    depthWrite: false,
+  });
+
+<IrisEffect material={ring} color="#ffcc00" />;
+```
+
+The factory runs once per eye for each `material` identity; the effect
+disposes what a factory created when the factory changes or on unmount,
+while an instance you pass stays yours. Read the live texture from
+`uniforms.map`: `context.texture` is only the texture known when the factory
+ran, often null. The built-in
+image shader shows it as authored, multiplies it by `color` only when
+`tintImage` is on, and fades with confidence.
+
+## When to use what
+
+- **Iris tracking**: eye decorations, pupil overlays, iris recolouring,
+  shader inputs tied to where the iris sits in the eye.
+- **Blendshapes** (`useFaceInfo` / `useFaceAction`): blinks, winks, squints,
+  eye-look-left/right/up/down categories as discrete expressions.
+- **Gaze estimation**: not provided. `offset` is not gaze; a real estimate
+  needs head pose and eye geometry.
+- **PinToMesh / FaceMesh materials**: content that must deform with the skin
+  around the eye rather than sit on the iris.
+
+## Hiding the iris behind the eyelids
+
+The quads are flat and draw over the lids. To clip them to the eye openings
+without a dedicated mask, draw the face mesh over them with the camera feed
+and no eye-opening triangles (the example's "eyelid mask" toggle). The
+material samples the feed through the mesh's live `faceUv` attribute with
+depth testing off, so it always lands on top of the quads:
+
+```tsx
+import { CanonicalFaceMeshRenderer, IrisEffect } from "@vincentt-xr/sdk/face-effects";
+import { useXRCameraTexture } from "@vincentt-xr/sdk/low-level";
+
+const feed = new THREE.ShaderMaterial({
+  uniforms: { cameraTexture: { value: useXRCameraTexture() } },
+  vertexShader: `attribute vec2 faceUv; varying vec2 vUv;
+    void main(){ vUv = faceUv; gl_Position = projectionMatrix * modelViewMatrix * vec4(position, 1.0); }`,
+  fragmentShader: `uniform sampler2D cameraTexture; varying vec2 vUv;
+    void main(){ gl_FragColor = texture2D(cameraTexture, vUv); #include <colorspace_fragment> }`,
+  transparent: true, // the quads are transparent; an opaque mask would draw before them
+  depthTest: false,
+  depthWrite: false,
+});
+
+// IrisEffect renders at order 100; the mask draws after it.
+<IrisEffect />;
+<CanonicalFaceMeshRenderer
+  showEyeGeometry={false}
+  showEyeMask={false}
+  renderOrder={101}
+  materialOverride={feed}
+/>;
+```
+
+Memoize the material on the camera texture and dispose it on unmount, as
+the example does.
+
+## Known limitations
+
+- Single-person per `faceIndex`; multi-face scenes mount one effect per face.
+- Iris points are present even for closed eyes; `confidence` is the gate.
+- The quads sit on the z=0 plane without eyeball curvature or head pose.
+- Hold replays the last position; with `predictMs` > 0 a sharp reversal can
+  overshoot for up to that window.
+
+## Manual validation
+
+Run `/iris-tracking`:
+
+| Case                 | Steps                                   | Expect                                                             |
+| -------------------- | --------------------------------------- | ------------------------------------------------------------------ |
+| Shader material      | default, change color / scale / opacity | Both irises recoloured live; pupil follows the iris offset         |
+| Image material       | mode `image`, presets or a URL          | Image covers each iris, tinted by color; mirror flips it           |
+| Custom material      | "custom shader (ring)"                  | Pulsing ring from the shared uniforms                              |
+| Tracking loss        | cover the face / "Person Shaking Head"  | Effect holds ~250 ms then disappears; no stale eyes                |
+| Multi-face           | "Two People Smiling", face index 0 / 1  | Effect moves to the selected face                                  |
+| Mirrored camera      | webcam                                  | Left/right stay anatomical; the eye on the viewer's left is `left` |
+| Portrait / landscape | resize the pane                         | Quads stay centred on the irises at the right size                 |
+| Closed eyes          | blink                                   | Effect fades with confidence                                       |