@vincentt-xr/sdk 2.0.0-alpha.0 → 2.0.0-alpha.10

package/docs/api/holistic-tracker.mdx

@@ -0,0 +1,164 @@
+---
+title: "Holistic Tracker"
+description: "One MediaPipe inference for face, hands and pose, with unified results, dropout bridging and the same TrackingAnchor targets as the individual trackers."
+sidebarTitle: "Holistic Tracker"
+icon: "person-rays"
+---
+
+`<HolisticTracker>` runs MediaPipe's Holistic Landmarker: a single graph that
+returns face landmarks and blendshapes, left and right hand landmarks, and the
+33-point pose for **one person**. It is additive and opt-in; `FaceTracker`,
+`HandTracker` and `BodyTracker` are unchanged.
+
+```tsx
+import { HolisticTracker, TrackingAnchor } from "@vincentt-xr/sdk/tracking";
+
+<HolisticTracker holdMs={250}>
+  <HolisticTracker.Face>
+    <TrackingAnchor target="face.noseTip">{/* … */}</TrackingAnchor>
+  </HolisticTracker.Face>
+  <HolisticTracker.Hand hand="right">
+    <TrackingAnchor target="hand.indexTip">{/* … */}</TrackingAnchor>
+  </HolisticTracker.Hand>
+  <HolisticTracker.Body>
+    <TrackingAnchor target="body.leftShoulder">{/* … */}</TrackingAnchor>
+  </HolisticTracker.Body>
+</HolisticTracker>;
+```
+
+## Composition
+
+- `<HolisticTracker>` acquires `XRModel.HOLISTIC_TRACKER` through the shared,
+  refcounted registry. Every descendant and every other `<HolisticTracker>`
+  in the session shares that one model and one inference per frame.
+- `<HolisticTracker.Face>`, `<HolisticTracker.Hand hand="left" | "right">` and
+  `<HolisticTracker.Body>` are the stream groups. Each provides the same
+  tracker context as `FaceTracker` / `HandTracker` / `BodyTracker`, so
+  `TrackingAnchor` targets (`face.*`, `hand.*`, `body.*`, including
+  `body.neck` and `face.custom`) and `smoothing` work unchanged. The group is
+  hidden while its stream is absent, and its origin is the nose tip, the
+  wrist, or the head.
+- `useHolisticTracking()` returns `{ node, ready, status, error }`. Read
+  `node.getResult()` inside `useFrame`; the result object is stable and
+  mutated in place.
+
+## The unified result
+
+`HolisticResult` carries `version`, `timestampMs` and four streams that are
+`null` when neither detected nor held:
+
+| Stream                  | Coordinates                                                                                                                        | Extras                                                                               |
+| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
+| `face`                  | 478 `NormalizedLandmark`s in MediaPipe image space (0..1, top-left, Y down) — the same space as `FaceTracker`'s `detectionResults` | `blendshapes` name → score, `confidence`, `held`                                     |
+| `leftHand`, `rightHand` | 21 `TrackerPoint`s: center origin, X right, Y up, Z toward the viewer — the same as `HandTracker`'s `coordinates`                  | `handedness`, `worldLandmarks` (metres, MediaPipe world space), `confidence`, `held` |
+| `pose`                  | 33 `TrackerPoint`s — the same as `BodyTracker`'s `getLandmarks2D()`                                                                | `visibility[]`, `worldLandmarks`, `confidence`, `held`                               |
+
+Mirroring of coordinates follows the individual trackers: no source-dependent
+flip is applied to any stream. Hand **labels** are the person's anatomical
+side on every source: holistic assigns hands from the pose, whose left/right
+follow the image, so the SDK swaps them on the mirrored webcam feed.
+`leftHand` is the person's left hand (on the viewer's left in the selfie view),
+the same hand `<HandTracker hand="left">` selects, and `hands="left"` keeps it.
+The pose's left/right landmark indices are swapped the same way, so `pose`
+wrist 15 and `leftHand` are always the same physical hand. This differs from
+`BodyTracker`'s raw `getLandmarks2D()`, whose indices follow the image (its
+`TrackingAnchor` targets are the person's side). `pose` and `pose.worldLandmarks` are already the
+person's own sides on every feed (index 11 = their left shoulder): never convert them again with
+`personSide` / `feedSidedName` / `bodyLandmarkIndex`, which would flip them back. The `face` stream keeps the
+mesh's indices; sided `TrackingAnchor` targets under `HolisticTracker.Face` are
+the person's own (GROUNDING "Left and right").
+
+**Pre-release note:** the default model is MediaPipe's published
+`holistic_landmarker.task` on `storage.googleapis.com` (integrity pinned)
+until the SDK CDN copy is published. Pass `modelUrl` to self-host it. The face has no transformation matrix in the holistic graph, so there is
+no head rotation; use `<FaceTracker>` when children must roll with the head.
+
+### Confidence contract
+
+- Face and hands: `1` while detected (the holistic graph does not report
+  per-landmark visibility for them).
+- Pose: the mean landmark `visibility`.
+- While a stream is **held** after a dropout, `held` is `true` and
+  `confidence` decays linearly from the last live value to `0` at `holdMs`.
+
+### Dropouts
+
+Each stream runs through `FaceTrackMemory`, the SDK's one hold/predict bridge:
+a missing stream is replayed for `holdMs` (default 250), extrapolated along its
+last velocity for the first `predictMs` (default 0), then dropped. Set
+`holdMs={0}` for immediate disappearance. Do not add app-side holds.
+
+## Options
+
+| Prop                                                                                                                                               | Applies        | Notes                                                                                  |
+| -------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | -------------------------------------------------------------------------------------- |
+| `modelUrl`                                                                                                                                         | model creation | Defaults to MediaPipe's published `holistic_landmarker.task` (13.7 MB), SHA-384 pinned |
+| `minFaceDetectionConfidence`, `minFacePresenceConfidence`, `minPoseDetectionConfidence`, `minPosePresenceConfidence`, `minHandLandmarksConfidence` | model creation | MediaPipe thresholds, default 0.5                                                      |
+| `outputFaceBlendshapes`                                                                                                                            | model creation | Default `true`                                                                         |
+| `delegate`                                                                                                                                         | model creation | `"auto"` (default), `"GPU"`, `"CPU"`; see below                                        |
+| `targetFps`                                                                                                                                        | run time       | Overrides `perf.targetFps` for this model                                              |
+| `hands`                                                                                                                                            | live           | `"both"` (default), `"left"`, `"right"`, `"none"`; holistic tracks at most one of each |
+| `analysisMaxDimension`                                                                                                                             | live           | Longest side of the analysed frame, default 768                                        |
+| `holdMs`, `predictMs`                                                                                                                              | live           | Dropout bridge                                                                         |
+| `active`                                                                                                                                           | live           | Pause inference without unmounting                                                     |
+| `onStatusChange(status, error)`                                                                                                                    | —              | `loading` → `ready` / `error`; a transient inference error recovers on the next frame  |
+
+Model-creation options come from the first mounted consumer (like `numFaces`
+on `FaceTracker`); changing them re-acquires the model. Live options apply
+through `node.setOptions` and the last writer wins.
+
+## Delegate: GPU with an automatic CPU fallback
+
+MediaPipe's holistic face-blendshape stage has no WebGL implementation for
+one of its ops, so with `outputFaceBlendshapes` the GPU graph fails on its
+first frame (`UNIMPLEMENTED … No support of const`). With the default
+`delegate="auto"` the tracker rebuilds the graph on the CPU once: one
+`console.warn`, status dips to `loading` during the rebuild, no refetch, and
+`node.getDelegate()` reports `"CPU"` afterwards. CPU inference is several
+times slower (tens to hundreds of milliseconds per frame on a laptop). When
+blendshapes are not needed, `outputFaceBlendshapes={false}` keeps the whole
+graph on the GPU. `delegate="GPU"` reports the failure as `error` instead of
+retrying; `delegate="CPU"` starts on the CPU.
+
+## Coexistence with the individual trackers
+
+- Holistic is a **separate model**. Mounting `<FaceTracker>` next to
+  `<HolisticTracker>` runs two inferences; pick one per scene.
+- Inside a holistic scene, use the sub-trackers instead of `FaceTracker` /
+  `HandTracker` / `BodyTracker`. `GestureTracker` (named gestures) and
+  `Segmentation` are unrelated models and combine freely.
+- `FaceMesh`, face readers (`useFaceInfo`, `useFaceResults`), face effects and
+  head binding read `XRModel.FACE_TRACKER`; they do not consume holistic
+  results. For those, keep `<FaceTracker>`.
+
+## Combined signals
+
+`getHolisticSignals(result)` derives presence, `smile` / `mouthOpen`
+blendshape scores, raised hands (pose wrist above its shoulder), open palms
+(three of four fingers extended) and `greeting`: a raised open hand while
+smiling. `isHandRaised` and `isPalmOpen` are exported on their own.
+
+## Migration and performance
+
+- Replace `<FaceTracker>` + `<HandTracker>` + `<BodyTracker>` with one
+  `<HolisticTracker>` and three sub-trackers when a scene needs all three; the
+  anchors move over unchanged. Keep the individual trackers when only one
+  stream is needed: the holistic model is heavier than any single one.
+- Default inference cap follows `perf.targetFps` (the template ships 24).
+  Lower `analysisMaxDimension` (512) when only the face and hands matter and
+  the subject is close.
+- First load downloads 13.7 MB plus the shared MediaPipe engine.
+
+## Manual validation
+
+Run `/holistic-tracking` in the playground:
+
+| Case              | Source                               | Expect                                                                                    |
+| ----------------- | ------------------------------------ | ----------------------------------------------------------------------------------------- |
+| All streams       | "Person Fullbody"                    | Face, both hands and pose dots follow; HUD shows every stream present                     |
+| Hands option      | panel `hands: left`                  | Right hand dots disappear immediately; left keeps tracking                                |
+| Hold              | panel `holdMs: 0` vs 250             | Dots vanish instantly on a dropout vs. hold for a quarter second with decaying confidence |
+| Combined gesture  | webcam, raise an open hand and smile | The raised hand turns yellow and the HUD reports the greeting                             |
+| Status            | reload                               | `loading` → `ready`; a bad `modelUrl` reports `error` with the message                    |
+| Delegate fallback | blendshapes on, then off             | HUD shows `delegate CPU` after one warning; with blendshapes off it stays on `GPU`        |
+| Coexistence       | mount `<FaceTracker>` too            | Two models in the debugger; no crash, no shared state                                     |

package/docs/api/instructions.mdx

@@ -0,0 +1,323 @@
+---
+title: "Instructions"
+description: "Prompts that ask for a gesture, an expression or a pose with a 3D hint and your copy — ready-made presets, and every layer underneath open for your own vocabulary, detection, look and layout."
+sidebarTitle: "Instructions"
+icon: "person-chalkboard"
+---
+
+An instruction asks the person to do something and waits until they do. It shows a 3D hint, puts
+your copy under it, and completes once the pose has been held for `holdMs`. Everything lives in one door:
+
+```tsx
+import { … } from "@vincentt-xr/sdk/instructions";
+```
+
+There are three ready-made presets:
+
+- [Hand Instructions](/api/hand-instructions)
+- [Face Instructions](/api/face-instructions)
+- [Body Instructions](/api/body-instructions)
+
+Each preset is built from four open layers, and you can use any layer on its own from outside the SDK.
+
+| Layer          | What you get                                                                                                                |
+| -------------- | --------------------------------------------------------------------------------------------------------------------------- |
+| **Vocabulary** | The built-in gestures, expressions and poses; `defineHandGesture` / `defineFaceExpression` / `defineBodyPose` for your own. |
+| **Detection**  | `useHandGesture`, `useFaceExpression`, `useBodyPose`, and `useInstructionHold` for any condition.                           |
+| **Hint**       | `HandGestureHint`, `FaceExpressionHint`, `BodyPoseHint` with `look` and `loop`, and the kit to build a hint on any model.   |
+| **Layout**     | `<Instruction>`: any detector, any symbol, your text style, motion and proportions.                                         |
+
+The rest of this page goes from small changes to a fully custom prompt.
+
+## 1. Restyle a preset
+
+Every preset takes the same layout and text options, plus `look` and `loop` for its hint. Each is
+merged over that preset's default, so you only write what you change.
+
+```tsx
+<HandInstruction
+  gesture="peace"
+  model={hand.scene}
+  poses={hand.animations}
+  size={0.2} // a fraction of the screen height, or "sm" | "md" | "lg"
+  look={{
+    ghost: { color: "#7dd3fc", core: 0.35 }, // tint, opacity facing you, at the rim (`rim`), rim whitening
+    halo: false, // turn a layer off
+    glow: { color: "#38bdf8", full: 1 },
+    highlight: { opacity: 1 },
+  }}
+  loop={{ seconds: 1.6, stagger: 0 }} // timing, hold share, start, easing, finger stagger (hand)
+  textStyle={{ fontFamily: "Georgia, serif", strokeColor: "#0c4a6e", strokeWidth: 6 }}
+  motion={{ ring: { color: "#38bdf8" }, pop: 0.2, bob: 0 }}
+  layout={{ gap: 0.2, textHeight: 0.35 }}
+/>
+```
+
+**`look`** (`HintLook`) controls how the hint is drawn:
+
+- `ghost` (`GhostOptions`): `color`, `core` (opacity facing you), `rim` (opacity at the silhouette),
+  `whiten`.
+- `parts`: overrides per model material name, e.g. `{ iris: { color: "#2a2732" } }`; `null` removes
+  a default tint.
+- `halo`: `width` in model units, `color`, `opacity`.
+- `glow`: `color`, `size`, `idle` and `full` opacity.
+- `highlight`: `opacity`, `whiten`, and `idle` (the share shown while relaxed).
+- `matcap`: your own shading texture.
+- `material`: any three.js material, replacing the ghost.
+
+Any layer can be `false` to turn it off. The defaults are exported:
+
+- `HAND_HINT_LOOK` and `HAND_HINT_LOOP`
+- `FACE_HINT_LOOK` and `FACE_HINT_LOOP`
+- `BODY_HINT_LOOK` and `BODY_HINT_LOOP`
+
+`mergeHintLook(base, over)` merges two looks the same way the presets do.
+
+**`loop`** (`HintLoop`) controls how the hint animates:
+
+- `seconds` per cycle; `0` holds the pose.
+- `hold`: the share of the cycle held at the full pose.
+- `from`: where each cycle starts, 0 = the model's rest.
+- `easing`.
+- The hand also takes `stagger`, how far apart the fingers move; `0` moves them together.
+
+**Text and layout** apply to every preset and to `<Instruction>`:
+
+- `textStyle` / `nudgeTextStyle`: any `TextLabel` style (font, weight, stroke, shadow, background),
+  merged over `INSTRUCTION_TEXT_STYLE`.
+- `color`: a shortcut for the text colour.
+- `motion` (`InstructionMotion`):
+  - `fadeIn` and `fadeOut`, in seconds;
+  - `rise`, `bob`, `pop`, `popSeconds` and `nudgePulse`;
+  - `ring`: `{ color, seconds, opacity, from, to }`, or `false`.
+- `layout` (`InstructionLayout`): `textHeight`, `gap`, `nudgeHeight`, `textWidth` and `edgeMargin`.
+  Each is relative to the size, except `edgeMargin`, which is a fraction of the screen height.
+- `renderText(slot)`: draw each line yourself. You get `{ text, role, width, height, renderOrder,
+style }`. Use transparent materials, because the layout fades every mesh under it.
+
+## 2. Your own gesture, expression or pose
+
+A definition holds three things:
+
+- its default copy (`label`);
+- how it is detected: a `rule`, a `match` function, or both, in which case both must pass;
+- how the hint shows it (`hint`).
+
+Pass a definition anywhere a built-in name goes: the presets, the hooks, the hints and the pure
+`match*` functions.
+
+```tsx
+const hangLoose = defineHandGesture({
+  label: "Hang loose",
+  rule: { digits: { thumb: true, index: false, middle: false, ring: false, pinky: true } },
+  match: (shape) => shape.curl.pinky < 0.35, // and the pinky really straight
+  hint: { clip: "callMe", roll: 103, highlight: ["thumb", "pinky"] },
+});
+
+const yawn = defineFaceExpression({
+  label: "Big yawn",
+  rule: { atLeast: { jawOpen: 0.7 } },
+  hint: { targets: { surprised: 1, smile: 0.3 } }, // a weighted mix of morph targets
+});
+
+const flex = defineBodyPose({
+  label: "Show your muscles",
+  match: (shape) =>
+    shape.arms.left.bend > 70 && shape.arms.right.bend > 70 && shape.arms.left.up > 0,
+  hint: { clip: "flex", arms: ["left", "right"] }, // a clip in your own figure
+});
+
+<HandInstruction gesture={hangLoose} model={hand.scene} poses={hand.animations} />;
+```
+
+The hint can name a clip in your own model:
+
+- **Hand:** a `HandHintPose`: an authored `clip`, or a procedural pose (`curl`, `spread`,
+  `thumbTuck`, `thumbTo`), plus the view (`yaw`, `roll`, `aimThumb`) and `highlight`.
+- **Face:** `{ target }` or `{ targets }`.
+- **Body:** `{ clip, arms }`.
+
+`resolveHandGesture`, `resolveFaceExpression` and `resolveBodyPose` turn a name or definition into
+the full definition.
+
+## 3. Tune detection
+
+Every hook takes the same kinds of input:
+
+- a built-in name;
+- a rule;
+- a match function:
+  - hand: `(shape, side) => boolean`;
+  - face: `(blendshapes, face) => boolean`;
+  - body: `(shape) => boolean`;
+- a definition.
+
+They also take `holdMs`, `graceMs`, `onComplete` and `active`.
+
+The hand and body hooks take `limits`: thresholds overridden for that detector only. The defaults,
+`HAND_SHAPE_LIMITS` and `BODY_SHAPE_LIMITS`, are frozen. The pure classifiers take them too:
+
+```tsx
+const thumb = useHandGesture("thumbUp", { limits: { thumbVertical: 0.5 }, holdMs: 800 });
+const shape = analyzeBodyShape(landmarks, { mirrored, limits: { raisedAboveShoulder: 1 } });
+```
+
+Every hook returns an **`InstructionDetector`** plus its own readout (`shapesRef`, `faceRef`,
+`shapeRef`):
+
+- `status`: re-renders only when it changes;
+- `holdRef`: `progress` 0..1;
+- `matchedRef`: matched on this frame;
+- `isInView()`;
+- `reset()`.
+
+**Any condition.** `useInstructionHold(match, options)` holds any test you write; `match` runs every
+frame. To combine detectors, read their `matchedRef`s, and call those hooks first in the same
+component:
+
+```tsx
+const hand = useHandGesture("palm");
+const face = useFaceExpression("smile");
+const both = useInstructionHold(() => hand.matchedRef.current && face.matchedRef.current, {
+  isInView: () => hand.isInView() && face.isInView(),
+  holdMs: 600,
+  onComplete: start,
+});
+```
+
+A new name, rule or definition value starts a new prompt. A match function is read live and doesn't
+restart the hold. For a new prompt with the same function, call `reset()` or change the component's
+React `key`.
+
+## 4. A fully custom prompt: `<Instruction>`
+
+The layout on its own: any detector, any symbol.
+
+```tsx
+<Instruction
+  detector={both} // any hook result, or your own { status, isInView }
+  text="Wave and smile"
+  nudgeText="Show your hand and your face" // default null (no nudge) here
+  symbol={(height, renderOrder) => (
+    <TextLabel
+      text="🖐️😊"
+      bgColor="transparent"
+      aspect={2}
+      scale={[height * 2, height, 1]}
+      renderOrder={renderOrder}
+    />
+  )}
+  renderText={({ text, width, height, renderOrder }) => (
+    <TextLabel
+      text={text}
+      bgColor="#fde68a"
+      color="#0f172a"
+      borderRadius={60}
+      bgFit="text"
+      aspect={width / height}
+      scale={[width, height, 1]}
+      renderOrder={renderOrder}
+    />
+  )}
+/>
+```
+
+It takes every option from section 1, plus `symbolAspect` (a taller slot, e.g. 2.1 for a standing
+figure) and `active`. Mount it inside `XRScene`; it draws in its own `ScreenSpaceUI`, above the AR
+content (`INSTRUCTION_RENDER_ORDER`, 2000 and up). The symbol can be:
+
+- a preset hint, e.g. `HandGestureHint` with your definition and `look`;
+- an image (`ScreenImage`), a `SpriteAnimation` or an emoji;
+- your own 3D model.
+
+## 5. A hint on your own model
+
+The presets' hints are built from a small kit you can use on any model:
+
+- **`useGhostModel(model, look, { base, renderOrder })`** makes a ghost copy of the model.
+  - It applies a matcap per part, a depth pre-pass, the halo and the glow, all from one `look`.
+  - It returns `object` to render, and `meshes` (each with its twins, which need the same morph
+    weights).
+  - `setHighlight(bone => 0..1)` lights up bones; call `setStrength(amount)` every frame.
+- **`HintStage`** handles the staging. It fits `span` to `height`, mirrors the model, draws the glow
+  disc behind it and runs the auto orbit.
+- **`sampleHintLoop(time, loop)`** samples the loop: rest → pose (held) → rest. It returns `{ amount,
+drive, phase, closing }`.
+- **`createClipBlend(root, fromClip, toClip)`** and **`createMorphBlend(meshes, targets)`** pose the
+  model; `apply(amount)` each frame.
+- **`orbitSpan(points, middle, amplitude)`** fits the model over the whole orbit.
+
+```tsx
+const MyHint = ({ height, renderOrder }: { height: number; renderOrder: number }) => {
+  const ghost = useGhostModel(
+    face.scene,
+    { ghost: { color: "#fde68a" } },
+    { base: FACE_HINT_LOOK, renderOrder },
+  );
+  const blend = useMemo(
+    () =>
+      createMorphBlend(
+        ghost.meshes.flatMap(({ mesh, twins }) => [mesh, ...twins]),
+        { surprised: 1, wink: 1 },
+      ),
+    [ghost.meshes],
+  );
+  const frame = useMemo(() => {
+    const box = new THREE.Box3().setFromObject(ghost.object);
+    const size = box.getSize(new THREE.Vector3());
+    return {
+      offset: box.getCenter(new THREE.Vector3()).negate(),
+      span: Math.max(size.x, size.y) * 1.2,
+    };
+  }, [ghost.object]);
+  useFrame(({ clock }) => {
+    const { amount } = sampleHintLoop(clock.elapsedTime, { seconds: 2 });
+    blend.apply(amount);
+    ghost.setStrength(amount);
+  });
+  return (
+    <HintStage
+      height={height}
+      span={frame.span}
+      orbit={20}
+      glow={ghost.glowMaterial}
+      renderOrder={renderOrder}
+    >
+      <group position={frame.offset}>
+        <primitive object={ghost.object} />
+      </group>
+    </HintStage>
+  );
+};
+```
+
+The material factories are exported for full control: `createGhostMaterial`, `createHaloMaterial`,
+`createGlowMaterial`, `createDepthPrepassMaterial`, `createHintMatcap`, `addHintTwins` and
+`assignHintMaterials`. Their opacities, colours and widths are uniforms (`userData.ghost`,
+`userData.halo`). The hold timing is pure too: `createGestureHoldState`, `stepGestureHold`,
+`GESTURE_HOLD_DEFAULT_MS` (400) and `GESTURE_HOLD_GRACE_MS` (150).
+
+## Example
+
+`/custom-instruction` has three demos, built only from this door:
+
+1. **Hang loose:** a custom gesture on the hand preset, with a restyled look, loop, text style and
+   ring.
+2. **Palm and smile:** an open palm and a smile held together. It uses an emoji symbol and a pill
+   text component.
+3. **Surprised wink:** your own hint, mixing two morph targets, for an expression with a match
+   function.
+
+## Manual visual test plan
+
+Open `/custom-instruction`.
+
+1. **Restyled preset**: the hang-loose hand is cyan with no halo and a blue glow. The thumb and
+   pinky are brighter, the fingers move together, and the text is a serif with a dark stroke. On the
+   webcam, do the gesture: a large cyan ring plays, and "Nice!" replaces the copy.
+2. **Any condition**: the emoji pair stands over a yellow pill. Show an open palm only: it stays
+   `detected`. Then smile with the palm up: it holds, then completes.
+3. **Own hint**: the gold face loops into a surprised wink with a dark-gold halo. A plain wink or a
+   plain open mouth doesn't complete it; both together do.
+4. **Presets unchanged**: `/hand-instructions`, `/face-instructions` and `/body-instructions` look
+   and behave as before. The exception is peace, point and the counts, which now show the palm.