@vincentt-xr/sdk 2.0.0-alpha.8 → 2.0.0-alpha.9

package/docs/api/hand-instructions.mdx

@@ -0,0 +1,222 @@
+---
+title: "Hand Instructions"
+description: "Ask for a hand gesture with a 3D hint hand and your own copy, detect it on the tracked hands, and complete when it is held."
+sidebarTitle: "Hand Instructions"
+icon: "hand-peace"
+---
+
+`HandInstruction` asks the person for a hand gesture and waits for it. A looping, translucent 3D hand
+demonstrates the gesture, your copy sits under it, and holding the gesture for `holdMs` completes it.
+There is no card and no progress bar. The pieces underneath are exported too: `useHandGesture` (the
+detection alone) and `HandGestureHint` (the hand alone). For face expressions, see [Face Instructions](/api/face-instructions):
+same layout and feedback.
+
+Import from `@vincentt-xr/sdk/instructions`. Everything is customisable: your own gestures, detection
+logic, thresholds, hint look and motion, text style, or a fully custom prompt. See
+[Instructions](/api/instructions).
+
+## Quick start
+
+```tsx
+import { useLoader } from "@react-three/fiber";
+import { GLTFLoader } from "three/examples/jsm/loaders/GLTFLoader.js";
+import { HandInstruction } from "@vincentt-xr/sdk/instructions";
+
+const PeaceToStart = ({ onStart }: { onStart: () => void }) => {
+  const hand = useLoader(GLTFLoader, handUrl); // hand-v1.glb, or any hand following the rig contract
+  return (
+    <HandInstruction
+      gesture="peace"
+      text="Peace to start"
+      model={hand.scene}
+      poses={hand.animations}
+      onComplete={onStart}
+    />
+  );
+};
+```
+
+Mount it inside `XRScene`. It acquires the hand model itself, so you don't need a `HandTracker`, and
+draws over the AR content in its own screen-space layer (`HAND_INSTRUCTION_RENDER_ORDER` = 2000 and
+up). Detection reads the hand model's world poses. It is separate from `GestureTracker` and its
+MediaPipe gesture names.
+
+## `HandInstruction`
+
+| Prop             | Default                          | What it does                                                                                                                                       |
+| ---------------- | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `gesture`        | —                                | What to ask for: a name from `HAND_GESTURES` or your own `defineHandGesture(…)`. A new value is a new prompt.                                      |
+| `model`          | —                                | The hint hand: a skinned hand following the [rig contract](/api/hand-rig#the-rig-contract-for-your-own-hand-model).                                |
+| `poses`          | —                                | The model's authored poses (the GLB's `animations`). Without them the hint is posed procedurally.                                                  |
+| `modelHand`      | `"right"`                        | The chirality the model is authored as.                                                                                                            |
+| `text`           | the gesture's `label`            | Your copy, in any language. Long text shrinks to fit.                                                                                              |
+| `doneText`       | keep `text`                      | Copy shown once completed, while it fades out.                                                                                                     |
+| `nudgeText`      | `"Show your hand to the camera"` | Second line after `nudgeAfterMs` with no hand in view; the hint pulses. `null` turns it off.                                                       |
+| `nudgeAfterMs`   | `3000`                           | How long without a hand before the nudge.                                                                                                          |
+| `hand`           | `"any"`                          | Which of the person's hands may do it: `"left"`, `"right"` or `"any"`.                                                                             |
+| `holdMs`         | `400`                            | How long the gesture must be held.                                                                                                                 |
+| `graceMs`        | `150`                            | Dropouts shorter than this don't restart the hold.                                                                                                 |
+| `limits`         | —                                | Classifier thresholds to override for this prompt (see `HAND_SHAPE_LIMITS`).                                                                       |
+| `size`           | `"md"`                           | `"sm"`, `"md"`, `"lg"`: the hint is 10 / 14 / 19 % of the screen height; or a fraction of it (`0.22`).                                             |
+| `placement`      | `"center"`                       | `"top"`, `"center"` or `"bottom"`.                                                                                                                 |
+| `offset`         | `[0, 0]`                         | Shift from the placement, in half-screens (`[0.5, 0]` = a quarter screen right).                                                                   |
+| `orbit`          | `35`                             | The hint's auto orbit, degrees to each side. `0` holds it still.                                                                                   |
+| `look`           | `HAND_HINT_LOOK`                 | The hint's look, merged over the default: ghost tint and opacities, halo, glow, highlight, your own material.                                      |
+| `loop`           | `HAND_HINT_LOOP`                 | The hint's loop: `seconds`, `hold`, `from`, `easing`, finger `stagger`.                                                                            |
+| `highlight`      | picked from the gesture          | The digits drawn brighter, or `false`.                                                                                                             |
+| `color`          | `"#ffffff"`                      | Text colour. The text has a soft dark shadow so it reads on any camera feed.                                                                       |
+| `textStyle` …    | —                                | `textStyle`, `nudgeTextStyle`, `renderText`, `motion`, `layout`: restyle the text, timings and proportions. See [Instructions](/api/instructions). |
+| `onComplete`     | —                                | Called once when the gesture is held long enough.                                                                                                  |
+| `onStatusChange` | —                                | Called once with the first status, then on every change (see below).                                                                               |
+| `hideOnComplete` | `true`                           | Fade out after completing. A small pop and a white ring play either way.                                                                           |
+| `active`         | `true`                           | `false` hides it and pauses detection.                                                                                                             |
+
+`onStatusChange` reports a `HandInstructionStatus`:
+
+- `"waiting"`: no usable hand in view.
+- `"detected"`: a hand in view, not doing the gesture.
+- `"holding"`: doing it; the hold is running.
+- `"done"`: completed.
+
+Use it for app logic or to announce progress to a screen reader. The instruction is drawn in WebGL, so
+it is invisible to assistive technology on its own.
+
+## Gestures
+
+| Name           | Default copy            | Recognised when                                           |
+| -------------- | ----------------------- | --------------------------------------------------------- |
+| `palm`         | Show your palm          | All five digits up.                                       |
+| `fist`         | Make a fist             | No digit up.                                              |
+| `peace`        | Show a peace sign       | Index and middle up, ring and pinky folded (thumb any).   |
+| `thumbUp`      | Give a thumbs up        | Thumb up and pointing up on screen, fingers folded.       |
+| `thumbDown`    | Give a thumbs down      | Thumb up and pointing down on screen, fingers folded.     |
+| `point`        | Point your index finger | Index up, the other fingers folded (thumb any).           |
+| `one` … `five` | Show 1 finger …         | Exactly that many digits up, thumb included, any of them. |
+| `rock`         | Rock on                 | Index and pinky up, middle and ring folded.               |
+| `loveYou`      | Sign I love you         | Thumb, index and pinky up, middle and ring folded.        |
+| `callMe`       | Call me                 | Thumb and pinky up, the rest folded.                      |
+| `ok`           | Make an OK sign         | Thumb and index tips touching, middle, ring and pinky up. |
+
+Several can match at once: a peace sign is also `two`. There is deliberately no gesture for the
+middle finger alone.
+
+## Detection only: `useHandGesture`
+
+```tsx
+import { useHandGesture } from "@vincentt-xr/sdk/instructions";
+
+const { status, holdRef, matchedRef, shapesRef, isInView, reset } = useHandGesture("thumbUp", {
+  hand: "any",
+  holdMs: 400,
+  limits: { thumbReach: 0.9 }, // optional, this detector only
+  onComplete: () => unlock(),
+});
+```
+
+`status` (`"waiting"` → `"holding"` → `"done"`) re-renders only when it changes. Read
+`holdRef.current.progress` (0..1), `matchedRef.current` (matched this frame) and
+`shapesRef.current.left` / `.right` (`HandShape`) inside `useFrame`, not in render. `reset()` asks
+again. The result is an `InstructionDetector`, so it can drive a custom `<Instruction>`.
+
+A rule, a match function or a definition works anywhere a gesture name does:
+
+```tsx
+useHandGesture({ digits: { index: true, pinky: true, middle: false }, thumbIndexTouch: false });
+useHandGesture((shape, side) => side === "left" && shape.curl.index > 0.8);
+```
+
+A `HandGestureRule` has these fields:
+
+- `digits`: per digit, `true` = up, `false` = folded, omitted = any.
+- `count`: the exact number of digits up.
+- `thumbDirection`: `"up"` or `"down"`.
+- `thumbIndexTouch`: whether the thumb and index tips touch.
+
+The classifier is pure. `analyzeHandShape(pose.world)` returns a `HandShape` with:
+
+- `extended` and `curl` (0..1) per digit;
+- `count`;
+- `thumbDirection`, relative to the screen;
+- `thumbIndexTouch`.
+
+`matchHandGesture(shape, nameOrRule)` tests one gesture; `matchingHandGestures(shape)` lists every
+name the shape satisfies. The default thresholds are `HAND_SHAPE_LIMITS` (read-only); override them per
+call with `analyzeHandShape(world, { limits })` or the hook's `limits`.
+
+## The hint hand: `HandGestureHint`
+
+```tsx
+<HandGestureHint gesture="ok" model={hand.scene} poses={hand.animations} height={0.4} />
+```
+
+The hand alone, for your own layout. It is fitted to `height` (parent units) and loops from a relaxed
+hand into the pose and back. The default look:
+
+- **Ghost:** translucent white, with a brighter rim.
+- **Highlight:** the gesture's key digits are brighter. Peace lights the index and middle, OK the
+  thumb and index, thumbs up the thumb. `highlight` overrides with a digit list or `false`.
+- **Halo:** a soft dark edge so it reads on light backgrounds (`look.halo`).
+- **Glow:** a soft white light from behind the hand, brightening as the gesture forms (`look.glow`).
+  It is a separate disc behind the hand, so it never shows as inner layers.
+- **Motion:** the fingers close one after another, pinky first (`loop`).
+- **Orbit:** a slow ±35° side-to-side turn (`orbit`, `orbitSeconds`).
+
+`look` is merged over `HAND_HINT_LOOK` (`false` turns a layer off; `look.material` replaces the
+ghost). `showHand` picks the chirality to show; the other one is the model
+mirrored.
+
+**Authored poses.** Each gesture's `hint.clip` names a single-key animation clip on the model.
+`hand-v1.glb` ships 11, posed in Blender: `palm`, `fist`, `peace`, `one`, `three`, `four`, `rock`,
+`loveYou`, `callMe`, `thumbUp`, `ok`. `two`, `point`, `five` and `thumbDown` reuse them. Pass the
+GLB's `animations` as `poses`. A model without the clip is posed procedurally from the same
+`HandHintPose`.
+
+**The view** is part of the hint too:
+
+- `yaw` turns the hand about the fingers' axis: 0 shows the palm (most gestures, including peace,
+  point and the counts), 180 the back of the hand (rock), 124 the thumb side (OK).
+- `roll` turns it in the screen plane: call me is held sideways, thumb up.
+- `aimThumb` turns a thumbs up or down until the thumb points that way.
+
+Every hint pose, authored and procedural, is tested to be recognised by the detector as its own
+gesture.
+
+The pieces underneath, also in `/instructions`:
+
+- `applyHandHintPose(binding, hint, amount, clips)`: `amount` 0..1, or one per digit
+  (`HandHintAmount`).
+- `readHandClipPose(clip)`.
+- The hold timing: `GESTURE_HOLD_DEFAULT_MS`, `GESTURE_HOLD_GRACE_MS`, `createGestureHoldState`,
+  `stepGestureHold`.
+
+## Example
+
+`/hand-instructions` covers:
+
+- every gesture as a chip, and a sequence mode that moves to the next gesture when one completes;
+- text, nudge, size, placement and hold-time controls;
+- a live readout of the status and what the classifier sees on each hand: digits up, count, thumb
+  direction, touch, matching gestures.
+
+Local clips in the git-ignored `example/src/assets/media/local/` are listed first.
+
+## Manual visual test plan
+
+Open `/hand-instructions`.
+
+1. **Every hint reads**: step through all 15 chips. Each hand forms its gesture, at its view: the palm
+   for peace, point and the counts, the back of the hand for rock, thumb side for OK, sideways for call me, thumb vertical for
+   thumbs up / down. The key digits are brighter, and it stays inside its slot through the orbit.
+2. **Looks**: the ghost is translucent with no inner fingers showing through, the halo separates it
+   from a white shirt, and the wrist fades out.
+3. **Motion**: the fingers close pinky first and settle. The hand swings about 35° each way.
+4. **Detection**: on the webcam, make each gesture with either hand. The readout shows `holding`,
+   then `done`. A wrong gesture stays `detected`. Peace also lists `two`.
+5. **Completion**: on done, a small pop and a white ring play. With sequence on, it fades out and the
+   next gesture comes in.
+6. **Nudge**: hide your hands for 3 s. The second line fades in and the hint pulses. Show a hand and
+   it fades out. With the nudge off it never appears.
+7. **Copy and layout**: type custom text (long text shrinks to fit), switch the sizes and top /
+   center / bottom placement. Nothing overlaps the screen edge.
+8. **Mirroring**: on the mirrored webcam the hint shows the hand you'd use, mirrored to match the
+   screen, with the thumb on the correct side.