@vincentt-xr/sdk 2.0.0-alpha.7 → 2.0.0-alpha.9

package/docs/api/hand-touch.mdx

@@ -0,0 +1,162 @@
+---
+title: "Touch objects"
+description: "Grab, carry, throw, push, poke, hover and tap 3D objects with the tracked hands, in a small built-in physics world."
+sidebarTitle: "Touch objects"
+icon: "hand-pointer"
+---
+
+`HandTouch` lets the hands play with 3D objects: pinch one to pick it up, carry it, let go to throw
+it, push it with the hand, poke it with a finger, point at it and tap it.
+
+- **The hand is a set of spheres.** A palm sphere and five fingertip spheres, sized from the tracked
+  hand.
+- **No physics engine.** A small world with gravity, a floor, walls, bounces and damping; objects
+  bump into each other.
+- **No exact depth needed.** One camera can't measure how far the hand is, so the hand reaches every
+  object on screen.
+
+Import from `@vincentt-xr/sdk/tracking`.
+
+## Quick start
+
+```tsx
+import { HandTouch, Touchable } from "@vincentt-xr/sdk/tracking";
+
+<HandTouch showColliders>
+  <Touchable shape="sphere" size={0.4} position={[-0.5, 0.5, 0]} color="#f97316" />
+  <Touchable shape="box" size={0.4} position={[0.5, 0.5, 0]} color="#38bdf8" />
+</HandTouch>;
+```
+
+Put `HandTouch` anywhere in the scene, not inside a tracker: it runs the hand tracker itself. Keep
+its parent at the world origin, because the world and the hands are in world space. Each
+`Touchable` inside it is an object. With no children it draws a ball or a cube in `color`, which
+glows while hovered or held.
+
+## What the hands do
+
+| interaction | how                                                                                                               | event                    |
+| ----------- | ----------------------------------------------------------------------------------------------------------------- | ------------------------ |
+| grab        | pinch (thumb tip on index tip) starting on or next to an object                                                   | `grab`                   |
+| carry       | keep pinching and move: the object follows the pinch                                                              | —                        |
+| throw       | open the pinch while moving: it flies at the hand's speed                                                         | `release` (with `speed`) |
+| push        | move the palm or a fingertip into an object                                                                       | `touch` (with `part`)    |
+| poke        | push with the index tip                                                                                           | `touch`                  |
+| hover       | point the index tip at an object                                                                                  | `hoverstart`, `hoverend` |
+| tap         | move the index tip into an object and out again within 350 ms; an object falling past a still finger is not a tap | `tap`                    |
+
+A pinch has to **start** near the object, so moving a closed pinch onto something doesn't pick it up.
+While the hand is closing to pinch, the thumb and index tips stop pushing, so they don't knock away
+what they are about to grab. A hand that disappears drops what it carried. The other hand can take
+an object from the first.
+
+## The world
+
+- **The box**: the screen at the hand plane (`z = 0`), `depth` scene units either side of it. The
+  floor is the bottom of the screen, raised by `floor`. Pass `bounds={{ min, max }}` to set it
+  yourself.
+- **Objects** collide as spheres. A cube uses its inscribed sphere and settles flat on the floor.
+  Balls roll.
+- **Units** are scene units at the hand plane. With the camera at `z = 5` and `fov` 45, the screen
+  is about 4.1 units tall and a palm about 0.5 to 1 unit long, depending on how far the hand is.
+
+## Props
+
+**`HandTouch`**
+
+| prop                                   | default              | meaning                                                                                      |
+| -------------------------------------- | -------------------- | -------------------------------------------------------------------------------------------- |
+| `gravity`                              | `9`                  | scene units per second², down                                                                |
+| `damping`                              | `0.4`                | air drag on motion and spin, per second                                                      |
+| `bounce`                               | `0.5`                | bounciness 0..1 (objects can override)                                                       |
+| `friction`                             | `4`                  | how fast objects stop sliding on the floor, per second                                       |
+| `grabReach`                            | `0.35`               | a pinch grabs within this many palm lengths of an object's surface                           |
+| `hoverReach`                           | `0.5`                | the index tip hovers within this many palm lengths                                           |
+| `tapMs`, `tapSpeed`                    | `350`, `1`           | a tap is shorter than `tapMs` and starts with the tip moving in at `tapSpeed` palm lengths/s |
+| `throwScale`, `maxSpeed`               | `1`, `14`            | throw speed multiplier and speed cap                                                         |
+| `flat`                                 | `true`               | hand-to-object contacts ignore depth                                                         |
+| `palmRadius`, `tipRadius`              | `0.5`, `0.12`        | collider sizes, in palm lengths (wrist to middle knuckle)                                    |
+| `pinchStart`, `pinchEnd`, `pinchReady` | `0.3`, `0.45`, `0.6` | thumb–index distances, in palm lengths: pinch on, pinch off, closing                         |
+| `smoothing`                            | `0.6`                | per-detection easing of the colliders, `1` = none                                            |
+| `hands`                                | `"both"`             | `"both"`, `"left"` or `"right"`                                                              |
+| `pointer`                              | off                  | the mouse or touch screen plays as a hand too (`true` or `{ radius }`, default 0.15)         |
+| `depth`, `floor`, `bounds`             | `0.6`, `0`           | the box                                                                                      |
+| `showColliders`                        | `false`              | draw the hand spheres (pink while pinching)                                                  |
+| `onEvent`                              | —                    | every event, for every object                                                                |
+| `paused`, `active`                     | —                    | freeze the objects; pause the hand tracker                                                   |
+
+**`Touchable`**
+
+| prop                               | default          | meaning                                                                                  |
+| ---------------------------------- | ---------------- | ---------------------------------------------------------------------------------------- |
+| `shape`                            | `"sphere"`       | `"sphere"` or `"box"`                                                                    |
+| `size`                             | `0.4`            | diameter or edge, scene units                                                            |
+| `position`, `velocity`             | origin, still    | where it starts; after that the world moves it                                           |
+| `mass`                             | `size³`          | bigger objects push smaller ones further                                                 |
+| `bounce`                           | the world's      | bounciness 0..1                                                                          |
+| `grabbable`, `pushable`, `gravity` | `true`           | what the hands can do with it, and whether it falls                                      |
+| `color`                            | `"#f97316"`      | the built-in look's colour                                                               |
+| `onGrab`, `onRelease`, `onTap`     | —                | the events, with `hand` (`"left"`, `"right"` or `"pointer"`)                             |
+| `onTouch`                          | —                | a hand part started touching it: `part` (`"palm"`, `"thumbTip"` … `"pinkyTip"`), `speed` |
+| `onHoverChange`                    | —                | `(hovered, hand)`                                                                        |
+| `children`                         | a ball or a cube | the look, centred on the object; a function gets `{ hovered, grabbed, hand }`            |
+
+A button that only hovers and taps: `<Touchable gravity={false} pushable={false} grabbable={false}>`.
+
+## The pointer
+
+`pointer` makes the mouse or the touch screen a hand too: handy on a desktop without a camera, and
+for testing.
+
+- It hovers what it is over.
+- A press on an object grabs it; letting go throws it.
+- A quick click taps it.
+- A press elsewhere, dragged across objects, pushes them.
+
+## Custom drivers
+
+`useHandTouch()` inside a `HandTouch` returns `{ world, hands }`: the live `TouchWorld` and the
+`TouchHand` colliders. The pure pieces are in `/low-level`:
+
+- `TouchWorld`: `add`, `remove`, `pick`, `step(dt, hands)` returning `TouchEvent[]`, `bounds`,
+  `settings`.
+- `createTouchHand` and `updateTouchHand(state, sceneJoints | null, timeMs)`: a hand's colliders from
+  `handPoseToScene`.
+- `updatePointerHand`: the pointer as a hand.
+
+## Limits
+
+- **Depth.** With `flat`, a hand in front of an object and one behind it both touch it. Objects are
+  spread in depth only for looks.
+- **Shapes.** Spheres and cubes, both colliding as spheres; no stacking.
+- **Tracking.** Fast throws depend on the hand tracker's frame rate. A hand that blurs out mid-throw
+  drops the object.
+
+## Example
+
+`/hand-touch`: balls and cubes on the floor of the screen. A tap changes an object's colour.
+
+- `?objects=mix|balls|cubes`, `?count=6`
+- `?gravity=0` (they float), `?bounce=0.5`
+- `?colliders=0` hides the hand spheres
+- `?pointer=1` lets the mouse or touch play too
+
+## Manual visual test plan
+
+Open `/hand-touch` on the webcam with the colliders on.
+
+1. **Colliders**: raise a hand. A large sphere sits on the palm and a small one on each fingertip;
+   they follow the hand and grow as it comes closer. Pinch: they turn pink.
+2. **Push**: sweep the open hand into the objects. They fly off faster than the hand and bounce off
+   the walls and each other.
+3. **Poke**: point and poke a ball with the index tip. It moves away from the finger.
+4. **Grab and carry**: pinch beside a ball. It glows, grows a little and follows the pinch. Move a
+   closed pinch onto another ball: nothing happens.
+5. **Throw**: carry a ball and let go mid-swing. It flies along the swing, then falls and bounces.
+6. **Hover and tap**: point at a cube (it glows), tap it quickly: it changes colour. Hold the finger
+   in it: no colour change. Let an object fall past a still finger: no colour change.
+7. **Both hands**: pass a ball from one hand to the other by pinching it with the second hand.
+8. **Loss**: carry a ball and drop the hand out of view. The ball falls.
+9. **Pointer** (`?pointer=1`, desktop): drag across the objects to push them, press on one to carry
+   and throw it, click one to tap it.
+10. **Phone**: about 30 fps with six objects.