@vincentt-xr/sdk 2.0.0-alpha.7 → 2.0.0-alpha.8

package/docs/api/iris-tracking.mdx

@@ -0,0 +1,242 @@
+---
+title: "Iris Tracking"
+description: "Stable left/right iris data from the shared face tracker, and an iris effect that draws a shader or image material over each iris."
+sidebarTitle: "Iris Tracking"
+icon: "eye"
+---
+
+Iris tracking reads the ten refined iris landmarks the face tracker already
+produces. It starts no second detector: `useIrisTracking` acquires the same
+shared `XRModel.FACE_TRACKER` as `FaceTracker` and `useFaceInfo`, and
+`<IrisEffect>` renders over it.
+
+```tsx
+import { useIrisTracking } from "@vincentt-xr/sdk/tracking";
+import { IrisEffect } from "@vincentt-xr/sdk/face-effects";
+
+// Rendering: built-in procedural iris, or an image, or your own material.
+<IrisEffect mode="shader" color="#4fa3ff" opacity={0.85} smoothing={15} />;
+<IrisEffect mode="image" textureUrl="/iris.png" scale={1.1} />;
+
+// Data: read inside useFrame.
+const { result, status } = useIrisTracking({ faceIndex: 0, smoothing: 15 });
+useFrame(() => {
+  if (!result.detected) return;
+  const { center, radius, offset, confidence } = result.left;
+});
+```
+
+## Landmarks and sides
+
+| Side (person's) | Center | Ring (outer, top, inner, bottom) | Inner / outer corner | Upper / lower lid |
+| --------------- | ------ | -------------------------------- | -------------------- | ----------------- |
+| `left`          | 473    | 474, 475, 476, 477               | 362 / 263            | 386 / 374         |
+| `right`         | 468    | 469, 470, 471, 472               | 133 / 33             | 159 / 145         |
+
+Sides follow the face-mesh topology, so they are the person's anatomical
+sides on every source: mirrored webcam, straight video or image. The data
+needs the refined mesh (478 points); with fewer landmarks each eye reports
+`detected: false`. These indices are exported as `IRIS_LANDMARKS`. The
+allocation-free helpers behind the hook (`createIrisResult`,
+`updateIrisResult`, `computeIrisEye`, `IrisLead`) live on
+`@vincentt-xr/sdk/low-level` for pipelines that already hold their own
+landmarks.
+
+## Coordinate spaces
+
+- `center`, `ring`, `eyeCenter`: MediaPipe image space, 0..1 with a top-left
+  origin and Y down, the same space as `FaceInfo.landmarks` and
+  `detectionResults`. `z` is tracker-relative depth.
+- `trackerCenter`: SDK tracker space (center origin, X right, Y up, Z toward
+  the viewer), the same as hand and body points.
+- `radius`, `eyeWidth`: image-width units (half the horizontal ring diameter
+  and the corner-to-corner distance). To draw in world space, unproject the
+  center and a point `radius` to its right and take their distance, which is
+  what `<IrisEffect>` does.
+- `offset`: the iris center relative to the eye center, each axis divided by
+  half the eye width and clamped to [-1, 1]; +x toward the person's left, +y
+  up, on every source (the horizontal sign is corrected on the mirrored
+  webcam feed, so it never flips with the camera). **This is the iris position inside the eyelid opening, not a gaze
+  direction**: it ignores head pose and eyeball geometry. Use it for
+  pupil-follow and shader inputs, not for "where is the person looking".
+- `openness`: eyelid gap divided by eye width (about 0.25–0.35 open, 0 closed).
+
+Mirroring: the session mirrors the webcam feed and no other source, and the
+landmarks are already in that feed space, so positions need no flip. The
+`mirrored` prop on `<IrisEffect>` flips the material horizontally (image
+orientation, shader +x), nothing else.
+
+## Confidence and lifecycle
+
+MediaPipe reports no score for iris points and emits them even when the eye
+is closed, so confidence is derived: `eye.confidence` is `openness / 0.2`
+clamped to 0..1, scaled by the hold decay. `result.confidence` is 1 while the
+face is live and decays linearly to 0 over `holdMs` while `held`. Gate
+effects on these; the built-in materials fade with `irisConfidence`.
+
+`useIrisTracking` returns `status` (`loading` → `ready` / `error`) for the
+shared face model plus `result.detected` / `held` / `version` /
+`timestampMs` / `faceIndex`. The `result` object is stable and mutated in
+place; read it inside `useFrame`.
+
+## Smoothing, hold and face selection
+
+- Dropouts go through `FaceTrackMemory`, the SDK's one bridge: `holdMs` 250
+  and `predictMs` 0 by default, like every other face surface. A face that
+  stays missing longer than `holdMs` is dropped and `<IrisEffect>` hides at
+  once, so no stale eye effect lingers.
+- `smoothing` runs the same One Euro filter as `TrackingAnchor` on the
+  centers and radius (default 0 = raw; ~10 tames jitter at rest). It is a
+  low-pass whose floor cutoff is `smoothing × 0.1` Hz, so 15 attenuates and
+  lags a 2–3 Hz head shake: keep it low when exact following matters. It
+  resets on re-acquire, so the iris never slides in from an old position.
+- `targetFps` requests an inference cap for the shared face model (0 keeps
+  the session's `perf.targetFps`, 24 in the template). The first mounted face
+  consumer's value wins, so pass the same number everywhere the face model
+  is acquired, and note that a later change only takes effect once every
+  holder has released the model (a lone `<IrisEffect>` re-acquires and
+  re-initializes it; a session pipeline entry pins the first value for the
+  session). The example makes the effect the only holder so its panel value
+  is the real rate. **60 gives the best following, but use it with caution**: it runs
+  face inference 2.5× as often as the template default and will heat and
+  throttle phones. Ship it only when the eye effect is the point of the
+  experience and you have checked the target devices; otherwise keep the
+  session default (or 30) and let `leadMs` bridge the gap.
+- `leadMs` (default 80, 0 disables) bridges throttled inference to the render
+  rate: between samples the centers advance along their last
+  sample-to-sample velocity for at most `leadMs`, so a 24 fps face model
+  reads continuous at 60+ fps instead of hold-and-jump. When the lag itself
+  is the problem, raise the face model's `targetFps` first (the example
+  requests 60).
+- `faceIndex` / `numFaces` select the face exactly like `useFaceInfo`: the
+  index is the result-array position, not a persistent identity, and the
+  first mounted face consumer decides `numFaces`.
+
+## `<IrisEffect>` props and materials
+
+`<IrisEffect>` takes the flat `IrisEffectSettings` (`eye`: both | left |
+right, `mode`: shader | image, `color`, `textureUrl`, `tintImage` (multiply the
+image by `color`; off by default so images show as authored), `scale`, `opacity`,
+`targetFps`, `smoothing`, `leadMs`, `mirrored`, `faceIndex`, `holdMs`) plus `numFaces`, `active`,
+`predictMs`, `texture` (a pre-loaded three texture, beats `textureUrl`),
+`material`, `renderOrder` and `onTracking(handle)`.
+
+Each eye is a quad spanning the iris diameter, positioned on the camera's
+z=0 plane. Every material receives the same uniforms (`IrisUniforms`):
+
+| Uniform          | Type        | Meaning                            |
+| ---------------- | ----------- | ---------------------------------- |
+| `irisCenter`     | `vec2`      | Image-space center                 |
+| `irisRadius`     | `float`     | Image-width radius                 |
+| `irisOffset`     | `vec2`      | Position inside the eye, [-1, 1]   |
+| `irisConfidence` | `float`     | 0..1, fade with it                 |
+| `irisOpenness`   | `float`     | Eyelid gap / eye width             |
+| `time`           | `float`     | Seconds since mount                |
+| `color`          | `vec3`      | Tint                               |
+| `opacity`        | `float`     |                                    |
+| `mirror`         | `float`     | 1 when mirrored                    |
+| `tintImage`      | `float`     | 1 to multiply the image by `color` |
+| `map`            | `sampler2D` | The image in `image` mode          |
+
+A minimal custom material:
+
+```tsx
+import { IrisEffect, type IrisMaterialContext } from "@vincentt-xr/sdk/face-effects";
+
+const ring = ({ uniforms }: IrisMaterialContext) =>
+  new THREE.ShaderMaterial({
+    uniforms, // the shared, live uniforms
+    vertexShader: `varying vec2 vUv; void main(){ vUv = uv; gl_Position = projectionMatrix * modelViewMatrix * vec4(position, 1.0); }`,
+    fragmentShader: `
+      uniform float irisConfidence; uniform vec3 color; uniform float opacity; varying vec2 vUv;
+      void main(){
+        float r = length(vUv - 0.5) * 2.0;
+        float ring = smoothstep(0.6, 0.7, r) * (1.0 - smoothstep(0.9, 1.0, r));
+        gl_FragColor = vec4(color, ring * opacity * irisConfidence);
+        #include <colorspace_fragment>
+      }`,
+    transparent: true,
+    depthTest: false,
+    depthWrite: false,
+  });
+
+<IrisEffect material={ring} color="#ffcc00" />;
+```
+
+The factory runs once per eye for each `material` identity; the effect
+disposes what a factory created when the factory changes or on unmount,
+while an instance you pass stays yours. Read the live texture from
+`uniforms.map`: `context.texture` is only the texture known when the factory
+ran, often null. The built-in
+image shader shows it as authored, multiplies it by `color` only when
+`tintImage` is on, and fades with confidence.
+
+## When to use what
+
+- **Iris tracking**: eye decorations, pupil overlays, iris recolouring,
+  shader inputs tied to where the iris sits in the eye.
+- **Blendshapes** (`useFaceInfo` / `useFaceAction`): blinks, winks, squints,
+  eye-look-left/right/up/down categories as discrete expressions.
+- **Gaze estimation**: not provided. `offset` is not gaze; a real estimate
+  needs head pose and eye geometry.
+- **PinToMesh / FaceMesh materials**: content that must deform with the skin
+  around the eye rather than sit on the iris.
+
+## Hiding the iris behind the eyelids
+
+The quads are flat and draw over the lids. To clip them to the eye openings
+without a dedicated mask, draw the face mesh over them with the camera feed
+and no eye-opening triangles (the example's "eyelid mask" toggle). The
+material samples the feed through the mesh's live `faceUv` attribute with
+depth testing off, so it always lands on top of the quads:
+
+```tsx
+import { CanonicalFaceMeshRenderer, IrisEffect } from "@vincentt-xr/sdk/face-effects";
+import { useXRCameraTexture } from "@vincentt-xr/sdk/low-level";
+
+const feed = new THREE.ShaderMaterial({
+  uniforms: { cameraTexture: { value: useXRCameraTexture() } },
+  vertexShader: `attribute vec2 faceUv; varying vec2 vUv;
+    void main(){ vUv = faceUv; gl_Position = projectionMatrix * modelViewMatrix * vec4(position, 1.0); }`,
+  fragmentShader: `uniform sampler2D cameraTexture; varying vec2 vUv;
+    void main(){ gl_FragColor = texture2D(cameraTexture, vUv); #include <colorspace_fragment> }`,
+  transparent: true, // the quads are transparent; an opaque mask would draw before them
+  depthTest: false,
+  depthWrite: false,
+});
+
+// IrisEffect renders at order 100; the mask draws after it.
+<IrisEffect />;
+<CanonicalFaceMeshRenderer
+  showEyeGeometry={false}
+  showEyeMask={false}
+  renderOrder={101}
+  materialOverride={feed}
+/>;
+```
+
+Memoize the material on the camera texture and dispose it on unmount, as
+the example does.
+
+## Known limitations
+
+- Single-person per `faceIndex`; multi-face scenes mount one effect per face.
+- Iris points are present even for closed eyes; `confidence` is the gate.
+- The quads sit on the z=0 plane without eyeball curvature or head pose.
+- Hold replays the last position; with `predictMs` > 0 a sharp reversal can
+  overshoot for up to that window.
+
+## Manual validation
+
+Run `/iris-tracking`:
+
+| Case                 | Steps                                   | Expect                                                             |
+| -------------------- | --------------------------------------- | ------------------------------------------------------------------ |
+| Shader material      | default, change color / scale / opacity | Both irises recoloured live; pupil follows the iris offset         |
+| Image material       | mode `image`, presets or a URL          | Image covers each iris, tinted by color; mirror flips it           |
+| Custom material      | "custom shader (ring)"                  | Pulsing ring from the shared uniforms                              |
+| Tracking loss        | cover the face / "Person Shaking Head"  | Effect holds ~250 ms then disappears; no stale eyes                |
+| Multi-face           | "Two People Smiling", face index 0 / 1  | Effect moves to the selected face                                  |
+| Mirrored camera      | webcam                                  | Left/right stay anatomical; the eye on the viewer's left is `left` |
+| Portrait / landscape | resize the pane                         | Quads stay centred on the irises at the right size                 |
+| Closed eyes          | blink                                   | Effect fades with confidence                                       |

package/docs/api/multi-subject.mdx

@@ -0,0 +1,95 @@
+---
+title: "Multi-Subject Tracking Policy"
+description: "The identity and association contract for faces, hands, bodies and holistic results: what an index means, how many subjects each tracker reports, how to select one, what happens on loss and reappearance, and how subjects are matched across trackers."
+sidebarTitle: "Multi-Subject Policy"
+icon: "people-group"
+---
+
+## Identity contract
+
+- **An index is a result-array position, not an identity.** `faceIndex`,
+  `handIndex`, `getFace(i)`, `getLandmarks2D(i)` read position `i` of the
+  current frame's result. The order is whatever MediaPipe emits; it can
+  change between frames when subjects move, overlap, or one is lost. Nothing
+  in the SDK promises that `faceIndex: 1` is the same person two frames
+  later.
+- **No learned identity.** The SDK never recognises people. The only
+  continuity it offers is spatial: the `sticky` selection below follows the
+  subject nearest to the previously selected one.
+- **Single-subject APIs are unchanged.** `useFaceInfo()`, `<FaceTracker>`,
+  `<HandTracker>`, `<TrackingAnchor>` keep reading index 0 by default; the
+  policy is an opt-in option on top.
+
+## Maximum subjects
+
+| Tracker  | Max | Default | Option                                                                                        |
+| -------- | --- | ------- | --------------------------------------------------------------------------------------------- |
+| face     | 10  | 1       | `numFaces` on `<FaceTracker>` / `useFaceInfo(s)`; the first acquirer of the shared model wins |
+| hand     | 2   | 2       | none (two hands, `handedness` labels them)                                                    |
+| gesture  | 1   | 1       | none                                                                                          |
+| body     | 1   | 1       | none                                                                                          |
+| holistic | 1   | 1       | none (single person by design)                                                                |
+
+`XR_MAX_SUBJECTS` carries this table. Every consumer that needs more than one
+face must ask for it; a consumer that asks for fewer than the first acquirer
+gets the larger count.
+
+## Selecting one subject
+
+```tsx
+import { useFaceInfo, selectSubject, faceCandidates } from "@vincentt-xr/sdk/tracking";
+
+// Hook form (faces): pick by policy instead of a fixed index.
+const face = useFaceInfo({ numFaces: 4, subject: { selection: "largest" } });
+const follow = useFaceInfo({ numFaces: 4, subject: { selection: "sticky", stickyRadius: 0.2 } });
+
+// Pure form (any tracker): build candidates, select, carry the memory.
+const result = selectSubject(faceCandidates(results.faceLandmarks), policy, previousMemory);
+```
+
+| `selection`              | Picks                                                                                                                                                                     |
+| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `index`                  | position `policy.index` (default 0) — the historical behaviour                                                                                                            |
+| `largest`                | the biggest subject (closest to the camera)                                                                                                                               |
+| `nearest-center`         | closest to the feed centre                                                                                                                                                |
+| `leftmost` / `rightmost` | by feed x, as shown in the preview                                                                                                                                        |
+| `sticky`                 | nearest to the previously selected subject within `stickyRadius` (feed units, default 0.2); same `label` (handedness) preferred; otherwise `fallback` (default `largest`) |
+
+`selectSubject` returns `{ index, reason, memory }`; `index` is `null` when
+there is no candidate. `faceCandidates(landmarkArrays)` and
+`jointCandidates(handsOrBodies)` build candidates (feed-space centre and
+size) from face landmarks or tracker-space joints.
+
+## Loss and reappearance
+
+- A subject that disappears for a few frames is bridged by the tracker's
+  hold (`holdMs`, 250 ms by default, through `FaceTrackMemory` / the readers'
+  hold): the last result is replayed, then dropped.
+- `sticky` keeps its memory through a miss, so a subject that reappears
+  near where it was is re-selected; one that reappears elsewhere is a new
+  subject and the policy's `fallback` decides. When the selection switches
+  to the fallback, the memory moves with it (a deliberate switch, not a jump
+  back later).
+- Reappearance never restores a previous index: read the policy's pick, not
+  a remembered position.
+
+## Association across trackers
+
+`associateSubjects(primary, secondary, maxDistance = 0.35)` greedily matches
+the nearest centres between two candidate lists (face ↔ body, hand ↔ body),
+each subject at most once, ignoring pairs farther than `maxDistance` (feed
+units). That proximity-in-the-same-frame rule is the whole association
+contract; with the single-person body / holistic models it reduces to
+"the body belongs to the nearest face".
+
+## Manual validation
+
+Run `/multi-subject` (starts on the two-people image, `numFaces` 4):
+
+| Case       | Steps                                           | Expect                                              |
+| ---------- | ----------------------------------------------- | --------------------------------------------------- |
+| Candidates | two / three people images                       | one grey dot per face, count in the status line     |
+| Rules      | largest / nearest-center / leftmost / rightmost | the cyan ring moves to the matching face            |
+| Index      | `index` with 0..3                               | ring on that array position, none when out of range |
+| Sticky     | `sticky` on "Two People Smiling" video          | the ring stays on the same person as they move      |
+| Loss       | cover a face (webcam)                           | ring holds ~250 ms, then follows the fallback       |

package/docs/api/person-cutout.mdx

@@ -0,0 +1,136 @@
+---
+title: "Person Cutout"
+description: "Background replacement as a two-layer ScreenImage composition: any background media below, the person-masked live camera above."
+sidebarTitle: "Person Cutout"
+icon: "user"
+---
+
+`PersonCutout` is the live camera, masked to the person by portrait
+segmentation, rendered as a full-screen `ScreenImage`. Put any background
+surface behind it with a lower `renderOrder` and you have background
+replacement without assembling masks, materials or coordinates yourself.
+
+```tsx
+import { FULL_SCREEN_ANCHORS, PersonCutout, ScreenImage, ScreenSpaceUI } from "@vincentt-xr/sdk";
+
+<ScreenSpaceUI>
+  {/* 1. Background: image, video texture or GIF — anything ScreenImage shows. */}
+  <ScreenImage
+    src="/backgrounds/beach.jpg"
+    fit="cover"
+    anchors={FULL_SCREEN_ANCHORS}
+    renderOrder={0}
+  />
+  {/* 2. Foreground: the camera, person only. */}
+  <PersonCutout renderOrder={100} />
+</ScreenSpaceUI>;
+```
+
+## The composition contract
+
+- **Background** is any full-screen surface drawn first: a `ScreenImage`
+  (`{ kind: "image" }`, `{ kind: "gif" }`, or `{ kind: "texture" }` with a
+  `THREE.VideoTexture`), or a custom shader plane inside
+  `<ScreenTransform anchors={FULL_SCREEN_ANCHORS}>`. Give it a `renderOrder`
+  below the cut-out's (default 100).
+- **Foreground** is `PersonCutout`: a `ScreenImage` whose `source` is the
+  session camera texture and whose `maskTexture` is the portrait segmentation
+  mask. It accepts the usual `ScreenImage` props (`opacity`, `fit`, `anchors`,
+  `blendMode`, `flipX`, `renderOrder`, `visible`) except `source` and the mask
+  props, which it owns.
+- **Ordering** is `renderOrder`; both layers are overlay-style screen planes
+  with depth testing off, so only the order matters.
+
+## Alignment: aspect ratio, mirroring, resolution
+
+- The camera texture is already cover-cropped to the viewport and mirrored
+  for webcams by the session, and the segmentation mask is produced in that
+  same feed space. `PersonCutout` samples the mask through the image's own
+  fit transform (`fit="cover"` by default), so mask and camera stay aligned
+  under any aspect ratio and any further crop or flip you apply.
+- Background media is not mirrored. If a background has to match a mirrored
+  webcam (a sign you are sitting in front of), set `flipX` on that
+  `ScreenImage`.
+- The mask is a render-size canvas texture with a half-pixel blur, so edges
+  are soft at the display resolution. The segmentation `quality` option on
+  the model trades edge detail for cost.
+
+## Mask, opacity, invert
+
+The SDK mask carries its signal in RGB: white (255) on the person, black on
+the background, with alpha 255 everywhere (three's `alphaMap` reads the green
+channel, which is how `VideoBackground` consumes the same texture). The
+cut-out reads `maskChannel="green"`, keeping the person and dropping the
+background. `invert` flips that to show only the background with the person
+cut out.
+`opacity` fades the whole person layer. `usePersonMaskTexture()` returns the
+raw mask texture with its `status` for custom compositions, and `ScreenImage`
+itself now accepts `maskTexture`, `maskChannel` (`alpha` | `red` | `green` |
+`luminance`), `maskInvert`, `maskFeather` and `maskErode` for any image.
+
+## Edge quality: feather and erode
+
+The segmenter's mask is hard-edged and sits a texel or two outside the person,
+so the raw cut-out carries a fringe of the source background (green-screen
+spill, a light halo on dark backgrounds). Two props clean it up:
+
+- `maskFeather` — radius in mask texels of a 3×3 blur over the mask. `0` keeps
+  the hard edge. Texel units keep the softness the same on screen whatever
+  resolution the mask arrives at.
+- `maskErode` — `0..1` of the feather width to shrink the kept region by. The
+  blurred edge is re-thresholded, so the silhouette moves inward and the fringe
+  falls outside it. It does nothing without feather.
+
+`PersonCutout` defaults to `maskFeather={4}` and `maskErode={0.7}`, chosen on
+the SDK's green-screen clips with the 720-px portrait mask. Raise erode when
+spill remains, lower it when fingers or hair tips disappear. Neither prop fixes
+the mask trailing a fast-moving arm; that is the segmenter's temporal lag.
+
+## Loading and fallback
+
+The segmentation model downloads on first use. `PersonCutout` reports
+`loading` → `ready` | `error` through `onStatusChange`. While not ready it
+follows `fallback`: `"camera"` (default) draws the unmasked camera so the
+experience degrades to a plain feed, `"hidden"` draws nothing so only the
+background shows. On error the same fallback applies and `error` carries
+the reason.
+
+## When to use this instead of 3D
+
+- **Use this 2D path** for background replacement, virtual backgrounds,
+  person-in-a-scene stickers, and any effect that only needs "the person
+  versus the rest of the frame". It is one quad, one mask, no geometry.
+- **Use the face mesh** when something must stick to the face surface with
+  correct perspective and shading (lipstick, face paint, UV-mapped textures).
+- **Use the head occluder** when 3D objects must disappear behind the head
+  (glasses arms, hats seen from the side). A 2D mask cannot occlude in depth.
+- The 2D mask also covers the body, hair and clothes, which neither the face
+  mesh nor the head occluder does.
+
+## Performance
+
+One extra full-screen quad with a masked shader; the segmentation model
+runs at its own `targetFps` (24 by default). Share the model: `VideoBackground`
+with `segmentation="portrait"`, `usePersonMask` overlays and `PersonCutout`
+all consume the same instance.
+
+## Manual visual test plan
+
+Run `/person-cutout` in the playground:
+
+| Case                     | Source / background                     | Expect                                                                          |
+| ------------------------ | --------------------------------------- | ------------------------------------------------------------------------------- |
+| Image background         | video source, Image                     | Person on the photo; no camera background visible around the edges              |
+| Video background         | video source, Video                     | Background plays behind the person; both layers keep playing                    |
+| GIF background           | video source, GIF                       | GIF animates behind the person                                                  |
+| Shader background        | video source, Shader                    | Animated plane behind the person; person edges unaffected                       |
+| Opacity                  | any, opacity 0.5                        | Person fades over the background; background unchanged                          |
+| Show background only     | any, invert                             | Person region shows the background through the camera layer                     |
+| Edge feather / erode     | video source, feather 0 vs 4, erode 0.7 | Feather 0 shows a green fringe on sleeves; the defaults remove it               |
+| Aspect / orientation     | resize the pane, rotate device          | Person stays aligned with the camera; background re-covers the screen           |
+| Mirrored webcam          | webcam                                  | Person mirrored like the feed; text on a background not mirrored unless `flipX` |
+| Movement                 | "Person Shaking Head", "Person ThumbUp" | Mask follows head and hands; thin fingers may soften                            |
+| Lighting                 | webcam, bright vs dim                   | Mask holds under both; dark clothes on dark background is the hard case         |
+| Loading, fallback camera | reload                                  | Plain camera until `ready`, then the background appears                         |
+| Loading, fallback hidden | reload                                  | Only the background until `ready`                                               |
+| Multi-person             | "Two People Smiling"                    | Both people kept; the segmenter is person-class                                 |

package/docs/api/product-ar.mdx

@@ -0,0 +1,111 @@
+---
+title: "Product Placement AR"
+description: "View-in-your-space AR for product pages on top of Google's model-viewer: capability detection, the ProductViewer wrapper, the useProductAr hook, and the camera handoff with the tracking session."
+sidebarTitle: "Product AR"
+icon: "cube"
+---
+
+Placement AR puts a product on the floor or a wall through the device's
+native path: WebXR (Android Chrome), Scene Viewer (Android) or Quick Look
+(iOS, from a USDZ). It is a different mode from the SDK's camera session,
+which has no world tracking. The SDK wraps Google's `<model-viewer>` for it
+rather than rebuilding it; the rule for every page: **placement AR is
+model-viewer, face/body AR is the session, never both at once.**
+
+```tsx
+import { ProductViewer, useProductAr } from "@vincentt-xr/sdk/product";
+
+const Product = () => {
+  const ar = useProductAr();
+  return (
+    <>
+      <ProductViewer
+        src="/hat.glb"
+        iosSrc="/hat.usdz"
+        poster="/hat.webp"
+        alt="Hat"
+        onReady={ar.bind}
+      />
+      {ar.supportedModes.length > 0 && (
+        <button onClick={() => ar.activate()} disabled={!ar.canActivate}>
+          View in your space
+        </button>
+      )}
+    </>
+  );
+};
+```
+
+## Install
+
+`@google/model-viewer` is an optional peer dependency: add it to the app
+that renders products. `<ProductViewer>` imports it lazily on first render,
+so other pages never pay for it. It bundles its own three.js (it cannot share
+the SDK's), roughly 1 MB; keep it on product routes only.
+
+## Capability
+
+`detectXRCapabilities` / `useXRCapabilities` report `placement`:
+
+| Field         | How it is detected                                                                  |
+| ------------- | ----------------------------------------------------------------------------------- |
+| `webxr`       | `navigator.xr.isSessionSupported("immersive-ar")`                                   |
+| `sceneViewer` | Android Chrome / Samsung Internet user agent                                        |
+| `quickLook`   | `<a rel="ar">` support (iOS Safari and WebKit browsers)                             |
+| `any`         | one of the above; otherwise `reason.code` `placement-unsupported` (not recoverable) |
+
+Offer "View in AR" only when `any` is true; desktop shows the viewer alone.
+
+## `<ProductViewer>`
+
+| Prop                                                                                         | model-viewer attribute                           |
+| -------------------------------------------------------------------------------------------- | ------------------------------------------------ |
+| `src`, `iosSrc`, `poster`, `alt`                                                             | `src`, `ios-src`, `poster`, `alt`                |
+| `ar` (default true), `arModes`                                                               | `ar`, `ar-modes` (preference order)              |
+| `placement: "floor" \| "wall"`                                                               | `ar-placement`                                   |
+| `scale: "auto" \| "fixed"`                                                                   | `ar-scale` (`fixed` = true-to-size, no pinch)    |
+| `cameraControls`, `autoRotate`, `loading`, `environmentImage`, `shadowIntensity`, `exposure` | same names, kebab-cased                          |
+| `attributes`                                                                                 | anything else, passed straight through           |
+| `onLoad`, `onError`, `onArStatus`, `onReady(element)`                                        | `load`, `error`, `ar-status` events; the element |
+
+Children are slotted into model-viewer (`slot="ar-button"`, `slot="poster"`,
+hotspots).
+
+## `useProductAr()`
+
+Returns `{ status, supportedModes, canActivate, reason, loading, error, bind, activate, handleStatus }`.
+`status` is model-viewer's `ar-status` (`not-presenting`, `session-started`,
+`object-placed`, `failed`) or `unavailable` when no mode is supported.
+`activate()` must run from a user gesture.
+
+### Camera handoff
+
+Only WebXR runs in the page and needs the camera the tracking session
+holds; Scene Viewer and Quick Look open a native activity and never touch
+it. When the hook runs under an `<XRProvider>` whose session is `running` or
+`paused` on the webcam and the mode about to be used is `webxr`,
+`activate()` stops the session first and restores the same facing and
+lifecycle (a paused session comes back paused) when `ar-status` returns to
+`not-presenting` (or `failed`). `handoffCamera: false` opts out. Stream /
+canvas sources are left alone. The pure form is
+`releaseSessionForAr(session, mode)`. Keep `arModes` on the viewer and
+`modes` on the hook identical; both default to `DEFAULT_PRODUCT_AR_MODES`.
+
+## Assets
+
+- GLB for every platform; keep it under a few MB, Draco / KTX2 compressed.
+- USDZ for iOS Quick Look via `iosSrc`; without it model-viewer exports one
+  on the fly (slower, lower fidelity). The GLB → USDZ step is a publishing
+  step (see the asset pipeline issue), never runtime code.
+- Host both on the CDN with long cache headers.
+
+## Manual validation
+
+Run `/product-ar`:
+
+| Device                          | Expect                                                                                                                                                             |
+| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| Desktop                         | viewer with orbit; `supported modes: none`, button disabled                                                                                                        |
+| Android Chrome                  | `scene-viewer` (and `webxr` on ARCore devices); tap → Scene Viewer / WebXR places the astronaut; status log `session-started` → `object-placed` → `not-presenting` |
+| iPhone                          | `quick-look`; tap → Quick Look opens the USDZ                                                                                                                      |
+| With a tracking session running | the session reads `stopped` during AR and `running` after                                                                                                          |

package/docs/api/product-details.mdx

@@ -0,0 +1,78 @@
+---
+title: "Product Variants, Hotspots & Dimensions"
+description: "Material variants (colour / size), surface-anchored hotspots with serializable authoring data, and true-size dimension labels on a ProductViewer."
+sidebarTitle: "Product Details"
+icon: "tags"
+---
+
+Builds on `<ProductViewer>` (Product Placement AR). Three details a product
+page needs on the same model: switching looks, pointing at parts, and
+telling the real size.
+
+```tsx
+import {
+  ProductViewer,
+  ProductHotspot,
+  ProductDimensions,
+  useProductVariants,
+  useProductHotspots,
+  useProductDimensions,
+  type ProductViewerElement,
+} from "@vincentt-xr/sdk/product";
+
+const [el, setEl] = useState<ProductViewerElement | null>(null);
+const variants = useProductVariants(el); // { variants, variant, setVariant }
+const dimensions = useProductDimensions(el); // { size: [w, h, d], center } in metres, null until loaded
+const hotspots = useProductHotspots(el, { initial: fromCms, onChange: save });
+
+<ProductViewer src="/shoe.glb" scale="fixed" onReady={(e) => setEl(e)}>
+  {hotspots.hotspots.map((h) => (
+    <ProductHotspot key={h.id} hotspot={h} onClick={open} />
+  ))}
+  <ProductDimensions dimensions={dimensions} unit="metric" />
+</ProductViewer>;
+```
+
+## Variants
+
+`useProductVariants(element)` reads the model's `KHR_materials_variants`
+after load and sets `variantName` on the element. Variants change materials
+only, so they carry into AR wherever the platform renders the glTF (WebXR,
+Scene Viewer). Quick Look renders the USDZ as exported: ship one USDZ per
+variant and switch `iosSrc` with the variant, or accept the default look.
+
+## Hotspots
+
+`ProductHotspotData = { id, position: [x, y, z], normal: [x, y, z], label?, description? }`
+in model metres; serializable and validated (`parseHotspots`,
+`serializeHotspots`). `<ProductHotspot>` renders a slotted callout model-viewer
+keeps over the surface as the camera orbits, fading it when the surface
+faces away.
+
+`useProductHotspots(element, { initial, onChange })` is the authoring and
+runtime state: `placeFromPointer(clientX, clientY, label)` raycasts the
+click onto the model (`positionAndNormalFromPoint`), `add` / `update` /
+`remove` / `clear`, `json` / `importJson`. The debug-ui
+`<ProductHotspotPanel>` wraps it with click-to-place, label editing and a
+JSON copy, so authored data goes back into the app as `initial`.
+
+## Dimensions
+
+`useProductDimensions(element)` returns the bounding box (`getDimensions`,
+`getBoundingBoxCenter`) after load. `<ProductDimensions>` labels width,
+height and depth on three box edges, anchored as hotspots (`dimensionLines`
+computes the edges); `unit` is `metric` (cm under 1 m) or `imperial`.
+Pair with `scale="fixed"` on the viewer so AR places the model at that
+size with no pinch-to-resize.
+
+## Manual validation
+
+Run `/product-details` (model-viewer's variants shoe):
+
+| Case       | Steps                               | Expect                                                         |
+| ---------- | ----------------------------------- | -------------------------------------------------------------- |
+| Variants   | tap the colourway buttons           | the shoe changes material; the panel select follows            |
+| Hotspots   | "place hotspot", click the shoe     | a labelled dot stays on that point while orbiting              |
+| Authoring  | edit the label, "copy JSON", reload | label persists; JSON matches the panel                         |
+| Dimensions | toggle, switch unit                 | W / H / D labels on the box edges, cm ↔ in                    |
+| AR (phone) | "View in your space"                | true-size placement; variant carried into Scene Viewer / WebXR |