@vincentt-xr/sdk 2.0.0-alpha.0 → 2.0.0-alpha.2

package/docs/api/audio.mdx

@@ -0,0 +1,36 @@
+---
+title: "Audio"
+description: "Silent-switch-safe background music, pooled sound effects, preload, and mute controls."
+sidebarTitle: "Audio"
+icon: "volume-2"
+---
+
+Use the core audio API instead of hand-rolling Web Audio or `new Audio()`.
+It starts muted, uses HTML audio for iOS silent-switch behavior, and unlocks on
+the user's unmute tap.
+
+```tsx
+import {
+  playOneShot,
+  prefetchMusic,
+  registerSound,
+  startMusic,
+  toggleMuted,
+  useMuted,
+} from "@vincentt-xr/sdk";
+
+await Promise.all([prefetchMusic(musicUrl), registerSound(hitUrl)]);
+startMusic(musicUrl);
+playOneShot(hitUrl);
+
+function SoundToggle() {
+  const muted = useMuted();
+  return <button onClick={toggleMuted}>{muted ? "Muted" : "Sound on"}</button>;
+}
+```
+
+Music is one looping element. Sound effects use bounded voice pools and
+per-URL coalescing, so repeated effects in one frame do not multiply playback.
+Preload both during the loading gate. Prefer AAC/M4A assets. Use
+`new AudioEngine()` only when an app genuinely needs an independent second
+mixer; free functions share the default engine.

package/docs/api/behavior-components.mdx

@@ -0,0 +1,164 @@
+---
+title: "3D Behavior Components"
+description: "LookAt and PinToMesh settings, target resolution, defaults, offsets, and current limitations."
+sidebarTitle: "Behavior Components"
+icon: "move-3d"
+---
+
+Behavior components are serialized scene-object components that affect another
+runtime object. They are currently focused on editor/API prototyping.
+
+```ts
+import { LookAt, PinToMesh } from "@vincentt-xr/sdk";
+import { createDefaultComponent } from "@vincentt-xr/sdk/scene-object";
+```
+
+## Target resolution
+
+`LookAt` and `PinToMesh` can receive a direct Three.js `Object3D` target, but the
+scene-object runtime path uses serialized target IDs:
+
+```ts
+lookAt.settings.targetObjectId = "target-object-id";
+pinToMesh.settings.targetObjectId = "base-plane";
+```
+
+`SceneObjectRenderer` registers rendered 3D scene objects by ID. Behavior
+components resolve `targetObjectId` against that runtime registry. Keep target
+IDs stable in serialized scene data.
+
+## `LookAt`
+
+```ts
+type LookAtSettings = {
+  enabled: boolean;
+  targetObjectId: string | null;
+  mode: "point" | "direction";
+  inheritChildren: boolean;
+  offsetRotation: { x: number; y: number; z: number };
+};
+```
+
+Default:
+
+```ts
+{
+  enabled: false,
+  targetObjectId: null,
+  mode: "point",
+  inheritChildren: false,
+  offsetRotation: { x: 0, y: 0, z: 0 },
+}
+```
+
+Usage:
+
+```tsx
+<LookAt target={targetObject} settings={lookAtSettings}>
+  <MeshRenderer value={meshRendererSettings} />
+</LookAt>
+```
+
+Current scope:
+
+- `mode: "point"` aims the child at the target object position.
+- `mode: "direction"` is available as serialized API surface, but axis
+  conventions and world-up controls are still minimal.
+- `offsetRotation` is in degrees.
+
+## `PinToMesh`
+
+```ts
+type PinToMeshSettings = {
+  enabled: boolean;
+  targetObjectId: string | null;
+  orientation: "none" | "position" | "positionAndDirection";
+  uv: { x: number; y: number };
+  vertexNormal: boolean;
+  offsetSpace: "local" | "world";
+  offsetPosition: { x: number; y: number; z: number };
+  offsetRotation: { x: number; y: number; z: number };
+};
+```
+
+Default:
+
+```ts
+{
+  enabled: false,
+  targetObjectId: null,
+  orientation: "none",
+  uv: { x: 0.5, y: 0.5 },
+  vertexNormal: false,
+  offsetSpace: "local",
+  offsetPosition: { x: 0, y: 0, z: 0 },
+  offsetRotation: { x: 0, y: 0, z: 0 },
+}
+```
+
+Usage:
+
+```tsx
+<PinToMesh target={targetObject} settings={pinToMeshSettings}>
+  <Transform3D value={transformSettings}>
+    <MeshRenderer value={meshRendererSettings} />
+  </Transform3D>
+</PinToMesh>
+```
+
+## Pin behavior
+
+- Pinning is disabled by default.
+- If disabled, the runtime pin wrapper resets to local position `0,0,0` and
+  identity rotation.
+- If the target is missing, the runtime pin wrapper also resets.
+- The child `Transform3D` remains the authored transform source of truth.
+- `orientation: "none"` and `orientation: "position"` use an identity pin-wrapper
+  rotation.
+- `orientation: "positionAndDirection"` aligns the wrapper to the target surface
+  normal before applying offset rotation.
+- `offsetRotation` is reapplied from a clean base every frame and does not
+  compound.
+
+## Offset fields
+
+`offsetPosition` and `offsetRotation` default to zero.
+
+```ts
+pinToMesh.settings.offsetSpace = "local";
+pinToMesh.settings.offsetPosition = { x: 0, y: 0.2, z: 0 };
+pinToMesh.settings.offsetRotation = { x: 0, y: 45, z: 0 };
+```
+
+`offsetSpace: "local"` applies position offset after the pin-wrapper orientation.
+`offsetSpace: "world"` applies position offset in scene axes. Rotation offsets
+are degrees.
+
+## UV and vertex normal
+
+`uv` selects the target mesh surface location. `vertexNormal` changes rotation
+pinning to use interpolated vertex normals when orientation needs direction.
+
+Current limitation: UV behavior is not final. Missing UV data falls back to the
+target bounds center instead of surfacing a visible invalid-target state.
+
+## Debug UI
+
+Use `PinToMeshSettingsPanel` from `@vincentt-xr/sdk/debug-ui`.
+
+```tsx
+<PinToMeshSettingsPanel
+  value={pinToMeshSettings}
+  targets={sceneObjects.map((object) => ({ id: object.id, name: object.name }))}
+  onChange={(next) => onComponentChange({ ...pinToMeshComponent, settings: next })}
+/>
+```
+
+Input behavior follows the Transform editor:
+
+- visible zero defaults
+- text inputs, not browser number spinners
+- decimal validation
+- arrow step `0.1`
+- Shift arrow step `1`
+- Alt arrow step `0.01`

package/docs/api/capture-recording.mdx

@@ -0,0 +1,103 @@
+---
+title: "Capture and Recording"
+description: "Take XR photos and record, pause, resume, stop, preview, save, or upload canvas video with observable lifecycle state."
+sidebarTitle: "Capture & Recording"
+icon: "camera"
+---
+
+Use `useFrameCapture` for UI and effect logic. It wraps the current XR session
+capture with observable status, result, error, and lifecycle events.
+
+Search terms: photo capture, take photo, snapshot, screenshot, frame capture,
+camera capture, video recording, canvas recorder, pause recording, resume
+recording, recording indicator, capture status, recording status, captured
+image, recorded video, and video Blob.
+
+## Photo capture: `useFrameCapture`
+
+```tsx
+import { dataURLtoFile, useFrameCapture } from "@vincentt-xr/sdk";
+
+const photo = useFrameCapture({
+  onCaptureStart: () => console.log("capturing"),
+  onCaptureComplete: (dataUrl) => upload(dataURLtoFile(dataUrl, "photo.png")),
+  onError: reportError,
+});
+
+await photo.capturePhoto();
+```
+
+`CaptureStatus` is `idle | capturing | completed | error`. Call `clear()` to
+return to `idle`. `capturePhoto()` resolves to the captured data URL, or `null`
+after exposing a failure through `status`, `error`, and `onError`. For
+synchronous low-level code, `session.captureFrame()` remains available through
+`useXRContext` from `@vincentt-xr/sdk/low-level`.
+
+| Returned field   | Meaning                                                      |
+| ---------------- | ------------------------------------------------------------ |
+| `status`         | `idle`, `capturing`, `completed`, or `error`                 |
+| `capturePhoto()` | Capture now; resolves to a PNG data URL or `null` on failure |
+| `clear()`        | Reset status, data URL, and error                            |
+| `dataUrl`        | Latest captured image for preview, download, or conversion   |
+| `error`          | Latest normalized capture error                              |
+
+Events are `onCaptureStart`, `onCaptureComplete(dataUrl)`, and
+`onError(error)`. Use `dataURLtoFile(dataUrl, filename)` when upload or sharing
+requires a `File`.
+
+## Video recording: `useMediaRecorder`
+
+Use `useMediaRecorder` to record a canvas `MediaStream`. It selects the first
+supported MP4 or WebM codec. Pass `audio: true` for microphone audio or provide
+an existing audio stream.
+
+```tsx
+import { useMediaRecorder } from "@vincentt-xr/sdk";
+
+const recorder = useMediaRecorder({
+  audio: true,
+  onRecordingStart: () => console.log("recording"),
+  onRecordingPause: () => console.log("paused"),
+  onRecordingResume: () => console.log("recording"),
+  onRecordingStop: () => console.log("stopping"),
+  onRecordingComplete: uploadVideo,
+  onError: reportError,
+});
+await recorder.start(canvas.captureStream(30));
+recorder.pause();
+recorder.resume();
+recorder.stop();
+```
+
+`RecordingStatus` is
+`idle | starting | recording | paused | stopping | completed | error`.
+`isRecording` and `isPaused` remain as convenience booleans. `recordedBlob`
+updates at `completed`; call `clear()` to release tracks and return to `idle`.
+
+| Returned field or control   | Meaning                                                  |
+| --------------------------- | -------------------------------------------------------- |
+| `status`                    | Single source of truth for the recording lifecycle       |
+| `start(stream, timeslice?)` | Start recording a canvas or supplied media stream        |
+| `pause()` / `resume()`      | Pause or continue the active recorder                    |
+| `stop()`                    | Enter `stopping`; completion publishes `recordedBlob`    |
+| `clear()`                   | Stop owned tracks and reset blob, status, and error      |
+| `recordedBlob`              | Completed MP4/WebM data for preview, download, or upload |
+| `error`                     | Latest normalized recorder error                         |
+
+Options are `audio`, `mimeType`, `bitsPerSecond`, `onData`,
+`passThroughStream`, `onRecordingStart`, `onRecordingPause`,
+`onRecordingResume`, `onRecordingStop`, `onRecordingComplete`, and `onError`.
+The old `onStop(blob)` callback remains deprecated; use
+`onRecordingComplete(blob)`.
+
+For an already composed stream, pass `passThroughStream` when creating the hook
+and call `start()` with no stream argument.
+
+Mount capture inside the active XR provider/session. Do not screenshot the DOM;
+the session path uses the current grouped renderer and render order.
+
+Use the verified `/capture-recording` example to exercise photo, start, pause,
+resume, stop, completion, error, and clear states against the webcam canvas. It
+shows a prominent recording-state HUD, elapsed recording time, captured-photo
+preview, and recorded-video playback. These controls and states were verified
+with a real webcam on 26 July 2026.

package/docs/api/debug-ui-scene-editor.mdx

@@ -0,0 +1,249 @@
+---
+title: "Debug UI Scene Editor API"
+description: "Reusable debug-ui primitives for temporary 3D scene authoring: provider, viewport, hierarchy, inspector, toolbar, and settings panels."
+sidebarTitle: "Debug UI Scene Editor"
+icon: "panel-right"
+---
+
+Import debug/authoring UI from `@vincentt-xr/sdk/debug-ui`.
+
+```tsx
+import {
+  FaceRetouchSettingsPanel,
+  MeshRendererSettingsPanel,
+  PinToMeshSettingsPanel,
+  SceneHierarchyPanel,
+  SceneInspectorPanel,
+  Transform3DSettingsSection,
+  ViewportToolbar,
+  XRSceneEditorProvider,
+  XRSceneViewport,
+} from "@vincentt-xr/sdk/debug-ui";
+import {
+  replaceSceneComponent,
+  replaceSceneObject,
+  type SceneComponent,
+  type SceneObject,
+} from "@vincentt-xr/sdk/scene-object";
+```
+
+Debug UI is optional. Use it for temporary client-app scene setup and remove it
+from final preview/runtime surfaces.
+
+## In-repo examples
+
+- `/viewport-editor`: general 3D scene-object editor.
+- `/unified-architecture`: unified face-tracking scene using
+  `XRSceneEditorProvider`, `XRSceneViewport`, hierarchy, inspector, toolbar, and
+  media switching.
+- `/testing-hook`: multi-face face mesh setup backed by the same reusable
+  editor primitives.
+- `/debugger`: debugger overlay mounted beside a minimal tracked scene.
+
+## `XRSceneEditorProvider`
+
+Shares editor context across viewport, hierarchy, inspector, and toolbar.
+
+```tsx
+const handleComponentChange = (component: SceneComponent) => {
+  setStore((current) => replaceSceneComponent(current, component));
+};
+
+const handleObjectChange = (object: SceneObject) => {
+  setStore((current) => replaceSceneObject(current, object));
+};
+
+<XRSceneEditorProvider
+  store={store}
+  selectedObjectId={selectedObjectId}
+  onSelectObject={setSelectedObjectId}
+  onStoreChange={setStore}
+  onComponentChange={handleComponentChange}
+  onObjectChange={handleObjectChange}
+>
+  {children}
+</XRSceneEditorProvider>;
+```
+
+Persistent data stays in `SceneObjectStore`. Selection, orbit controls, grid
+visibility, video visibility, and editor camera state are transient editor state.
+
+## `XRSceneViewport`
+
+Optional viewport renderer.
+
+```tsx
+<XRSceneViewport
+  camera={{ position: { x: 0, y: 0.2, z: 5.5 }, fov: 42 }}
+  grid={{ size: 12, divisions: 12, position: { x: 0, y: -2.5, z: 0 } }}
+/>
+```
+
+The viewport renders:
+
+- `XRScene`
+- editor-owned camera-texture plane with `depthTest` and `depthWrite`
+- optional editor grid
+- default perspective camera
+- ambient light
+- `SceneObjectRenderer` entries from `createRenderPlan`
+- `SequentialRenderPasses`
+- optional orbit controls
+
+The editor-owned video plane is separate from core `VideoBackground`.
+
+## `SceneHierarchyPanel`
+
+Store-driven wrapper around `HierarchyPanel`.
+
+```tsx
+<SceneHierarchyPanel />
+```
+
+It reads groups and objects from the editor provider, routes selection through
+`onSelectObject`, and routes visibility changes back to the store. Dragging an
+object onto another object parents it through
+`setSceneObjectParentPreserveWorldTransform`; dragging a nested object back onto
+the render group clears its parent. Drag parenting preserves the child's current
+visual/world transform by recalculating its local Transform for the new parent.
+Pass `rootOnly` to show combined parent objects without exposing their modular
+children, as used by the face-binding rose example.
+
+## `SceneInspectorPanel`
+
+Store-driven wrapper around `ObjectInspectorPanel`.
+
+```tsx
+<SceneInspectorPanel initialPosition={{ x: 980, y: 20 }} />
+```
+
+It renders sections for components present on the selected object:
+
+- `Transform`
+- `MeshRenderer`
+- `Light`
+- `LookAt`
+- `PinToMesh`
+- `ScreenShape`
+- `TrackingAnchor`
+
+All edits route through `onComponentChange` or `onObjectChange`.
+
+## `ViewportToolbar`
+
+Small optional toolbar for transient viewport controls.
+
+```tsx
+<ViewportToolbar position="top-center" />
+```
+
+Initial control:
+
+- Orbit Controls on/off
+
+This state is not serialized into `SceneObjectStore`.
+
+## `MeshRendererSettingsPanel`
+
+Edits `MeshRendererSettings`.
+
+```tsx
+const handleMeshRendererChange = (next: MeshRendererSettings) => {
+  onComponentChange({ ...meshRendererComponent, settings: next });
+};
+
+<MeshRendererSettingsPanel value={meshRendererSettings} onChange={handleMeshRendererChange} />;
+```
+
+Includes material render-state controls:
+
+- Depth Test
+- Depth Write
+- Depth Function
+- Color Write
+- RGBA Color Mask
+- Cull Mode
+- Blend Mode
+- Polygon Offset
+
+## `PinToMeshSettingsPanel`
+
+Edits `PinToMeshSettings`.
+
+```tsx
+const handlePinToMeshChange = (next: PinToMeshSettings) => {
+  onComponentChange({ ...pinToMeshComponent, settings: next });
+};
+
+<PinToMeshSettingsPanel
+  value={pinToMeshSettings}
+  targets={[
+    { id: "mic", name: "Mic" },
+    { id: "emoji", name: "Thinking Emoji" },
+  ]}
+  onChange={handlePinToMeshChange}
+/>;
+```
+
+Input behavior matches the Transform editor:
+
+- visible zero defaults
+- text inputs, not browser number spinners
+- decimal validation
+- arrow step `0.1`
+- Shift arrow step `1`
+- Alt arrow step `0.01`
+
+## `FaceRetouchSettingsPanel`
+
+Thin wrapper around the generic `SettingsPanel` for the legacy
+`FaceRetouchControls` schema.
+
+```tsx
+import { FaceRetouchSettingsPanel } from "@vincentt-xr/sdk/debug-ui";
+import {
+  patchFaceRetouchControls,
+  resolveFaceRetouchControls,
+  type FaceRetouchControls,
+} from "@vincentt-xr/sdk/face-effects";
+
+const [controls, setControls] = useState(() => resolveFaceRetouchControls());
+
+const updateControls = (next: Partial<FaceRetouchControls>) => {
+  setControls((current) => patchFaceRetouchControls(current, next));
+};
+
+<FaceRetouchSettingsPanel value={controls} onChange={updateControls} />;
+```
+
+It owns no retouch defaults itself; labels, ranges, and normalization come from
+`@vincentt-xr/sdk/face-effects`.
+
+## Full editor composition
+
+```tsx
+const handleComponentChange = (component: SceneComponent) => {
+  setStore((current) => replaceSceneComponent(current, component));
+};
+
+const handleObjectChange = (object: SceneObject) => {
+  setStore((current) => replaceSceneObject(current, object));
+};
+
+<XRSceneEditorProvider
+  store={store}
+  selectedObjectId={selectedObjectId}
+  onSelectObject={setSelectedObjectId}
+  onStoreChange={setStore}
+  onComponentChange={handleComponentChange}
+  onObjectChange={handleObjectChange}
+>
+  <XRSceneViewport />
+  <ViewportToolbar />
+  <SceneHierarchyPanel />
+  <SceneInspectorPanel />
+</XRSceneEditorProvider>;
+```
+
+Each piece is optional. Client apps can provide their own layout and mount only
+the viewport, only the panels, or only specific settings panels.

package/docs/api/face-actions.mdx

@@ -0,0 +1,104 @@
+---
+title: "Face Actions"
+description: "Semantic blink, brow, mouth, and calibrated head actions derived from FaceInfo."
+sidebarTitle: "Face Actions"
+icon: "face-smile"
+---
+
+Face Actions turn `FaceInfo` pose and blendshapes into stable, typed names. Use
+them instead of guessing MediaPipe strings, pose signs, or thresholds.
+
+```tsx
+import { useFrameCapture } from "@vincentt-xr/sdk";
+import { useFaceAction, useFaceInfo } from "@vincentt-xr/sdk/tracking";
+
+function BlinkToCapture() {
+  const face = useFaceInfo();
+  const blink = useFaceAction(face, "eyeBlink");
+  const photo = useFrameCapture();
+
+  useEffect(() => {
+    if (blink.justStarted) void photo.capturePhoto();
+  }, [blink.justStarted, photo.capturePhoto]);
+
+  return null;
+}
+```
+
+Use `getFaceActionValue(face, action)` for continuous animation,
+`hasFaceAction(face, action)` for a simple threshold check, and
+`useFaceAction(face, action)` when start/end edges, hold timing, and hysteresis
+matter.
+
+```ts
+const openness = getFaceActionValue(face, "mouthOpen");
+const isPouting = hasFaceAction(face, "mouthPout");
+```
+
+## Actions
+
+| Name                            | Meaning                      |
+| ------------------------------- | ---------------------------- |
+| `browRaise`                     | Either brow is raised        |
+| `eyeBlink`                      | Both eyes blink              |
+| `eyeBlinkEither`                | Either eye blinks            |
+| `eyeBlinkLeft`, `eyeBlinkRight` | Viewer-left/right eye blinks |
+| `mouthOpen`                     | Jaw opens                    |
+| `mouthPout`                     | Lips pucker                  |
+| `headUp`, `headDown`            | Pitch from neutral           |
+| `headTurnLeft`, `headTurnRight` | Yaw from neutral             |
+| `headTiltLeft`, `headTiltRight` | Roll from neutral            |
+
+Blendshape values are in `[0, 1]`. Head values are non-negative radians. Up and
+left follow the SDK convention and are positive before they are converted to an
+action magnitude.
+
+## Neutral head pose
+
+Capture the resting rotation and pass it to head actions when the camera or
+user's neutral posture is not centered:
+
+```tsx
+const [neutralRotation, setNeutralRotation] = useState<FaceVector3>();
+const turnLeft = useFaceAction(face, "headTurnLeft", { neutralRotation });
+
+// Run from a "Set neutral pose" button while a face is visible.
+setNeutralRotation(face ? { ...face.pose.rotation } : undefined);
+```
+
+Initial SDK thresholds are `0.45–0.55` for expressions and 10° (`π / 18`) for
+head actions. Head release is 6° (`π / 30`). Expression release is 0.15 below
+its activation threshold. Expression hold is 50 ms and head hold is 80 ms.
+Override `threshold`, `releaseThreshold`, or `holdMs` when an effect needs
+different sensitivity.
+
+`useFaceAction` returns `phase: "begin" | "stay" | "end" | "none"` in
+addition to `active`, `justStarted`, `justEnded`, and `value`. For a multi-face
+effect, select the strongest matching face without another tracking reader:
+
+```tsx
+const faces = useFaceInfos({ numFaces: 5 });
+const { face, value } = getStrongestFaceAction(faces, "mouthOpen");
+const mouth = useFaceAction(face, "mouthOpen");
+```
+
+## Movement Mapping
+
+Use `eyeBlinkLeft`, `eyeBlinkRight`, `eyeBlink`, and `eyeBlinkEither` for the
+left, right, both, and either-eye movement cases. Use `mouthOpen`, `mouthPout`,
+and `browRaise` for the matching physical movements. These actions report
+observable pose/blendshape movement; they do not infer emotion.
+
+Left/right blink names use the mirrored viewer-facing convention used by effect
+editors. Raw `FaceInfo.blendshapes` retain MediaPipe's anatomical-side names.
+The lifecycle guarantees one `stay` frame between `begin` and `end`, including
+for a blink that drops below its release threshold on the next input frame.
+
+For image-based happy, sad, angry, fear, disgust, surprise, neutral, and unknown
+classification, use the separate `useFaceAttributeExpression` hook. The SDK
+does not fabricate these labels from movement thresholds. The verified
+`/facial-movement-expression` page exposes both independent surfaces.
+
+Use `/face-actions` to view every value, calibrate neutral pose, adjust the
+selected action, and repeat the live verification checklist. All actions and
+the default thresholds were verified with the real-camera page on 26 July 2026.