@vincentt-xr/sdk 2.0.0-alpha.1 → 2.0.0-alpha.2

package/docs/SDK_DOCUMENTATION.md

@@ -0,0 +1,1155 @@
+# XR SDK Documentation
+
+## Table of Contents
+
+- [Introduction](#introduction)
+- [Installation](#installation)
+- [Quick Start](#quick-start)
+- [XRViewport Quick Start](#xrviewport-quick-start)
+- [Architecture](#architecture)
+- [Core Concepts](#core-concepts)
+- [Components](#components)
+- [Tracking Models](#tracking-models)
+- [Complete Example](#complete-example)
+- [API Reference](#api-reference)
+
+## Introduction
+
+The XR SDK (`@vincentt-xr/sdk`) is a WebAR SDK that provides AI-powered tracking capabilities for React Three Fiber applications. It integrates MediaPipe models for face, hand, body, and gesture tracking with a declarative React component API.
+
+### Key Features
+
+- **AI-Powered Tracking**: Face, hand, body, and gesture recognition using MediaPipe
+- **React Three Fiber Integration**: Seamless integration with R3F ecosystem
+- **Unified Architecture**: Single scene rendering with proper event handling
+- **Background Segmentation**: Real-time person segmentation for custom backgrounds
+- **Screen-Space UI**: 2D UI positioning system
+- **TypeScript Support**: Full type safety and IntelliSense support
+
+### Current Example Routes
+
+Source of truth: `example/src/navigation/routes.tsx`. This list is a readable summary only.
+
+- `/unified-architecture`: debug-ui scene editor composition for face tracking
+- `/testing-hook`: multi-face face-mesh sanity example
+- `/body`: visible body-tracking anchors
+- `/debugger`: debugger overlay and metrics example
+- `/viewport-editor`: reusable 3D scene editor fixture
+- `/face-retouch-legacy`: legacy face-retouch shader settings example
+
+## Installation
+
+```bash
+npm install @vincentt-xr/sdk
+# or
+pnpm add @vincentt-xr/sdk
+# or
+yarn add @vincentt-xr/sdk
+```
+
+### Peer Dependencies
+
+The SDK requires the following peer dependencies:
+
+```json
+{
+  "react": "^18.0.0",
+  "react-dom": "^18.0.0",
+  "@react-three/fiber": "8.18.0",
+  "@react-three/drei": "9.105.6",
+  "three": "0.173.0"
+}
+```
+
+## Quick Start
+
+For new responsive builds, prefer `XRViewport`.
+
+`XRScene` is now legacy-first:
+
+- supported for compatibility
+- still used internally by `XRViewport mode="r3f"` / `mode="hybrid"`
+- not recommended as new app root
+- deprecation path: new examples and responsive features should land on `XRViewport`
+
+## XRViewport Quick Start
+
+### DOM Mode
+
+```tsx
+import { XRProvider, XRViewport } from "@vincentt-xr/sdk";
+
+export default function App() {
+  return (
+    <XRProvider>
+      <XRViewport mode="dom" source={{ kind: "webcam" }} fitMode="auto" />
+    </XRProvider>
+  );
+}
+```
+
+### R3F Mode
+
+```tsx
+import { PerspectiveCamera } from "@react-three/drei";
+import { XRProvider, XRViewport, VideoBackground } from "@vincentt-xr/sdk";
+
+export default function App() {
+  return (
+    <XRProvider>
+      <XRViewport mode="r3f" mediaStrategy="external" showMediaControlsByDefault={false}>
+        <XRViewport.Scene>
+          <PerspectiveCamera makeDefault position={[0, 0, 5]} fov={45} />
+          <VideoBackground />
+        </XRViewport.Scene>
+      </XRViewport>
+    </XRProvider>
+  );
+}
+```
+
+### Hybrid Mode
+
+```tsx
+import { PerspectiveCamera } from "@react-three/drei";
+import { XRProvider, XRViewport, VideoBackground, BoundingBoxOverlay } from "@vincentt-xr/sdk";
+
+export default function App() {
+  return (
+    <XRProvider>
+      <XRViewport mode="hybrid" mediaStrategy="external">
+        <XRViewport.Scene>
+          <PerspectiveCamera makeDefault position={[0, 0, 5]} fov={45} />
+          <VideoBackground />
+        </XRViewport.Scene>
+        <XRViewport.Overlay>
+          <BoundingBoxOverlay boundingBox={{ left: 0.2, top: 0.2, width: 0.3, height: 0.4 }} />
+        </XRViewport.Overlay>
+      </XRViewport>
+    </XRProvider>
+  );
+}
+```
+
+Here's a minimal example to get you started with face tracking:
+
+```tsx
+import { PerspectiveCamera } from "@react-three/drei";
+import { useEffect } from "react";
+import {
+  XRProvider,
+  XRScene,
+  useXRContext,
+  HeadTracker,
+  VideoBackground,
+  XRMediaSource,
+  XRModel,
+} from "@vincentt-xr/sdk";
+
+const App = () => {
+  const { session } = useXRContext();
+
+  useEffect(() => {
+    const initSession = async () => {
+      // 1. Set media source (webcam)
+      await session.setMediaSource({ source: XRMediaSource.WEBCAM });
+
+      // 2. Register tracking models
+      await session.registerXRPipeline([{ model: XRModel.FACE_TRACKER }]);
+
+      // 3. Start the session
+      await session.start();
+    };
+
+    initSession();
+  }, []);
+
+  return (
+    <XRScene>
+      <PerspectiveCamera makeDefault position={[0, 0, 5]} fov={45} />
+      <VideoBackground />
+
+      <ambientLight intensity={1} />
+
+      <HeadTracker>
+        <mesh>
+          <boxGeometry args={[1, 1, 1]} />
+          <meshStandardMaterial color="hotpink" />
+        </mesh>
+      </HeadTracker>
+    </XRScene>
+  );
+};
+
+export default function MyApp() {
+  return (
+    <XRProvider>
+      <App />
+    </XRProvider>
+  );
+}
+```
+
+## Architecture
+
+The XR SDK is built on a layered architecture:
+
+```
+┌─────────────────────────────────────────┐
+│         React Components Layer          │
+│  (XRProvider, XRScene, HeadTracker)    │
+├─────────────────────────────────────────┤
+│         Session Management Layer        │
+│    (useXRSession, XRPipeline)          │
+├─────────────────────────────────────────┤
+│         AI Models Layer                 │
+│  (MediaPipe: Face, Hand, Body, etc.)   │
+├─────────────────────────────────────────┤
+│         Video Processing Layer          │
+│  (MediaStream → Canvas → Texture)      │
+└─────────────────────────────────────────┘
+```
+
+### Rendering Architecture
+
+The SDK uses a **unified scene with layer-based compositing**:
+
+```
+Layer: Background (-1000 to -900)
+  └── Video feed, custom backgrounds, segmentation
+
+Layer: AR Content (0 to 900)
+  └── Face/hand/body tracked objects, 3D content
+
+Layer: UI (900+)
+  └── Screen-space 2D UI, text overlays
+```
+
+**Benefits:**
+
+- ✅ Standard R3F events work (onClick, onPointerOver, etc.)
+- ✅ Postprocessing works (EffectComposer, Bloom, etc.)
+- ✅ All Drei components work (Html, Text, etc.)
+- ✅ Single perspective camera (no portal isolation)
+
+## Core Concepts
+
+### XRProvider
+
+The `XRProvider` component wraps your application and provides the XR session context to all child components.
+
+```tsx
+import { XRProvider } from "@vincentt-xr/sdk";
+
+function App() {
+  return (
+    <XRProvider>
+      {/* Your XR content */}
+    <XRProvider>
+  );
+}
+```
+
+### XRSession
+
+The session object manages the lifecycle of your XR experience. Access it via `useXRContext()`:
+
+```tsx
+const { session } = useXRContext();
+
+// Session lifecycle
+await session.setMediaSource({ source: XRMediaSource.WEBCAM });
+await session.registerXRPipeline([{ model: XRModel.FACE_TRACKER }]);
+await session.start();
+
+// Check readiness
+if (session.ready) {
+  // Access model data
+  const faceNode = session.getModelNode<XRModelNodeFaceTracking>(XRModel.FACE_TRACKER);
+}
+```
+
+### XR Pipeline
+
+The pipeline is an array of models that will process the video stream:
+
+```tsx
+await session.registerXRPipeline([
+  { model: XRModel.FACE_TRACKER },
+  { model: XRModel.HAND_TRACKER },
+  { model: XRModel.BACKGROUND_SEGMENTATION },
+]);
+```
+
+### Raw MediaPipe Data Access
+
+The SDK exposes raw tracking data for low-level effect work, debugging, and
+AI-assisted development. Use `useXRModelNode()` to read the live model node,
+then pull raw data from the face node directly.
+
+```tsx
+import { XRModel, useXRModelNode, type XRModelNodeFaceTracking } from "@vincentt-xr/sdk";
+
+const faceNode = useXRModelNode<XRModelNodeFaceTracking>(XRModel.FACE_TRACKER);
+const rawFace = faceNode?.getRawFaceTrackingData(0);
+
+const landmarks = rawFace?.landmarks;
+const blendshapes = rawFace?.blendshapes;
+const matrix = rawFace?.transformationMatrix;
+```
+
+Use high-level tracker components for composition and the raw-data accessors
+when you need direct MediaPipe-level inspection.
+
+Segmentation nodes also expose low-level accessors:
+
+```tsx
+const segNode = useXRModelNode<XRModelNodeSegmentation>(XRModel.BACKGROUND_SEGMENTATION);
+const maskCanvas = segNode?.getMaskCanvas();
+const rawSegmentation = segNode?.getRawSegmentationData();
+```
+
+This gives custom workflows access to the rendered mask canvas and the raw
+category-mask bytes coming from the segmentation pipeline.
+
+## Components
+
+### XRScene
+
+The main canvas component that renders your XR experience.
+
+`XRScene` remains supported, but for new work prefer `XRViewport mode="r3f"` or
+`mode="hybrid"` so media, layout, and overlays share one root.
+
+Migration window:
+
+- direct `XRScene` entry stays supported while old demos and downstream apps migrate
+- new viewport-first examples are the default reference
+- removal should only happen after viewport parity is validated for webcam, image, and video flows
+
+```tsx
+import { XRScene } from "@vincentt-xr/sdk";
+
+<XRScene>{/* Your 3D content */}</XRScene>;
+```
+
+### XRViewport
+
+`XRViewport` is the responsive public entry point for new workflows. It owns:
+
+- viewport sizing
+- media source selection
+- cover/contain fitting
+- mirror state
+- DOM overlay composition
+- optional scene adapter mounting
+
+Shared viewport state:
+
+- `useXRViewportState()` returns the live viewport render state for descendants
+- scene content and overlay content can both consume the same media/layout/mirror truth
+- `XRViewport.Scene` and `XRViewport.Overlay` slot children may be render functions
+- in `mediaStrategy="external"` flows, viewport state can now bridge session face results into `faces` for scene/overlay consumers
+
+Modes:
+
+- `dom` — responsive preview + DOM/canvas overlays
+- `r3f` — responsive viewport + internal `XRScene` adapter
+- `hybrid` — scene adapter plus overlay layer under one root
+
+Media strategies:
+
+- `internal` — viewport owns webcam/image/video and tracking preview lifecycle
+- `external` — legacy session/media owner stays in control while viewport provides responsive scene/overlay composition
+
+Core props:
+
+- `source`
+  - `{ kind: "webcam" }`
+  - `{ kind: "image", assetId?: string }`
+  - `{ kind: "video", assetId?: string }`
+- `mode` — `dom`, `r3f`, or `hybrid`
+- `fitMode` — `auto`, `cover`, or `contain`
+- `initialMirrored` — initial mirror toggle state
+- `mediaStrategy` — `internal` or `external`
+- `imageOptions` / `videoOptions` — selectable assets for internal viewport-owned media
+
+Legacy compatibility props still exist:
+
+- `initialSource`
+- `initialImageId`
+- `initialVideoId`
+
+Prefer `source` for new code so one prop owns the initial media selection contract.
+
+### VideoBackground
+
+Renders the camera feed with optional segmentation masking and custom backgrounds.
+
+```tsx
+import { VideoBackground } from "@vincentt-xr/sdk";
+
+<VideoBackground
+  segmentationMask={mask} // Optional: Three.js Texture from segmentation
+  customBackground="#6366f1" // Optional: Color/image shows through transparent areas
+  renderOrder={-999} // Optional: Default is -999
+/>;
+```
+
+**Features:**
+
+- Automatically calculates screen-filling dimensions
+- Supports background segmentation for person cutout effects
+- Custom background color/texture shown through transparent areas
+- Proper depth ordering with `renderOrder`
+- Flat background semantics: the material keeps depth testing and depth writing
+  disabled. Use the debug-ui `XRSceneViewport` editor plane when the camera feed
+  should behave as 3D geometry.
+
+### XRLayer
+
+`XRLayer` is a lightweight DOM composition primitive for stacking display
+surfaces, debug overlays, and screen-space canvases without introducing a
+temporary `v3`-named API surface.
+
+```tsx
+<XRLayer
+  config={{
+    name: "overlay-target",
+    clearColor: "transparent",
+    pointerEvents: "none",
+    zIndex: 2,
+  }}
+>
+  <div>Overlay content</div>
+</XRLayer>
+```
+
+### XRMediaSurface and XRTrackingOverlay
+
+These components provide production-named canvas-based media rendering and
+tracking-preview rendering for workflows that still need DOM/canvas layering
+outside the main R3F scene.
+
+```tsx
+<XRMediaSurface
+  canvasRef={displayCanvasRef}
+  imageRef={imageRef}
+  videoRef={videoRef}
+  layer={liveLayer}
+  layout={layout}
+  mirrored
+  sourceKind="video"
+/>
+
+<XRTrackingOverlay
+  canvasRef={trackingCanvasRef}
+  imageRef={imageRef}
+  videoRef={videoRef}
+  layer={trackingLayer}
+  layout={layout}
+  mirrored
+  sourceKind="video"
+  debugVisible={showTrackingCanvas}
+  viewportSize={viewportSize}
+/>
+```
+
+### HeadTracker
+
+A component that automatically follows the detected face position and rotation.
+
+```tsx
+import { HeadTracker } from "@vincentt-xr/sdk";
+
+<HeadTracker faceIndex={0}>
+  {" "}
+  {/* Optional: which face to track (default: 0) */}
+  <mesh>
+    <boxGeometry args={[1, 1, 1]} />
+    <meshStandardMaterial color="red" />
+  </mesh>
+</HeadTracker>;
+```
+
+**How it works:**
+
+- Uses `useFrame` to continuously update position/rotation from face tracking data
+- Automatically converts MediaPipe coordinates to Three.js world space
+- Supports all standard R3F events (onClick, onPointerOver, etc.)
+
+### MeshRenderer render state
+
+`MeshRenderer` supports primitive meshes and asset-backed GLTF/GLB meshes through
+serialized `MeshRendererSettings`.
+
+Advanced draw state lives under `material.renderState`:
+
+```ts
+material: {
+  type: "standard",
+  color: "#ffffff",
+  opacity: 1,
+  wireframe: false,
+  roughness: 0.42,
+  metalness: 0.08,
+  renderState: {
+    depthTest: true,
+    depthWrite: true,
+    depthFunc: "lessEqual",
+    colorWrite: true,
+    colorMask: { r: true, g: true, b: true, a: true },
+    cullMode: "back",
+    blendMode: "normal",
+    polygonOffset: { enabled: false, factor: 0, units: 0 },
+  },
+}
+```
+
+`colorMask` controls per-channel RGBA writes. It is applied as scoped WebGL draw
+state and restored after the mesh renders. `instanceCount` is not part of
+material render state; it belongs to a future instanced-mesh API.
+
+### PinToMesh
+
+`PinToMesh` pins a wrapper object to a target mesh surface resolved from a
+scene-object ID or direct `Object3D` target.
+
+```ts
+{
+  enabled: false,
+  targetObjectId: null,
+  orientation: "none",
+  uv: { x: 0.5, y: 0.5 },
+  vertexNormal: false,
+  offsetSpace: "local",
+  offsetPosition: { x: 0, y: 0, z: 0 },
+  offsetRotation: { x: 0, y: 0, z: 0 },
+}
+```
+
+When disabled, or when the target is unavailable, the runtime wrapper resets to
+local position `0,0,0` and identity rotation. Authored object placement stays in
+the child `Transform3D`.
+
+`PinToMeshSettingsPanel` in `@vincentt-xr/sdk/debug-ui` uses Transform-style text
+number inputs for UV and offsets: visible zero defaults, decimal validation, and
+0.1 arrow-step increments.
+
+### Reusable XR scene editor debug UI
+
+Temporary editor surfaces should use `@vincentt-xr/sdk/debug-ui` instead of
+rewriting hierarchy, inspector, viewport, and toolbar wiring per app.
+
+```tsx
+import {
+  SceneHierarchyPanel,
+  SceneInspectorPanel,
+  ViewportToolbar,
+  XRSceneEditorProvider,
+  XRSceneViewport,
+} from "@vincentt-xr/sdk/debug-ui";
+
+<XRSceneEditorProvider
+  store={store}
+  selectedObjectId={selectedObjectId}
+  onSelectObject={setSelectedObjectId}
+  onStoreChange={setStore}
+  onComponentChange={replaceComponent}
+  onObjectChange={replaceObject}
+>
+  <XRSceneViewport />
+  <ViewportToolbar />
+  <SceneHierarchyPanel />
+  <SceneInspectorPanel />
+</XRSceneEditorProvider>;
+```
+
+The editor components adapt to `SceneObjectStore`; they do not own persistent
+scene data. Selection, orbit controls, grid visibility, and editor camera state
+are transient editor state. `XRSceneViewport` owns a separate camera-texture
+plane with `depthTest` and `depthWrite` so the camera feed behaves as real
+editor-scene geometry without changing core `VideoBackground` behavior.
+
+### ScreenSpaceUI
+
+Provides 2D UI positioning with helper components for common positions.
+
+```tsx
+import { ScreenSpaceUI } from "@vincentt-xr/sdk";
+import { Text } from "@react-three/drei";
+
+<ScreenSpaceUI>
+  <ScreenSpaceUI.TopLeft>
+    <Text fontSize={0.15} color="white">
+      Top Left UI
+    </Text>
+  </ScreenSpaceUI.TopLeft>
+
+  <ScreenSpaceUI.BottomCenter>
+    <mesh onClick={() => console.log("clicked")}>
+      <planeGeometry args={[1, 0.4]} />
+      <meshBasicMaterial color="blue" depthTest={false} />
+    </mesh>
+  </ScreenSpaceUI.BottomCenter>
+
+  <ScreenSpaceUI.TopRight>
+    <Text fontSize={0.1} color="green">
+      Top Right Status
+    </Text>
+  </ScreenSpaceUI.TopRight>
+</ScreenSpaceUI>;
+```
+
+**Available positions:**
+
+- `TopLeft`, `TopCenter`, `TopRight`
+- `CenterLeft`, `Center`, `CenterRight`
+- `BottomLeft`, `BottomCenter`, `BottomRight`
+
+**Features:**
+
+- Calculates screen dimensions based on camera FOV
+- Uses `depthTest={false}` for always-on-top rendering
+- Supports all R3F events
+- High `renderOrder` (999+) for proper layering
+
+### FaceMesh
+
+Renders a 3D mesh that follows the face geometry.
+
+```tsx
+import { FaceMesh } from "@vincentt-xr/sdk";
+
+<FaceMesh faceIndex={0} wireframe={true} color="blue" />;
+```
+
+### HeadOccluder
+
+Renders an invisible mesh that occludes objects behind the head.
+
+```tsx
+import { HeadOccluder } from "@vincentt-xr/sdk";
+
+<HeadOccluder faceIndex={0} />;
+```
+
+## Tracking Models
+
+### Available Models
+
+The SDK provides five tracking models through the `XRModel` enum:
+
+```tsx
+enum XRModel {
+  FACE_TRACKER = "vincentt.xr.face-tracker",
+  HAND_TRACKER = "vincentt.xr.hand-tracker",
+  GESTURE_TRACKER = "vincentt.xr.gesture-tracker",
+  BODY_TRACKER = "vincentt.xr.body-tracker",
+  BACKGROUND_SEGMENTATION = "vincentt.xr.background-segmentation",
+}
+```
+
+### Face Tracking
+
+Provides 478 facial landmarks, face blendshapes, and rotation matrices.
+
+```tsx
+import { XRModel, XRModelNodeFaceTracking } from "@vincentt-xr/sdk";
+
+// Register model
+await session.registerXRPipeline([{ model: XRModel.FACE_TRACKER }]);
+
+// Access face data
+const faceNode = session.getModelNode<XRModelNodeFaceTracking>(XRModel.FACE_TRACKER);
+
+const face = faceNode?.getFace(0); // Get first detected face
+if (face) {
+  console.log("Position:", face.position); // { x, y, z }
+  console.log("Rotation:", face.rotation); // { x, y, z }
+  console.log("Scale:", face.scale); // number
+  console.log("Blendshapes:", face.blendshapes); // Map of facial expressions
+}
+```
+
+**Face Object Structure:**
+
+```typescript
+type Face = {
+  position: { x: number; y: number; z: number };
+  rotation: { x: number; y: number; z: number };
+  scale: number;
+  blendshapes?: Map<string, number>;
+  transformationMatrices?: Matrix[];
+};
+```
+
+### Hand Tracking
+
+Provides 21 hand landmarks per hand.
+
+```tsx
+import { XRModel, XRModelNodeHandTracking } from "@vincentt-xr/sdk";
+
+// Register model
+await session.registerXRPipeline([{ model: XRModel.HAND_TRACKER }]);
+
+// Access hand data
+const handNode = session.getModelNode<XRModelNodeHandTracking>(XRModel.HAND_TRACKER);
+
+if (handNode?.coordinates) {
+  const { left, right } = handNode.coordinates;
+
+  if (left) {
+    console.log("Left hand landmarks:", left); // Array of 21 coordinates
+  }
+
+  if (right) {
+    console.log("Right hand landmarks:", right); // Array of 21 coordinates
+  }
+}
+
+// Gesture detection (simple open/closed)
+console.log("Gesture:", handNode?.gesture);
+```
+
+### Gesture Tracking
+
+Hand landmarks with advanced gesture recognition.
+
+```tsx
+import { XRModel, XRModelNodeGestureTracking } from "@vincentt-xr/sdk";
+
+// Register model
+await session.registerXRPipeline([{ model: XRModel.GESTURE_TRACKER }]);
+
+// Access gesture data
+const gestureNode = session.getModelNode<XRModelNodeGestureTracking>(XRModel.GESTURE_TRACKER);
+
+if (gestureNode?.gesture) {
+  const { left, right } = gestureNode.gesture;
+
+  // Recognized gestures:
+  // - ClosedFist
+  // - OpenPalm
+  // - PointingUp
+  // - ThumbDown
+  // - ThumbUp
+  // - Victory
+  // - ILoveYou
+
+  console.log("Left hand gesture:", left);
+  console.log("Right hand gesture:", right);
+}
+```
+
+### Body Tracking
+
+Provides 33 body landmarks using BlazePose.
+
+```tsx
+import { XRModel, XRModelNodeBodyTracking } from "@vincentt-xr/sdk";
+
+// Register model
+await session.registerXRPipeline([{ model: XRModel.BODY_TRACKER }]);
+
+// Access body data
+const bodyNode = session.getModelNode<XRModelNodeBodyTracking>(XRModel.BODY_TRACKER);
+
+const landmarks = bodyNode?.getLandmarks2D(0); // Get first detected person
+if (landmarks) {
+  console.log("Body landmarks:", landmarks); // Array of 33 2D coordinates
+}
+```
+
+### Background Segmentation
+
+Provides real-time person segmentation mask with configurable quality settings.
+
+```tsx
+import { useState, useEffect } from "react";
+import { Texture } from "three";
+import { XRModel, XRModelNodeSegmentation } from "@vincentt-xr/sdk";
+
+const [segmentationMask, setSegmentationMask] = useState<Texture | undefined>();
+
+// Register model with quality option
+await session.registerXRPipeline([
+  {
+    model: XRModel.BACKGROUND_SEGMENTATION,
+    options: { quality: "high" }, // 'low' | 'medium' | 'high'
+  },
+]);
+
+// Get segmentation mask
+useEffect(() => {
+  if (session.ready) {
+    const segmentationNode = session.getModelNode<XRModelNodeSegmentation>(
+      XRModel.BACKGROUND_SEGMENTATION,
+    );
+    setSegmentationMask(segmentationNode?.backgroundSegmentationMask);
+  }
+}, [session.ready]);
+
+// Use with VideoBackground
+<VideoBackground
+  segmentationMask={segmentationMask}
+  customBackground="#6366f1" // Background color shows through transparent areas
+/>;
+```
+
+**Quality Options:**
+
+- `quality: "low"` (default) - 0.5x resolution, fastest performance, lower quality edges
+- `quality: "medium"` - 0.75x resolution, balanced performance and quality
+- `quality: "high"` - 1.0x resolution, best quality edges, higher GPU usage
+
+**Performance Impact:**
+
+- Low: Processes 4x fewer pixels, recommended for mobile devices
+- Medium: Good balance for most desktop applications
+- High: Best quality for high-end devices or when quality is critical
+
+**Example with different quality levels:**
+
+```tsx
+// Mobile/low-end devices
+await session.registerXRPipeline([
+  { model: XRModel.BACKGROUND_SEGMENTATION, options: { quality: "low" } },
+]);
+
+// Desktop/balanced
+await session.registerXRPipeline([
+  { model: XRModel.BACKGROUND_SEGMENTATION, options: { quality: "medium" } },
+]);
+
+// High-end/quality-critical
+await session.registerXRPipeline([
+  { model: XRModel.BACKGROUND_SEGMENTATION, options: { quality: "high" } },
+]);
+```
+
+## Complete Example
+
+Here's a complete example demonstrating face tracking, segmentation, and screen-space UI:
+
+```tsx
+import { PerspectiveCamera } from "@react-three/drei";
+import { useEffect, useState } from "react";
+import { Texture } from "three";
+import {
+  HeadTracker,
+  ScreenSpaceUI,
+  useXRContext,
+  VideoBackground,
+  XRMediaSource,
+  XRModel,
+  XRModelNodeSegmentation,
+  XRProvider,
+  XRScene,
+} from "@vincentt-xr/sdk";
+
+const Content = () => {
+  const { session } = useXRContext();
+  const [segmentationMask, setSegmentationMask] = useState<Texture | undefined>();
+  const [clickCount, setClickCount] = useState(0);
+
+  // Initialize XR session
+  useEffect(() => {
+    const initSession = async () => {
+      await session.setMediaSource({ source: XRMediaSource.WEBCAM });
+      await session.registerXRPipeline([
+        { model: XRModel.FACE_TRACKER },
+        { model: XRModel.BACKGROUND_SEGMENTATION, options: { quality: "high" } },
+      ]);
+      await session.start();
+    };
+
+    initSession();
+  }, []);
+
+  // Get segmentation mask when ready
+  useEffect(() => {
+    if (session.ready) {
+      const segmentationNode = session.getModelNode<XRModelNodeSegmentation>(
+        XRModel.BACKGROUND_SEGMENTATION,
+      );
+      setSegmentationMask(segmentationNode?.backgroundSegmentationMask);
+    }
+  }, [session.ready]);
+
+  return (
+    <XRScene>
+      {/* Camera */}
+      <PerspectiveCamera makeDefault position={[0, 0, 5]} fov={45} />
+
+      {/* Video background with segmentation */}
+      <VideoBackground segmentationMask={segmentationMask} customBackground="#6366f1" />
+
+      {/* Lighting */}
+      <ambientLight intensity={1} />
+      <directionalLight position={[5, 5, 5]} intensity={1} />
+
+      {/* Face-tracked 3D content with events */}
+      <HeadTracker>
+        <mesh onClick={() => setClickCount((prev) => prev + 1)} position={[0, 0, -5]}>
+          <boxGeometry args={[10, 10, 1]} />
+          <meshStandardMaterial color="hotpink" />
+        </mesh>
+      </HeadTracker>
+
+      {/* Screen-space UI */}
+      <ScreenSpaceUI>
+        <ScreenSpaceUI.TopLeft>
+          <Text
+            fontSize={0.15}
+            color="white"
+            anchorX="left"
+            anchorY="top"
+            material-depthTest={false}
+          >
+            {`Clicks: ${clickCount}`}
+          </Text>
+        </ScreenSpaceUI.TopLeft>
+      </ScreenSpaceUI>
+    </XRScene>
+  );
+};
+
+export default function App() {
+  return (
+    <XRProvider>
+      <Content />
+    </XRProvider>
+  );
+}
+```
+
+## API Reference
+
+### Components
+
+#### `<XRProvider>`
+
+Provides XR session context to child components.
+
+**Props:** None
+
+**Usage:**
+
+```tsx
+<XRProvider>
+  <App />
+</XRProvider>
+```
+
+#### `<XRScene>`
+
+Main canvas component for rendering XR content.
+
+**Props:** Accepts all standard R3F Canvas props
+
+**Usage:**
+
+```tsx
+<XRScene>{/* Your 3D content */}</XRScene>
+```
+
+#### `<VideoBackground>`
+
+Renders the camera feed with optional effects.
+
+**Props:**
+
+- `segmentationMask?: Texture` - Segmentation mask texture
+- `customBackground?: string | Texture` - Background shown through transparent areas
+- `renderOrder?: number` - Default: -999
+
+#### `<HeadTracker>`
+
+Container that follows face tracking data.
+
+**Props:**
+
+- `faceIndex?: number` - Which face to track (default: 0)
+- `children: ReactNode` - 3D objects to track
+
+#### `<ScreenSpaceUI>`
+
+Container for screen-space UI elements.
+
+**Props:** None
+
+**Helper Components:**
+
+- `ScreenSpaceUI.TopLeft`
+- `ScreenSpaceUI.TopCenter`
+- `ScreenSpaceUI.TopRight`
+- `ScreenSpaceUI.CenterLeft`
+- `ScreenSpaceUI.Center`
+- `ScreenSpaceUI.CenterRight`
+- `ScreenSpaceUI.BottomLeft`
+- `ScreenSpaceUI.BottomCenter`
+- `ScreenSpaceUI.BottomRight`
+
+#### `<FaceMesh>`
+
+Renders a 3D mesh following face geometry.
+
+**Props:**
+
+- `faceIndex?: number` - Which face to render (default: 0)
+- `wireframe?: boolean` - Render as wireframe
+- `color?: string` - Mesh color
+
+#### `<HeadOccluder>`
+
+Invisible mesh for occluding objects behind the head.
+
+**Props:**
+
+- `faceIndex?: number` - Which face to occlude (default: 0)
+
+### Hooks
+
+#### `useXRContext()`
+
+Access the XR session context.
+
+**Returns:**
+
+```typescript
+{
+  session: XRSession;
+}
+```
+
+**XRSession Methods:**
+
+- `setMediaSource(options: { source: XRMediaSource })` - Set video source
+- `registerXRPipeline(models: Array<{ model: XRModel }>)` - Register tracking models
+- `start()` - Start the XR session
+- `getModelNode<T>(model: XRModel): T | undefined` - Get model data
+
+**XRSession Properties:**
+
+- `ready: boolean` - Whether session is ready
+- `error: object | null` - Error state if any
+
+#### `useScreenSpace()`
+
+Access screen space dimensions for custom UI positioning.
+
+**Returns:**
+
+```typescript
+{
+  screenWidth: number,
+  screenHeight: number
+}
+```
+
+### Enums
+
+#### `XRMediaSource`
+
+```typescript
+enum XRMediaSource {
+  WEBCAM = "webcam",
+  STREAM = "stream",
+}
+```
+
+#### `XRModel`
+
+```typescript
+enum XRModel {
+  FACE_TRACKER = "vincentt.xr.face-tracker",
+  HAND_TRACKER = "vincentt.xr.hand-tracker",
+  GESTURE_TRACKER = "vincentt.xr.gesture-tracker",
+  BODY_TRACKER = "vincentt.xr.body-tracker",
+  BACKGROUND_SEGMENTATION = "vincentt.xr.background-segmentation",
+}
+```
+
+### Types
+
+#### `XRModelNodeFaceTracking`
+
+```typescript
+type XRModelNodeFaceTracking = {
+  detectionResults?: FaceLandmarkerResult;
+  getFace: (faceIndex?: number) => Face | undefined;
+};
+
+type Face = {
+  position: { x: number; y: number; z: number };
+  rotation: { x: number; y: number; z: number };
+  scale: number;
+  blendshapes?: Map<string, number>;
+  transformationMatrices?: Matrix[];
+};
+```
+
+#### `XRModelNodeHandTracking`
+
+```typescript
+type XRModelNodeHandTracking = {
+  coordinates?: {
+    left?: Coordinate[];
+    right?: Coordinate[];
+  };
+  gesture?: {
+    left?: string;
+    right?: string;
+  };
+};
+```
+
+#### `XRModelNodeGestureTracking`
+
+```typescript
+type XRModelNodeGestureTracking = {
+  coordinates?: {
+    left?: Coordinate[];
+    right?: Coordinate[];
+  };
+  gesture?: {
+    left?: GestureType;
+    right?: GestureType;
+  };
+};
+
+type GestureType =
+  | "ClosedFist"
+  | "OpenPalm"
+  | "PointingUp"
+  | "ThumbDown"
+  | "ThumbUp"
+  | "Victory"
+  | "ILoveYou";
+```
+
+#### `XRModelNodeBodyTracking`
+
+```typescript
+type XRModelNodeBodyTracking = {
+  getLandmarks2D: (bodyIndex?: number) => Coordinate[] | undefined;
+};
+```
+
+#### `XRModelNodeSegmentation`
+
+```typescript
+type XRModelNodeSegmentation = {
+  backgroundSegmentationMask?: Texture;
+};
+```
+
+---
+
+For more examples, check the `example/src/pages/` directory in the repository.
+
+**Additional Resources:**
+
+- [Migration Guide](./MIGRATION_GUIDE.md) - Migrating from older versions