@vincentt-xr/sdk 2.0.0-alpha.0 → 2.0.0-alpha.2

package/docs/MIGRATION_GUIDE.md

@@ -0,0 +1,589 @@
+# Migration Guide
+
+This guide helps you migrate from the old portal-based architecture to the new unified architecture introduced in January 2025.
+
+## Table of Contents
+
+- [What Changed](#what-changed)
+- [Benefits of the New Architecture](#benefits-of-the-new-architecture)
+- [Migration Steps](#migration-steps)
+- [Component Changes](#component-changes)
+- [Breaking Changes](#breaking-changes)
+- [Side-by-Side Comparison](#side-by-side-comparison)
+
+## What Changed
+
+The SDK has been completely refactored from a **portal-based dual-camera system** to a **unified scene architecture**:
+
+### Removed
+
+- ❌ Portal-based scene isolation
+- ❌ Manual event handling with `onPointerEvent`
+- ❌ Manual `gl.render()` calls
+- ❌ Custom `OrthographicCamera` component
+- ❌ `SegmentationPlane` component
+
+### Added
+
+- ✅ `VideoBackground` component for camera feed rendering
+- ✅ `ScreenSpaceUI` component for 2D UI positioning
+- ✅ Standard R3F rendering pipeline
+- ✅ Native event system support
+- ✅ Postprocessing support
+
+### Deprecated (but still available)
+
+- ⚠️ Custom `PerspectiveCamera` (use drei's `PerspectiveCamera` instead)
+- ⚠️ Custom `OrthographicCamera` (use `VideoBackground` instead)
+
+## Benefits of the New Architecture
+
+### 1. Events Work Correctly
+
+**Old:** Manual event handling was required
+
+```tsx
+// Old - Manual event handling
+<mesh onPointerEvent={(event) => {
+  if (event.type === 'click') {
+    handleClick();
+  }
+}}>
+```
+
+**New:** Standard R3F events work
+
+```tsx
+// New - Standard events
+<mesh
+  onClick={handleClick}
+  onPointerOver={handleHover}
+  onPointerOut={handleUnhover}
+>
+```
+
+### 2. Postprocessing Works
+
+**Old:** Postprocessing didn't work due to portal isolation
+
+**New:** EffectComposer and all effects work
+
+```tsx
+import { EffectComposer, Bloom, Glitch } from "@react-three/postprocessing";
+
+<XRScene>
+  {/* Your content */}
+  <EffectComposer>
+    <Bloom intensity={0.5} />
+    <Glitch />
+  </EffectComposer>
+</XRScene>;
+```
+
+### 3. All Drei Components Work
+
+**Old:** Components like `Html`, `Text`, etc. had issues
+
+**New:** All Drei components work correctly
+
+```tsx
+import { Html, Text } from "@react-three/drei";
+
+<HeadTracker>
+  <Html>
+    <div>HTML UI attached to face</div>
+  </Html>
+  <Text fontSize={0.5}>3D Text</Text>
+</HeadTracker>;
+```
+
+### 4. Simpler Architecture
+
+**Old:** Two cameras, portal system, manual rendering
+
+**New:** Single camera, standard R3F rendering
+
+## Migration Steps
+
+## XRViewport Migration Direction
+
+For new responsive apps, use `XRViewport` as app root and treat direct
+`XRScene` usage as legacy-first.
+
+### Old API -> New API Mapping
+
+- `XRScene` root -> `XRViewport mode="r3f"` or `mode="hybrid"`
+- `XRScene` + separate DOM overlay -> `XRViewport mode="hybrid"`
+- calibration/debug DOM preview -> `XRViewport mode="dom"`
+- session-owned legacy media flow -> `XRViewport mediaStrategy="external"`
+- viewport-owned new media flow -> `XRViewport mediaStrategy="internal"`
+- `initialSource` / `initialImageId` / `initialVideoId` -> `source={{ kind, assetId }}`
+
+### Example
+
+```tsx
+<XRProvider>
+  <XRViewport mode="r3f" mediaStrategy="external">
+    <XRViewport.Scene>
+      <PerspectiveCamera makeDefault position={[0, 0, 5]} fov={45} />
+      <VideoBackground />
+      <FaceTracker>{/* AR content */}</FaceTracker>
+    </XRViewport.Scene>
+  </XRViewport>
+</XRProvider>
+```
+
+### Migration Gotchas
+
+- `XRViewport mode="dom"` with `mediaStrategy="internal"` owns preview media itself. Do not also start a parallel session media source for same page.
+- `XRViewport mode="r3f"` and `mode="hybrid"` can use `mediaStrategy="external"` while legacy pages still bootstrap session media separately.
+- New responsive tracking flows should prefer direct frame/canvas input through viewport-managed internals instead of `captureStream()` bridges.
+- During migration, keep one owner for media truth per page. Mixed ownership is where resize and mirror bugs appear.
+- `mediaStrategy="external"` viewport state can still expose `faces` to overlay/scene slot consumers, but selected source kind should be passed through `source={{ kind: ... }}` for correct source-aware transforms.
+
+### Removal Criteria For Direct XRScene Entry
+
+- old demos that matter must run under `XRViewport`
+- calibration workflow must stay correct for webcam, image, and video
+- resize must stay correct without refresh in viewport-owned flows
+- docs/examples must point to `XRViewport` first
+
+### Step 1: Update Camera Setup
+
+**Before:**
+
+```tsx
+<XRScene>
+  <OrthographicCamera>
+    <SegmentationPlane alphaMask={mask} />
+  </OrthographicCamera>
+
+  <PerspectiveCamera>{/* AR content */}</PerspectiveCamera>
+</XRScene>
+```
+
+**After:**
+
+```tsx
+import { PerspectiveCamera } from "@react-three/drei";
+
+<XRScene>
+  <PerspectiveCamera makeDefault position={[0, 0, 5]} fov={45} />
+
+  <VideoBackground segmentationMask={mask} />
+
+  {/* AR content */}
+</XRScene>;
+```
+
+### Step 2: Replace SegmentationPlane with VideoBackground
+
+**Before:**
+
+```tsx
+<OrthographicCamera>
+  <SegmentationPlane alphaMask={segmentationMask} backgroundColor="#6366f1" />
+</OrthographicCamera>
+```
+
+**After:**
+
+```tsx
+<VideoBackground
+  segmentationMask={segmentationMask}
+  customBackground="#6366f1"
+  renderOrder={-999}
+/>
+```
+
+### Step 3: Update 2D UI to Use ScreenSpaceUI
+
+**Before:**
+
+```tsx
+// Manual positioning for 2D UI
+<OrthographicCamera>
+  <Text position={[-5, 3, 0]} fontSize={0.15}>
+    Top Left Text
+  </Text>
+</OrthographicCamera>
+```
+
+**After:**
+
+```tsx
+<ScreenSpaceUI>
+  <ScreenSpaceUI.TopLeft>
+    <Text fontSize={0.15} color="white" anchorX="left" anchorY="top">
+      Top Left Text
+    </Text>
+  </ScreenSpaceUI.TopLeft>
+</ScreenSpaceUI>
+```
+
+### Step 4: Update Event Handlers
+
+**Before:**
+
+```tsx
+<mesh
+  onPointerEvent={(event) => {
+    if (event.type === 'click') {
+      setClickCount(prev => prev + 1);
+    }
+    if (event.type === 'pointerover') {
+      setHovered(true);
+    }
+  }}
+>
+```
+
+**After:**
+
+```tsx
+<mesh
+  onClick={() => setClickCount(prev => prev + 1)}
+  onPointerOver={() => setHovered(true)}
+  onPointerOut={() => setHovered(false)}
+>
+```
+
+### Step 5: Remove Manual Render Calls
+
+**Before:**
+
+```tsx
+// Manual render calls in useFrame
+useFrame(({ gl, scene, camera }) => {
+  gl.render(scene, camera);
+});
+```
+
+**After:**
+
+```tsx
+// No manual rendering needed - R3F handles it
+useFrame(() => {
+  // Your frame logic
+});
+```
+
+## Component Changes
+
+### VideoBackground
+
+Replaces `SegmentationPlane` and renders the camera feed.
+
+**Props:**
+
+- `segmentationMask?: Texture` - Person segmentation mask
+- `customBackground?: string | Texture` - Background color/texture
+- `renderOrder?: number` - Render order (default: -999)
+
+**Example:**
+
+```tsx
+<VideoBackground segmentationMask={segmentationMask} customBackground="#6366f1" />
+```
+
+### ScreenSpaceUI
+
+New component for 2D UI positioning with helper components.
+
+**Helper Components:**
+
+- `TopLeft`, `TopCenter`, `TopRight`
+- `CenterLeft`, `Center`, `CenterRight`
+- `BottomLeft`, `BottomCenter`, `BottomRight`
+
+**Example:**
+
+```tsx
+<ScreenSpaceUI>
+  <ScreenSpaceUI.TopLeft>
+    <Text>UI Element</Text>
+  </ScreenSpaceUI.TopLeft>
+</ScreenSpaceUI>
+```
+
+### HeadTracker
+
+No changes - works the same but now supports standard events.
+
+**Example:**
+
+```tsx
+<HeadTracker faceIndex={0}>
+  <mesh onClick={handleClick}>
+    <boxGeometry args={[1, 1, 1]} />
+    <meshStandardMaterial color="red" />
+  </mesh>
+</HeadTracker>
+```
+
+## Breaking Changes
+
+### 1. OrthographicCamera Removed
+
+**Impact:** High - Required for old architecture
+
+**Migration:** Replace with `VideoBackground` for camera feed rendering
+
+### 2. SegmentationPlane Removed
+
+**Impact:** High - Used for background effects
+
+**Migration:** Use `VideoBackground` with `segmentationMask` prop
+
+### 3. Custom PerspectiveCamera Deprecated
+
+**Impact:** Medium - Still works but deprecated
+
+**Migration:** Use drei's `PerspectiveCamera` with `makeDefault`
+
+### 4. Manual Event Handling
+
+**Impact:** Medium - Custom event system removed
+
+**Migration:** Use standard R3F events (onClick, onPointerOver, etc.)
+
+### 5. Portal System
+
+**Impact:** Low - Internal implementation detail
+
+**Migration:** No action needed - automatically handled
+
+## Side-by-Side Comparison
+
+### Complete Old Example
+
+```tsx
+import { useEffect, useState } from "react";
+import {
+  XRProvider,
+  XRScene,
+  useXRContext,
+  OrthographicCamera,
+  PerspectiveCamera,
+  SegmentationPlane,
+  HeadTracker,
+  XRMediaSource,
+  XRModel,
+} from "@vincentt-xr/sdk";
+
+const Content = () => {
+  const { session } = useXRContext();
+  const [segmentationMask, setSegmentationMask] = useState();
+
+  useEffect(() => {
+    const initSession = async () => {
+      await session.setMediaSource({ source: XRMediaSource.WEBCAM });
+      await session.registerXRPipeline([
+        { model: XRModel.FACE_TRACKER },
+        { model: XRModel.BACKGROUND_SEGMENTATION },
+      ]);
+      await session.start();
+    };
+    initSession();
+  }, []);
+
+  useEffect(() => {
+    if (session.ready) {
+      const segNode = session.getModelNode(XRModel.BACKGROUND_SEGMENTATION);
+      setSegmentationMask(segNode?.backgroundSegmentationMask);
+    }
+  }, [session.ready]);
+
+  return (
+    <XRScene>
+      {/* Old: Two cameras with portal isolation */}
+      <OrthographicCamera>
+        <SegmentationPlane alphaMask={segmentationMask} backgroundColor="#6366f1" />
+      </OrthographicCamera>
+
+      <PerspectiveCamera>
+        <ambientLight intensity={1} />
+
+        {/* Old: Custom event handling */}
+        <HeadTracker>
+          <mesh
+            onPointerEvent={(event) => {
+              if (event.type === "click") {
+                console.log("Clicked!");
+              }
+            }}
+          >
+            <boxGeometry args={[1, 1, 1]} />
+            <meshStandardMaterial color="red" />
+          </mesh>
+        </HeadTracker>
+      </PerspectiveCamera>
+    </XRScene>
+  );
+};
+
+export default function App() {
+  return (
+    <XRProvider>
+      <Content />
+    </XRProvider>
+  );
+}
+```
+
+### Complete New Example
+
+```tsx
+import { PerspectiveCamera } from "@react-three/drei";
+import { useEffect, useState } from "react";
+import { Texture } from "three";
+import {
+  XRProvider,
+  XRScene,
+  useXRContext,
+  VideoBackground,
+  HeadTracker,
+  ScreenSpaceUI,
+  XRMediaSource,
+  XRModel,
+  XRModelNodeSegmentation,
+} from "@vincentt-xr/sdk";
+
+const Content = () => {
+  const { session } = useXRContext();
+  const [segmentationMask, setSegmentationMask] = useState<Texture | undefined>();
+
+  useEffect(() => {
+    const initSession = async () => {
+      await session.setMediaSource({ source: XRMediaSource.WEBCAM });
+      await session.registerXRPipeline([
+        { model: XRModel.FACE_TRACKER },
+        { model: XRModel.BACKGROUND_SEGMENTATION },
+      ]);
+      await session.start();
+    };
+    initSession();
+  }, []);
+
+  useEffect(() => {
+    if (session.ready) {
+      const segNode = session.getModelNode<XRModelNodeSegmentation>(
+        XRModel.BACKGROUND_SEGMENTATION,
+      );
+      setSegmentationMask(segNode?.backgroundSegmentationMask);
+    }
+  }, [session.ready]);
+
+  return (
+    <XRScene>
+      {/* New: Single camera from drei */}
+      <PerspectiveCamera makeDefault position={[0, 0, 5]} fov={45} />
+
+      {/* New: VideoBackground component */}
+      <VideoBackground segmentationMask={segmentationMask} customBackground="#6366f1" />
+
+      <ambientLight intensity={1} />
+
+      {/* New: Standard R3F events work */}
+      <HeadTracker>
+        <mesh onClick={() => console.log("Clicked!")}>
+          <boxGeometry args={[1, 1, 1]} />
+          <meshStandardMaterial color="red" />
+        </mesh>
+      </HeadTracker>
+
+      {/* New: ScreenSpaceUI for 2D UI */}
+      <ScreenSpaceUI>
+        <ScreenSpaceUI.TopLeft>
+          <Text fontSize={0.15} color="white">
+            Click the box!
+          </Text>
+        </ScreenSpaceUI.TopLeft>
+      </ScreenSpaceUI>
+    </XRScene>
+  );
+};
+
+export default function App() {
+  return (
+    <XRProvider>
+      <Content />
+    </XRProvider>
+  );
+}
+```
+
+## Common Migration Issues
+
+### Issue 1: Events Not Working
+
+**Problem:** Click events not firing after migration
+
+**Solution:** Make sure you're using drei's `PerspectiveCamera` with `makeDefault`:
+
+```tsx
+import { PerspectiveCamera } from "@react-three/drei";
+
+<PerspectiveCamera makeDefault position={[0, 0, 5]} fov={45} />;
+```
+
+### Issue 2: UI Elements Not Visible
+
+**Problem:** Screen-space UI not appearing
+
+**Solution:** Use `material-depthTest={false}` and proper `renderOrder`:
+
+```tsx
+<ScreenSpaceUI>
+  <ScreenSpaceUI.TopLeft>
+    <Text material-depthTest={false} renderOrder={999}>
+      UI Text
+    </Text>
+  </ScreenSpaceUI.TopLeft>
+</ScreenSpaceUI>
+```
+
+### Issue 3: Video Background Not Showing
+
+**Problem:** Camera feed not rendering
+
+**Solution:** Ensure `VideoBackground` is placed before other content and has correct `renderOrder`:
+
+```tsx
+<VideoBackground renderOrder={-999} />
+```
+
+### Issue 4: Postprocessing Not Working
+
+**Problem:** Effects not applying to scene
+
+**Solution:** Make sure `EffectComposer` is inside `XRScene` and using drei's camera:
+
+```tsx
+import { EffectComposer, Bloom } from "@react-three/postprocessing";
+import { PerspectiveCamera } from "@react-three/drei";
+
+<XRScene>
+  <PerspectiveCamera makeDefault position={[0, 0, 5]} fov={45} />
+  {/* Your content */}
+  <EffectComposer>
+    <Bloom intensity={0.5} />
+  </EffectComposer>
+</XRScene>;
+```
+
+## Need Help?
+
+If you encounter issues during migration:
+
+1. Check the [Getting Started Guide](./GETTING_STARTED.md) for complete API reference
+2. Review the example in `example/src/pages/UnifiedArchitecture`
+3. Ensure you're using the correct peer dependency versions:
+   - `@react-three/fiber`: `8.18.0`
+   - `@react-three/drei`: `9.105.6`
+   - `three`: `0.173.0`
+
+---
+
+**Migration Date:** January 2025
+**SDK Version:** 2.0.0+