@vincentt-xr/sdk 2.0.0-alpha.0

package/GROUNDING.md

@@ -0,0 +1,537 @@
+# Vincentt XR SDK — Grounding Reference
+
+The complete, authoritative reference for the `@vincentt-xr/sdk` package:
+every component, hook, and prop you may use, plus the R3F conventions for
+composing them. Use the components and props documented here; do not invent props
+or spelunk `node_modules` for the API.
+
+## Import doors — one per task
+
+The API is split into task-domain entry points. Import each symbol from the door
+this reference shows for it; the imports here are literal.
+
+- `@vincentt-xr/sdk` — **core**: providers, scene, screen-space layout, `TextLabel`, `Panel`, `SpriteAnimation`, `VideoBackground`, `AspectRatioContainer`, capture/share (`useMediaRecorder`, `dataURLtoFile`), audio.
+- `@vincentt-xr/sdk/tracking` — **trackers**: face/hand/gesture/body/segmentation, `TrackingAnchor`, `GestureTrigger`, the bare `FaceMesh`, `useFaceResults`.
+- `@vincentt-xr/sdk/face-effects` — **face deep**: FaceMesh material config, retouch, canonical mesh, head-binding.
+- `@vincentt-xr/sdk/low-level` — **escape hatch**: raw model-node reads, selector hooks, custom-tracker plumbing. Reach here only when core + tracking can't express it.
+
+Trackers (`<FaceTracker>`, `<GestureTracker>`, `<HandTracker>`, `<BodyTracker>`)
+**self-register** when mounted — no explicit `registerXRPipeline` call. Per-frame
+logic is R3F `useFrame`; per-mount setup is `useEffect`.
+
+> This file ships inside the SDK tarball. In a v2-template project it is combined
+> with the template's own grounding (template-local helpers + project notes +
+> patterns) into the project's `GROUNDING.md`. Edit the SDK API here, in the repo
+> that owns the API.
+
+---
+
+## Screen-space layout (the primary tool)
+
+Most Vincentt projects use screen-space UI: decorative frames, prompts, badges, controls — overlaid on the live camera feed. This is more common than world-space landmark attachment.
+
+### `<ScreenSpaceUI>` — Container for screen-space content
+
+Wraps all screen-anchored children. Must be the parent of any `<ScreenTransform>`.
+
+```tsx
+import { ScreenSpaceUI, ScreenTransform } from "@vincentt-xr/sdk";
+
+<ScreenSpaceUI>
+  <ScreenTransform anchors={{ left: -1, right: 1, top: 1, bottom: -1 }}>
+    {/* full-screen content */}
+  </ScreenTransform>
+</ScreenSpaceUI>;
+```
+
+### `<ScreenTransform anchors={{ left, right, top, bottom }}>` — Place an element within a normalized screen rect
+
+Coordinates are normalized: -1 to +1, with +y up. The center of the screen is `(0, 0)`. The anchors rect controls BOTH size and placement of the child — it stretches the child (plane, TextLabel, sprite) to fill the rect. Don't add `scale`/`position`/`planeGeometry args` to size or move a screen-anchored child; set the anchors.
+
+```tsx
+// Full-screen frame
+<ScreenTransform anchors={{ left: -1, right: 1, top: 1, bottom: -1 }}>
+  <mesh name="frame"><planeGeometry args={[1, 1]} /><meshBasicMaterial map={frameTex} transparent /></mesh>
+</ScreenTransform>
+
+// Bottom-center prompt
+<ScreenTransform anchors={{ left: -0.4, right: 0.4, top: -0.5, bottom: -0.9 }}>
+  <TextLabel name="prompt" text="Show your hand" fontSize={220} />
+</ScreenTransform>
+
+// Full-width strip pinned FLUSH to the top edge (e.g. a colored banner plane)
+<ScreenTransform anchors={{ left: -1, right: 1, top: 1, bottom: 0.6 }}>
+  <mesh name="topBar"><planeGeometry args={[1, 1]} /><meshBasicMaterial color="#2255ff" /></mesh>
+</ScreenTransform>
+```
+
+**Converting a screen-region annotation (% coords) to anchors.** Annotation marks come as top-left percentages (x: 0%=left→100%=right, y: 0%=top→100%=bottom). Anchors are normalized -1..+1 with +y UP, so the y sign flips:
+`left = x%/50 - 1`, `right = x%/50 - 1`, `top = 1 - y%/50`, `bottom = 1 - y%/50`. A mark at x 0–100%, y 0–28% → `anchors={{ left: -1, right: 1, top: 1, bottom: 0.44 }}`. **Snap a near-edge mark to that edge** (within ~10%): top→`top: 1`, bottom→`bottom: -1`, left→`left: -1`, right→`right: 1`. A strip drawn near the bottom (y 78–90%) → `anchors={{ left: -1, right: 1, top: -0.6, bottom: -1 }}` (flush at -1, not the literal -0.8).
+
+### `useScreenRectAspect()` + `computeContainScale(rectAspect, contentAspect)` — Aspect-correct fitting
+
+Use together when you need a texture or mesh to fit inside its anchor box without stretching.
+
+```tsx
+import { useScreenRectAspect, computeContainScale } from "@vincentt-xr/sdk";
+
+const rectAspect = useScreenRectAspect();
+const [sx, sy] = computeContainScale(rectAspect, 1200 / 800); // content w/h ratio
+
+<mesh name="badge" scale={[sx * 0.5, sy * 0.5, 1]}>
+  <planeGeometry args={[1, 1]} />
+  <meshBasicMaterial map={tex} transparent />
+</mesh>;
+```
+
+### `<SpriteAnimation sheet={{ url, cols, rows, frames, fps }}>` — Animated transparent overlay
+
+The primitive for an animated transparent overlay (a frame border, a gesture-instruction graphic, an animated sticker). One transparent WebP atlas of `cols`×`rows` cells; the first `frames` play left-to-right, top-to-bottom at `fps`. It is a single static GPU texture with a per-frame UV offset — no video decode — so it works in every native and in-app browser and costs ~zero heat.
+
+**Use this, not transparent video.** No single video codec plays everywhere with alpha (HEVC-alpha is iOS-Safari-only; VP8/VP9-alpha WebM renders blank as a WebGL texture on iOS Safari), and two video-texture decodes on top of face tracking overheat the phone. Export the atlas as a real 8-bit-alpha WebP (with `cwebp`; ffmpeg's `libwebp` drops alpha). Pair opaque H.264 video only for full-frame backgrounds.
+
+Defaults to full-screen; pass `anchors` to place it and `contentAspect` (frame width/height) to contain-fit without stretching. Must live inside a `<ScreenSpaceUI>`.
+
+```tsx
+import { SpriteAnimation, ScreenSpaceUI } from "@vincentt-xr/sdk";
+
+<ScreenSpaceUI>
+  {/* full-screen animated frame border */}
+  <SpriteAnimation sheet={{ url: frameUrl, cols: 8, rows: 6, frames: 48, fps: 8 }} name="frame" />
+
+  {/* a placed, contain-fit instruction clip */}
+  <SpriteAnimation
+    sheet={{ url: startUrl, cols: 6, rows: 4, frames: 24, fps: 12 }}
+    anchors={{ left: -0.6, right: 0.6, top: 0.4, bottom: -0.4 }}
+    contentAspect={1}
+    name="startHint"
+  />
+</ScreenSpaceUI>;
+```
+
+---
+
+## Trackers (self-registering)
+
+Mount the tracker component; it registers the underlying ML model automatically. No `registerXRPipeline` call needed.
+
+### `<FaceTracker>` — Face tracking (468 landmarks)
+
+Detects faces and provides face transform to children via context. Supports `faceIndex` (default 0) and `scaleWithFace`.
+
+```tsx
+import { FaceTracker, TrackingAnchor } from "@vincentt-xr/sdk/tracking";
+
+<FaceTracker>
+  <TrackingAnchor target="face.forehead">
+    <mesh name="crown" position={[0, 0.1, 0]}>
+      <planeGeometry args={[0.3, 0.15]} />
+      <meshBasicMaterial map={crownTex} transparent />
+    </mesh>
+  </TrackingAnchor>
+</FaceTracker>;
+```
+
+### `<FaceTracker.Mesh />` / `<FaceMesh>` — Deforming 468-point face mesh
+
+For masks, makeup, and full-face decoration that deforms with skin.
+
+```tsx
+import { FaceTracker, FaceMesh } from "@vincentt-xr/sdk/tracking";
+
+<FaceTracker>
+  <FaceMesh appearance={{ map: maskTex }} />
+</FaceTracker>;
+```
+
+### `<HandTracker>` — Hand skeleton (21 joints)
+
+The hand-skeleton tracker. Use it for 21-joint hand landmarks — anchoring to `hand.indexTip`, `hand.wrist`, etc. It also exposes a crude open-vs-closed grip on `node.gesture` (only `closed_fist` / `open_palm`), but that is a side effect of the skeleton, not a gesture recognizer. **For any named gesture, use `<GestureTracker>`** (next section) — it is the gesture door; `<HandTracker>` is the skeleton door.
+
+```tsx
+import { HandTracker } from "@vincentt-xr/sdk/tracking";
+
+<HandTracker />;
+```
+
+### `<GestureTracker>` — Named gestures
+
+The gesture door — the component that recognizes named gestures. Its `node.gesture` is one of these exact strings (copy verbatim — these are the only valid values):
+
+`closed_fist`, `open_palm`, `pointing_up`, `thumb_down`, `thumb_up`, `victory`, `i_love_you`, `none`.
+
+Catches people out:
+
+- It is `thumb_up` / `thumb_down` — singular, no trailing "s" (not `thumbs_up`).
+- The peace sign is `victory`.
+- There is no `open_hand`; the open-hand gesture is `open_palm`.
+
+```tsx
+import { GestureTracker } from "@vincentt-xr/sdk/tracking";
+
+<GestureTracker />;
+```
+
+### `<BodyTracker>` — Pose tracking (33 landmarks)
+
+Head, shoulders, elbows, wrists, hips, knees, ankles.
+
+```tsx
+import { BodyTracker, TrackingAnchor } from "@vincentt-xr/sdk/tracking";
+
+<BodyTracker>
+  <TrackingAnchor target="body.leftShoulder">
+    <mesh name="badge-left">
+      <sphereGeometry args={[0.05]} />
+      <meshBasicMaterial color="red" />
+    </mesh>
+  </TrackingAnchor>
+</BodyTracker>;
+```
+
+### `<Segmentation type="portrait">` — Background segmentation (declarative)
+
+Registers the portrait segmentation pipeline. The resulting mask feeds into `<VideoBackground segmentationMask={...} />` for background replacement. Mount it alongside a ref or state that holds the mask texture.
+
+---
+
+## Tracker performance & lifecycle
+
+ML tracker models are the heaviest thing in an XR app — each one runs a MediaPipe pipeline on the camera feed. Two rules keep apps from overheating the device and from a scene-transition gesture bug.
+
+### Mount shared trackers ONCE, above any scene/phase switch
+
+A tracker is a refcounted model: mounting acquires it, unmounting disposes it. If you mount `<GestureTracker />` (or `<FaceTracker>`) **inside** each scene of a multi-scene flow (home → game → photo …), every transition tears the model down and rebuilds it. That (a) churns the GPU and (b) hits a dispose/re-acquire race where the gesture stops being reported after the transition (it works if you deep-link straight to the scene, but not when arriving from another scene).
+
+Fix: mount trackers that more than one scene needs **once**, above the phase switch, so the model stays alive for the whole session:
+
+```tsx
+// Scene.tsx
+<>
+  <GestureTracker /> {/* persistent — every phase reads its gesture */}
+  {renderCurrentPhase()}
+</>
+```
+
+Scope a tracker to a single scene only when just that scene uses it (e.g. a `<FaceTracker>` catcher that exists only during gameplay).
+
+### `active` — pause an off-scene tracker without unmounting it
+
+The "mount once" rule keeps a model alive across scenes, but a persistent tracker then runs its MediaPipe pipeline **every frame for the whole session** — even in scenes that don't use it. Each running model does a GPU→CPU readback per inference, and that readback is often the single largest cost in the app; two persistent trackers (e.g. gesture on the menus, face in gameplay) stack two readbacks the whole time when only one is ever needed at once.
+
+Every tracker takes an `active` prop (default `true`). Set `active={false}` and the model **pauses inference** — it stays mounted and initialized (no unmount, so no dispose/re-acquire race), it just stops doing per-frame work. Flip it back to `true` and it resumes on the next frame with no re-init. Gate each tracker to the phases that use it:
+
+```tsx
+// Scene.tsx — both trackers persistent (no race), but each only runs when needed.
+<>
+  <GestureTracker active={phase !== "game"} /> {/* menus/photo, not gameplay */}
+  <FaceTracker active={phase === "game"}>{/* the catcher */}…</FaceTracker>
+  {renderCurrentPhase()}
+</>
+```
+
+When several consumers share one model, it runs while **any** of them is active. `active` gates whether the model runs; `perf.targetFps` (below) caps how often it runs while active — use both. `<Segmentation>` also takes `active` (distinct from its `visible` prop, which only gates mask exposure while inference keeps running).
+
+### `perf` — cap inference rate + camera resolution
+
+Set `perf` on `<XRProvider>` to trade tracking fidelity for lower load and heat. This is the single biggest lever if the device runs hot.
+
+```tsx
+import { XRProvider } from "@vincentt-xr/sdk";
+
+<XRProvider perf={{ targetFps: 24, camRes: 720 }}>
+  <App />
+</XRProvider>;
+```
+
+- `targetFps` — caps every tracker model's inference rate (fps). The model's RAF loop still runs, but the expensive detect call is skipped between intervals. Omit → Face/Hand run uncapped, Gesture stays at its 15fps default, Segmentation at 24fps.
+- `camRes` — target width (px) of the image the models **analyze**. This is NOT the visible camera background the user sees — lowering it cuts ML work with no visible quality loss to the feed.
+- `cameraZoom` — **framing** control (not perf): how tightly the webcam is cover-cropped into the viewport. `1` = full native field of view (most of the room visible); default `1.2` fills the frame like a native selfie / Instagram front camera. Applies to the visible feed, the captured photo, and what the trackers see (all read the same crop). Raise for a closer crop, set `1` for the widest shot.
+
+Per-tracker override beats the session default:
+
+```tsx
+<GestureTracker targetFps={30} /> {/* crisp gestures … */}
+<FaceTracker targetFps={20}>…</FaceTracker> {/* … cheaper face */}
+```
+
+The base template ships `perf={{ targetFps: 24, camRes: 720 }}` by default — good for both dev laptops and end-user phones. Don't lower it further without a reason; don't remove it to "get full quality" unless you know the target device can take the heat.
+
+---
+
+## Reading tracker state
+
+### `<TrackingAnchor target="...">` — Declarative landmark following
+
+Wraps children; the child's world position is updated to the tracked landmark each frame. Use for all decoration that simply follows a face/hand/body point.
+
+Valid targets:
+
+- **Face:** `face.forehead`, `face.noseTip`, `face.chin`, `face.leftEyeOuter`, `face.rightEyeOuter`, `face.leftCheek`, `face.rightCheek`, `face.mouthCenter`
+- **Hand:** `hand.wrist`, `hand.thumbTip`, `hand.indexTip`, `hand.middleTip`, `hand.ringTip`, `hand.pinkyTip`, `hand.indexBase`, `hand.middleBase`, `hand.ringBase`, `hand.pinkyBase`
+- **Body:** `body.head`, `body.neck`, `body.leftShoulder`, `body.rightShoulder`, `body.leftElbow`, `body.rightElbow`, `body.leftWrist`, `body.rightWrist`, `body.leftHip`, `body.rightHip`, `body.leftKnee`, `body.rightKnee`, `body.leftAnkle`, `body.rightAnkle`
+
+```tsx
+<FaceTracker>
+  <TrackingAnchor target="face.noseTip">
+    <mesh name="noseDot" scale={[0.04, 0.04, 0.04]}>
+      <sphereGeometry />
+      <meshBasicMaterial color="red" />
+    </mesh>
+  </TrackingAnchor>
+</FaceTracker>
+```
+
+**`smoothing` — smooth motion from throttled inference.** Inference is throttled (the `perf` API caps it at 24fps by default) while rendering runs at 60-120fps, so a raw anchor holds then jumps on each new sample — visible stepping under head/hand movement. A plain lerp only hides it at rest; under motion it still steps. Pass `smoothing` (`~15` is a good default) and the anchor runs a **One Euro filter** — the standard adaptive low-pass for interactive tracking: it smooths hard when the landmark moves slowly (kills jitter) and loosens as it speeds up (kills lag), and never extrapolates so it can't overshoot or wobble. Higher `smoothing` is snappier (less lag, slightly more jitter); lower is smoother. It snaps (resets the filter) on first placement and on re-acquire, so it never slides in from a stale position. `0` (default) keeps the raw snap. Rotation is not smoothed. Prefer this over a hand-rolled `useFrame` lerp — a hand-rolled lerp reintroduces the stepping.
+
+```tsx
+<FaceTracker>
+  <TrackingAnchor target="face.noseTip" smoothing={15}>
+    <mesh name="sticker">{/* follows the nose smoothly, even while moving */}</mesh>
+  </TrackingAnchor>
+</FaceTracker>
+```
+
+### `useXRModelNode(XRModel.GESTURE_TRACKER)` — Read gesture state (the default read path)
+
+The gen-3 selector — subscribe to the live gesture node from `useFrame` when you need the gesture name or landmark coordinates for a state machine or fine-grained control. This is the blessed read layer; prefer it over raw node reads.
+
+```tsx
+import { XRModel } from "@vincentt-xr/sdk";
+import { useXRModelNode } from "@vincentt-xr/sdk/low-level";
+import type { XRModelNodeGestureTracking } from "@vincentt-xr/sdk/low-level";
+import { useFrame } from "@react-three/fiber";
+
+const gesture = useXRModelNode<XRModelNodeGestureTracking>(XRModel.GESTURE_TRACKER);
+
+useFrame(() => {
+  if (gesture?.gesture === "victory") {
+    // handle peace sign
+  }
+});
+```
+
+> **Escape hatch (rarely needed):** `useXRContext()` + `session.getModelNode(...)` from
+> `@vincentt-xr/sdk/low-level` reads the raw node imperatively. Prefer the selector
+> above; reach for the raw read only when you need the session handle for something the
+> selector can't express.
+
+### `<GestureTrigger gestures={["victory"]} onTrigger={(g) => ...}>` — Declarative one-shot gesture handler
+
+Fires once per gesture recognition event. The simplest way to act on a gesture — prefer it over a manual `useFrame` poll. Two modes:
+
+- **default (`mode="arm"`)** — fires the first time the gesture appears _after_ the user has _not_ been showing it for `debounceMs` (default 800). Use for "raise a NEW gesture to advance," where a stale frame from the prior scene must not fire.
+- **`mode="hold"`** — fires as soon as the gesture is held for `holdMs` (default 200), with **no arm gate**. Use for an **instruction-driven scene** ("show ✌️ to start") where the UI tells the user to raise the gesture and the hand is already up on mount — the arm gate needs a not-showing stretch first, so in `arm` mode it would never fire.
+
+```tsx
+import { GestureTracker, GestureTrigger } from "@vincentt-xr/sdk/tracking";
+
+<GestureTracker />
+
+// "show ✌️ to start" — hand is up on mount, so use hold mode
+<GestureTrigger
+  mode="hold"
+  gestures={["victory"]}
+  onTrigger={() => startGame()}
+/>
+
+// "make a NEW gesture to advance" — default arm mode
+<GestureTrigger
+  gestures={["thumb_up"]}
+  onTrigger={() => next()}
+/>
+```
+
+---
+
+## Camera background
+
+### `<VideoBackground customBackground="#6366f1" segmentationMask={maskTex} renderOrder={-999} />`
+
+In a v2-template project this is already mounted in `src/App.tsx` — do NOT add a second one there. Mentioned here because `customBackground` is how you do background replacement: pair it with `<Segmentation type="portrait" />` and pass the segmentation mask texture as `segmentationMask`.
+
+---
+
+## Mesh primitives and textures
+
+Use R3F intrinsic `<mesh>`, `<group>`, `<planeGeometry>`, `<meshBasicMaterial>`, etc. Load image textures with drei's `useTexture`.
+
+**Always name your meshes** (`name="myLabel"`). Names enable runtime lookup AND visual-feedback editing by the editor.
+
+**Materials accept `map`, not `alphaMap`, for image asset slots.**
+
+```tsx
+import { useTexture } from "@react-three/drei";
+
+const tex = useTexture("https://cdn.../badge.webp");
+
+<mesh name="badge" position={[0, 0.5, 0]}>
+  <planeGeometry args={[1, 1]} />
+  <meshBasicMaterial map={tex} transparent />
+</mesh>;
+```
+
+---
+
+## `<TextLabel>` — Canvas-rendered text panel
+
+Import from `@vincentt-xr/sdk`. Billboarded in 3D space.
+
+| prop           | type                | default                | notes                                                                                         |
+| -------------- | ------------------- | ---------------------- | --------------------------------------------------------------------------------------------- |
+| `text`         | `string`            | **required**           | The text to render.                                                                           |
+| `color`        | `string`            | `#ffffff`              | Text color.                                                                                   |
+| `fontSize`     | `number`            | `48`                   | Canvas px at the default 512px canvas — NOT world units. Chips/badges typically need 200-320. |
+| `bgColor`      | `string`            | `rgba(20,40,110,0.88)` | Panel background; use `"transparent"` for text-only.                                          |
+| `borderColor`  | `string`            | —                      | Border color; omit for no border.                                                             |
+| `borderRadius` | `number`            | `24`                   | Corner radius; 0 for square.                                                                  |
+| `aspect`       | `number`            | `2.5`                  | Panel shape (width ÷ height). >1 = wide/banner, <1 = tall.                                    |
+| `scale`        | `[x,y,z] \| number` | `1`                    | World size. Combine with `aspect` to make wide banners.                                       |
+| `position`     | `[x,y,z]`           | `[0,0,0]`              | World position. +Y up, -Y down.                                                               |
+| `name`         | `string`            | —                      | Set on every element you create.                                                              |
+
+**Gotchas:**
+
+- `fontSize` is canvas px, not world units. Chips/badges need 200-320, not 16-48.
+- `fontSize` does not change panel shape — use `aspect` (shape) and `scale` (size).
+- `TextLabel` ignores the `visible` prop — wrap in `<group visible={false}>` to start hidden.
+- **Inside a `<ScreenTransform>`, the parent stretches the label to fill the anchor rect — `aspect` must equal the rect's aspect or the banner distorts and leaves dead space around the text.** The rect's aspect is `(right − left) / (top − bottom)`. So for a full-width top banner `anchors={{ left: -1, right: 1, top: 1, bottom: 0.76 }}` → aspect `= 2 / 0.24 ≈ 8.3`. Set `borderRadius={0}` for a true edge-to-edge banner. (The rect controls size + placement — don't add `scale`/`position` to fill it; set the anchors instead.)
+
+```tsx
+// Full-width banner pinned to the top, INSIDE a ScreenTransform.
+// aspect = rect width (2) / rect height (0.24) ≈ 8.3 — matches the rect so it fills cleanly.
+<ScreenTransform anchors={{ left: -1, right: 1, top: 1, bottom: 0.76 }}>
+  <TextLabel name="topBanner" text="Head-Tilt Trivia" aspect={8.3} borderRadius={0} fontSize={270} />
+</ScreenTransform>
+
+// Full-width banner via position+scale, NOT inside a ScreenTransform.
+// Here aspect should match scale.x / scale.y (4 / 0.8 = 5).
+<TextLabel name="topBanner" text="Welcome" position={[0, 1.4, 0]} scale={[4, 0.8, 1]} aspect={5} fontSize={180} />
+```
+
+---
+
+## `<Panel>` — 3D billboarded panel for grouping UI content
+
+```tsx
+import { Panel, TextLabel } from "@vincentt-xr/sdk";
+
+<Panel>
+  <TextLabel name="title" text="Score: 10" fontSize={240} />
+</Panel>;
+```
+
+---
+
+## App shell
+
+### `<AspectRatioContainer>` — Letterbox the app into a 9:16 portrait frame
+
+Wrap the app root to keep the AR composition framed the same across phones and
+desktop: on wide viewports it pillarboxes to the 9:16 ratio; on portrait it fills
+the width.
+
+```tsx
+import { AspectRatioContainer } from "@vincentt-xr/sdk";
+
+<AspectRatioContainer>
+  <XRProvider>{/* ... */}</XRProvider>
+</AspectRatioContainer>;
+```
+
+---
+
+## Capture & share
+
+Grab a photo of the live render or record the canvas, then save or share it.
+
+### `session.captureFrame()` + `dataURLtoFile()` — Single-shot photo
+
+`captureFrame()` forces a fresh render and returns a `data:` URL of the current
+frame. `dataURLtoFile(dataUrl, filename)` turns it into a `File` for upload or the
+Web Share API.
+
+```tsx
+import { dataURLtoFile } from "@vincentt-xr/sdk";
+import { useXRContext } from "@vincentt-xr/sdk/low-level";
+
+const { session } = useXRContext();
+const dataUrl = session.captureFrame();
+const file = dataURLtoFile(dataUrl, "photo.png");
+```
+
+### `useMediaRecorder()` — Record the canvas to a video Blob
+
+Records a `MediaStream` (from the canvas's `captureStream()`) to a `Blob`. Picks
+the first supported codec (mp4 → webm). Pass `audio: true` to mix in the mic, or
+`audio: <MediaStream>` for your own audio track.
+
+```tsx
+import { useMediaRecorder } from "@vincentt-xr/sdk";
+
+const rec = useMediaRecorder();
+const canvas = document.querySelector("#vincentt-xr-canvas canvas");
+await rec.start(canvas.captureStream(30));
+// ...later
+rec.stop(); // -> rec.recordedBlob is the video Blob
+```
+
+---
+
+## Audio
+
+A silent-switch-safe sound engine for AR filters. Audio on iOS has three traps
+every app otherwise re-solves; this bundles the fix. Use it — do not hand-roll
+`new Audio()` or Web Audio.
+
+1. **Gesture-started audio is blocked.** Sound kicked off from a camera-detected
+   gesture (a peace sign to start) is not a user-input event, so iOS blocks it.
+   Run **muted by default** and let the tap that unmutes be the gesture that
+   unlocks playback (IG/Snap-style speaker toggle).
+2. **Web Audio is silenced by the hardware silent switch.** The engine uses
+   `<audio>` elements (the switch-immune "playback" category), not Web Audio.
+3. **The rewind seek janks a frame.** Pooled SFX rewind on their `ended` event,
+   off the hot path — so firing an effect never does a synchronous seek.
+
+Two paths, deliberately separate: **background music** is one looping element
+(plays/pauses on mute, never pooled); **interaction SFX** are short one-shots
+through a per-sound voice pool (a concurrency cap bounds decode load) with a
+per-URL coalesce window (the same sound twice in a frame plays once; two
+different sounds at once both play).
+
+The API is free functions over one default engine. Preload during the loading
+screen (`registerSound` / `prefetchMusic` return promises to await) so the
+unmute tap is instant.
+
+```tsx
+import {
+  registerSound,
+  prefetchMusic,
+  startMusic,
+  playOneShot,
+  toggleMuted,
+  useMuted,
+} from "@vincentt-xr/sdk";
+
+// During the loading gate: buffer every sound so unmute is instant.
+await Promise.all([
+  prefetchMusic(musicUrl), // one looping track
+  registerSound(catchUrl), // an SFX voice pool
+  registerSound(hitUrl),
+]);
+startMusic(musicUrl); // requested now; actually plays on unmute
+
+// Fire an effect from anywhere — including a non-React r3f collision callback.
+// No-op while muted; coalesced per URL; no seek on the hot path.
+playOneShot(catchUrl);
+
+// The speaker toggle. The first unmute is the gesture that unlocks iOS audio.
+function SoundToggle() {
+  const muted = useMuted();
+  return <button onClick={toggleMuted}>{muted ? "🔇" : "🔊"}</button>;
+}
+```
+
+Audio assets: m4a (AAC). A rare app needing a second, independent mixer can
+`new AudioEngine()` and drive it directly; the free functions always target the
+one default engine.

package/LICENSE

@@ -0,0 +1,7 @@
+Copyright (c) Vincentt. All rights reserved.
+
+This software and its source code are proprietary and confidential. The
+published package is provided for use with the Vincentt platform only. No
+license is granted to copy, modify, redistribute, reverse-engineer, or create
+derivative works, or to use this software to build a competing product, except
+under a separate written agreement with Vincentt.

package/README.md

@@ -0,0 +1,85 @@
+# RTX AR Tracking SDK
+
+This SDK is intended for implementing AI-powered tracking for WebAR projects by rangoon.tech.
+
+## Installation
+First, install the library by running the command
+
+```
+npm install @rangoon/track
+```
+
+If you're using yarn,
+
+```
+yarn add @rangoon/track
+```
+
+## How to use
+The usage of `@rangoon/track` usually involves two steps.
+
+### 1. Start a video stream
+Initialize the SDK by attaching the `@rangoon/track`'s `Webcam` component to your React Project. 
+The `Webcam` component will start a video stream which will be used for tracking. 
+
+```ts
+import { Webcam } from "@rangoon/track"
+import { useRef } from "react"
+
+function App() {
+
+  const videoRef = useRef<HTMLVideoElement>(null)
+
+  return (
+    <div>
+      <Webcam videoReference={videoRef}/>
+    </div>
+  )
+}
+
+export default App
+
+```
+
+### 2. Initiate a tracking model
+
+Currently, `@rangoon/track` supports three types of tracking models.
+* Hand Tracking (Landmarks)
+* Hand Gesture Tracking (Landmarks + Gestures)
+* Face Tracking
+
+By calling the Hook associated with the respective model, you can obtain the tracking data.
+
+Below is an example of obtaining hand-tracking data with `useHandTracking` Hook.
+```ts
+import { Webcam, useHandTracking } from "@rangoon/track"
+import { useEffect, useRef } from "react"
+
+function App() {
+
+  const videoRef = useRef<HTMLVideoElement>(null)
+
+  //Initiate the model by passing the webcam reference
+  const handData = useHandTracking({
+    videoReference: videoRef
+  })
+
+  //Watch the data changes with useEffect
+  useEffect(() => {
+    console.log(handData)
+  }, [handData])
+
+  return (
+    <div>
+      <Webcam videoReference={videoRef}/>
+    </div>
+  )
+}
+
+export default App
+```
+
+
+## Feedback
+
+If you have any feedback, please reach out to us at waing@rangoon.tech

package/dist/DebuggerUI-CEfdddkZ.cjs

@@ -0,0 +1 @@
+"use strict";Object.defineProperty(exports,Symbol.toStringTag,{value:"Module"});const e=require("react/jsx-runtime"),x=require("react"),C=require("react-dom"),s=require("./internal-DH5d1Xnq.cjs"),P={position:"fixed",top:"10px",left:"10px",zIndex:9999,fontFamily:"'Monaco', 'Menlo', 'Ubuntu Mono', 'Consolas', monospace",fontSize:"12px",color:"#fff",pointerEvents:"auto",userSelect:"none"},h={background:"rgba(0, 0, 0, 0.85)",borderRadius:"6px",padding:"12px 16px",boxShadow:"0 4px 12px rgba(0, 0, 0, 0.5)",backdropFilter:"blur(10px)",border:"1px solid rgba(255, 255, 255, 0.1)",minWidth:"240px"},k={...h,padding:"8px 12px",minWidth:"auto"},F={display:"flex",justifyContent:"space-between",alignItems:"center",marginBottom:"12px",paddingBottom:"8px",borderBottom:"1px solid rgba(255, 255, 255, 0.2)"},M={fontSize:"13px",fontWeight:"bold",color:"#00ff9f",letterSpacing:"0.5px",textTransform:"uppercase"},z={display:"flex",gap:"6px"},f={background:"rgba(255, 255, 255, 0.1)",border:"1px solid rgba(255, 255, 255, 0.2)",borderRadius:"4px",color:"#fff",padding:"4px 8px",fontSize:"11px",cursor:"pointer",transition:"all 0.2s"},R={display:"grid",gridTemplateColumns:"1fr 1fr",gap:"10px"},i={display:"flex",flexDirection:"column",gap:"4px"},o={fontSize:"10px",color:"rgba(255, 255, 255, 0.6)",textTransform:"uppercase",letterSpacing:"0.5px"},d={fontSize:"16px",fontWeight:"bold",color:"#fff"},u={...d,color:"#00ff9f"},g={...d,color:"#ffaa00"},m={...d,color:"#ff4444"},l={fontSize:"11px",color:"rgba(255, 255, 255, 0.5)",marginLeft:"4px"},N={width:"100%",height:"4px",background:"rgba(255, 255, 255, 0.2)",borderRadius:"2px",overflow:"hidden",marginTop:"4px"},w=t=>({width:`${t}%`,height:"100%",background:t>90?"#ff4444":t>70?"#ffaa00":"#00ff9f",transition:"width 0.3s ease, background 0.3s ease"}),D={fontSize:"9px",color:"rgba(255, 255, 255, 0.4)",marginTop:"8px",textAlign:"center",fontStyle:"italic"};function a(t,n){return n==="fps"?t>=30?u:t>=20?g:m:t<=40?u:t<=60?g:m}function B({metrics:t,collapsed:n,onCollapse:y,onClose:b,position:c="top-left",keyboardShortcut:j="ctrl+shift+d"}){const[p,v]=x.useState(null);if(x.useEffect(()=>{const r=document.createElement("div");return r.id="xr-debugger-overlay",document.body.appendChild(r),v(r),()=>{document.body.removeChild(r)}},[]),!p)return null;const S={...P,...c==="top-right"&&{left:"auto",right:"10px"},...c==="bottom-left"&&{top:"auto",bottom:"10px"},...c==="bottom-right"&&{top:"auto",bottom:"10px",left:"auto",right:"10px"}};return C.createPortal(e.jsx("div",{style:S,children:e.jsxs("div",{style:n?k:h,children:[e.jsxs("div",{style:F,children:[e.jsx("div",{style:M,children:n?"AR Debugger":"AR Performance"}),e.jsxs("div",{style:z,children:[e.jsx("button",{style:f,onClick:()=>y(!n),onMouseEnter:r=>{r.currentTarget.style.background="rgba(255, 255, 255, 0.2)"},onMouseLeave:r=>{r.currentTarget.style.background="rgba(255, 255, 255, 0.1)"},children:n?"+":"−"}),e.jsx("button",{style:f,onClick:b,onMouseEnter:r=>{r.currentTarget.style.background="rgba(255, 255, 255, 0.2)"},onMouseLeave:r=>{r.currentTarget.style.background="rgba(255, 255, 255, 0.1)"},children:"✕"})]})]}),!n&&e.jsxs(e.Fragment,{children:[e.jsxs("div",{style:R,children:[e.jsxs("div",{style:i,children:[e.jsx("div",{style:o,children:"Camera FPS"}),e.jsxs("div",{children:[e.jsx("span",{style:a(t.cameraFPS,"fps"),children:s.formatNumber(t.cameraFPS,0)}),e.jsx("span",{style:l,children:"fps"})]})]}),e.jsxs("div",{style:i,children:[e.jsx("div",{style:o,children:"Resolution"}),e.jsx("span",{style:d,children:t.resolution})]}),e.jsxs("div",{style:i,children:[e.jsx("div",{style:o,children:"Detection"}),e.jsxs("div",{children:[e.jsx("span",{style:a(t.detectionTime,"time"),children:s.formatNumber(t.detectionTime,1)}),e.jsx("span",{style:l,children:"ms"})]})]}),e.jsxs("div",{style:i,children:[e.jsx("div",{style:o,children:"Rendering"}),e.jsxs("div",{children:[e.jsx("span",{style:a(t.renderTime,"time"),children:s.formatNumber(t.renderTime,1)}),e.jsx("span",{style:l,children:"ms"})]})]}),e.jsxs("div",{style:i,children:[e.jsx("div",{style:o,children:"Effective FPS"}),e.jsxs("div",{children:[e.jsx("span",{style:a(t.effectiveFPS,"fps"),children:s.formatNumber(t.effectiveFPS,0)}),e.jsx("span",{style:l,children:"fps"})]})]}),e.jsxs("div",{style:i,children:[e.jsx("div",{style:o,children:"Memory"}),t.memory?e.jsxs(e.Fragment,{children:[e.jsxs("div",{children:[e.jsx("span",{style:d,children:s.formatNumber(t.memory.used,0)}),e.jsxs("span",{style:l,children:["/ ",s.formatNumber(t.memory.total,0)," MB"]})]}),e.jsx("div",{style:N,children:e.jsx("div",{style:w(t.memory.percentage)})})]}):e.jsx("span",{style:d,children:"N/A"})]})]}),t.modelPerformance.length>0&&e.jsx(e.Fragment,{children:e.jsxs("div",{style:{borderTop:"1px solid rgba(255, 255, 255, 0.15)",marginTop:"12px",paddingTop:"12px"},children:[e.jsx("div",{style:o,children:"Model Performance"}),e.jsx("div",{style:{marginTop:"8px",display:"flex",flexDirection:"column",gap:"6px"},children:t.modelPerformance.map((r,T)=>e.jsxs("div",{style:{display:"flex",justifyContent:"space-between",alignItems:"center",fontSize:"11px"},children:[e.jsx("span",{style:{color:"rgba(255, 255, 255, 0.7)"},children:r.name}),e.jsxs("div",{children:[e.jsx("span",{style:a(r.processingTime,"time"),children:s.formatNumber(r.processingTime,1)}),e.jsx("span",{style:l,children:"ms"})]})]},T))})]})}),e.jsxs("div",{style:D,children:["Press ",j.toUpperCase()," to toggle"]})]})]})}),p)}exports.DebuggerUI=B;