@myzonerocks/gosslens 0.9.0 → 0.10.1

package/README.md

@@ -1,44 +1,79 @@
 # Gosslens - TypeScript SDK
 
-TypeScript SDK for [Gosslens](../../include/gosslens.h), a camera engine
-with a Zig core behind one C ABI, compiled to `wasm32`. Wraps it as
-`GossEngine`, `GossSession`, and `Gosslens`, the same names the
-[Swift](../swift/README.md) and [Kotlin](../kotlin/README.md) SDKs use.
+TypeScript SDK for [Gosslens](../../include/gosslens.h), a camera engine behind
+one C ABI, compiled to `wasm32`. It wraps the engine as `GossEngine`,
+`GossSession`, and `Gosslens`, the same names the [Swift](../swift/README.md) and
+[Kotlin](../kotlin/README.md) SDKs use.
 
-This SDK owns camera capture through `getUserMedia`, the render loop,
-and decoding the PNGs the core has no decoder for. The frame graph, lens
-runtime, and effect pipeline live in the core.
-
-[docs/INTEGRATION-WEB.md](../../docs/INTEGRATION-WEB.md) is the start-to-finish
-guide: build and host the wasm and model assets, the render loop, and the
-tracking worker.
+This SDK owns camera capture through `getUserMedia`, the render loop, and
+decoding the PNGs the core has no decoder for. The frame graph, lens runtime, and
+effect pipeline live in the core. You write TypeScript; the engine is a prebuilt
+WebAssembly build you host and hand the SDK. The [demo](demo/) is a full working
+page, and the cross-platform SDK overview is in the [root README](../../README.md).
 
 ## Install
 
-```
+The npm package is the JavaScript wrapper you add with bun and write TypeScript
+against. It does not carry the engine. The engine it drives is the emscripten
+`gosslens_web.js`/`.wasm` (a WebGL2 build and a WebGPU build) plus
+`gosslens_tracking.wasm`, which you host next to your app.
+
+```sh
 bun add @myzonerocks/gosslens
 ```
 
-Inside this monorepo, consume it from the workspace instead:
+Because the browser cannot fetch a `.wasm` out of `node_modules` the way a native
+app links an archive, you serve the prebuilt engine files from your own static
+host and hand the SDK their URLs. The SDK never guesses a path. Every release
+attaches `gosslens-web-engine.zip` - the two `gosslens_web` builds (WebGPU and
+WebGL2 are separate artifacts, not a runtime toggle) and the tracking wasm - so
+unzip it into your static assets. Grab it from the
+[releases page](https://github.com/myzonerocks/gosslens/releases). `pickEngineUrl`
+picks the WebGL2 or WebGPU build at load time from the URLs you point it at, after
+confirming a real WebGPU adapter.
+
+### Building from source
+
+Prefer compiling the engine and the SDK yourself, from a clone or your own
+fork? Build the engine artifacts:
+
+```sh
+zig build wasm-emscripten
+zig build wasm-emscripten-webgpu
+zig build tracking-wasm
+zig build fetch-models
+```
+
+Inside this monorepo, consume the SDK as a workspace dependency instead of the
+published package:
 
 ```json
 { "dependencies": { "@myzonerocks/gosslens": "workspace:*" } }
 ```
 
-`bun run build` compiles `src/` to `dist/src/` (the package points `main`
-and `types` there); run it once so `dist/` exists before a workspace
+`bun run build` compiles `src/` to `dist/src/` (the package points `main` and
+`types` there); run it once in `sdk/ts` so `dist/` exists before a workspace
 consumer resolves it.
 
-This package is the JS wrapper only. `zig build wasm-emscripten` builds
-the WebGL2 `gosslens_web.js`/`.wasm` pair; `zig build
-wasm-emscripten-webgpu` builds the WebGPU pair. `pickEngineUrl` picks
-between them at load, so `wasmJsUrl` points at whichever one it selects.
-They are separate artifacts, not bundled.
+## The render loop
+
+`GossPreviewSession` does the engine, renderer, session, and capture loop in one
+call - most apps want this:
 
-## Use
+```typescript
+import { GossPreviewSession, pickEngineUrl } from "@myzonerocks/gosslens";
+
+const wasmJsUrl = await pickEngineUrl(webgpuUrl, webgl2Url);
+const preview = await GossPreviewSession.create(canvas, wasmJsUrl);
+preview.activateLens(manifestJson);
+```
 
-```ts
-import { Gosslens, GossEngine, GossSession, GossPreviewSession } from "@myzonerocks/gosslens";
+`pickEngineUrl` confirms a real WebGPU adapter before choosing, and falls back to
+the WebGL2 URL. `create` takes an optional third `events` argument for the
+capture-loop callbacks. If you drive the loop yourself, the pieces are public too:
+
+```typescript
+import { Gosslens, GossEngine, GossSession } from "@myzonerocks/gosslens";
 
 const gosslens = await Gosslens.load(canvas, wasmJsUrl);
 const engine = GossEngine.create(gosslens);
@@ -47,37 +82,374 @@ const session = GossSession.create(engine);
 
 session.submitFrameRgbaCopy(rgba, width * 4, width, height);
 engine.renderFrame(session);
+```
+
+## Camera controls
 
-session.setWhiten(0.6);
+The engine never touches the camera. It holds declarative intent you set,
+normalizes it, and hands it back for you to apply as getUserMedia track
+constraints. The controls live on `GossSession`, so reach them through
+`preview.session` if you took the `GossPreviewSession` path:
+
+```typescript
+session.setCameraControls({ ...session.cameraControls(), flashMode: 2, zoomFactor: 2 });
+
+const applied = session.cameraControls();
+await track.applyConstraints({ advanced: [{ zoom: applied.zoomFactor,
+                                             torch: applied.torch === 1 }] });
+```
+
+`GossCameraControls` also carries focus and exposure mode and points, the
+exposure bias, and the front-camera mirror-save policy. `setRecordingPolicy`/
+`recordingPolicy` (clip cap, segment and loop mode, speed preset, mic mute,
+save-original, stabilization) round-trips for your `MediaRecorder`, and
+`setCaptureUi`/`captureUi` (grid, level, shutter mode, self-timer, night mode,
+front-screen flash) for the capture chrome you draw. The engine validates and
+clamps every field; you read it back and apply it.
+
+## Lenses
+
+A lens is a manifest plus its assets. Activate one from its manifest JSON, and
+drop it again with `deactivateLens`:
+
+```typescript
 session.activateLens(manifestJson);
+session.deactivateLens();
 ```
 
-`GossPreviewSession.create(canvas, wasmJsUrl, events)` does all three setup
-steps and owns the capture loop too; most app code wants this one.
+The web build activates from the manifest JSON directly. The directory-bundle
+path the native SDKs also take is not wired here: a wasm target has no file IO,
+and the `shader.pass`, `lut.pass`, and `blend.pass` nodes it would carry need
+compiled resources this SDK has no way to hand over yet. A lens built entirely
+from `beauty.*` nodes (the beauty-baseline lens, say) activates and runs for real
+regardless, through the same beauty chain the effects below drive.
 
-WebGPU and WebGL2 are two separate build artifacts, not a runtime
-toggle; `pickEngineUrl` picks the right one after confirming a real
-adapter.
+A lens that reads face, hand, pose, or segmentation data renders nothing until
+you feed the matching tracker result each frame (see Tracking); it stays silent,
+with no error. A lens's triggers also react to signals you already feed:
+`camera.zoom`, `camera.focus` and `camera.exposure` follow the camera controls
+above, `geo.in_region` follows the geofence below, and `gaze.*` and
+`head.nod`/`head.shake`/`head.tilt` follow the face tracker. The full grammar is
+in [the lens spec](../../lenses/SPEC.md).
 
-Capture reads a PNG back, and world tracking feeds WebXR frames in:
+Advance the lens's own clock, triggers, and script nodes once per display frame
+with `tickLens`, passing the live signals this tick evaluates against:
 
-```ts
-import { GossWebXRWorldSource } from "@myzonerocks/gosslens";
+```typescript
+session.tickLens(dtUs, {
+  hasFace: result.presence >= 0.5,
+  blendshapes: result.blendshapes,
+});
+```
+
+Omitted signal fields read as false or zero, so a bare `tickLens(dtUs)` only
+advances triggers with no `when` gate. `fireEvent(name)` delivers a named event
+to the lens's `event('name')` triggers for the next tick, and
+`parameterValue(name)` reads a live lens parameter back, including whatever a
+script node last wrote.
+
+## Beauty and makeup
+
+The beauty effects are direct session calls, each an amount in 0..1:
+
+```typescript
+session.setSmooth(0.6);
+session.setWhiten(0.5);
+session.setThinFace(0.3);
+session.setBigEye(0.2);
+session.setLipstick(0.7);
+session.setBlush(0.4);
+```
 
-const png = await engine.captureFrame(session);   // data URL
+Whiten reads four lookup textures, and lipstick and blush their own source
+images, none of which the core decodes. Load them once after setup; the SDK
+decodes the PNGs, and until they resolve the matching setter stays a no-op:
+
+```typescript
+await session.loadWhitenLuts(new URL("./res/", baseUrl));     // the four lookup_*.png
+await session.loadMakeupTextures(new URL("./res/", baseUrl)); // mouth.png, blusher.png
+```
+
+`setMakeupReference` samples a reference photo's makeup color per face part: the
+lips, eyes, brows, and a cheek-and-forehead skin patch, so a lens's `tint.pass`
+with a reference source paints the live face in that color and a foundation over
+`face_skin` matches the reference's skin tone. Pass the reference RGBA and its
+478-point face landmarks; an empty landmarks array clears it:
+
+```typescript
+session.setMakeupReference(refRgba, refWidth, refHeight, refLandmarks);
+```
+
+## Tracking
+
+Face, hand, pose, and segmentation run in the `gosslens_tracking.wasm` module,
+off the main thread in a Worker. Each pipeline is a class that takes the module
+bytes and a model bundle and runs inference synchronously inside the worker:
+
+```typescript
+// tracking-worker.ts
+import { GossFaceTracker, GossHandTracker, GossPoseTracker } from "@myzonerocks/gosslens";
+
+const moduleBytes = await (await fetch(trackingWasmUrl)).arrayBuffer();
+const face = await GossFaceTracker.create(moduleBytes, faceTaskBytes);
+const hands = await GossHandTracker.create(moduleBytes, gestureTaskBytes);
+const pose = await GossPoseTracker.create(moduleBytes, poseTaskBytes);
+// per frame, on RGBA pixels from the camera canvas:
+const faceResult = face.process(rgba, width, height, timestampUs);
+const handResult = hands.process(rgba, width, height, timestampUs);
+const poseResult = pose.process(rgba, width, height, timestampUs);
+```
+
+Standing a tracker up is the web equivalent of the native SDKs' enable calls:
+this build runs no internal engine tracker, so you run the pipeline you need and
+feed its result back. `GossHandTracker` returns up to two hands, each with 21
+landmarks, a handedness score, and a canned gesture when the gesture bundle is
+loaded; `GossPoseTracker` returns the 33-point skeleton.
+[`demo/tracking-worker.ts`](demo/tracking-worker.ts) is the reference worker, and
+[`demo/track-worker.ts`](demo/track-worker.ts) stands all four pipelines up over
+still images.
+
+The `.task`/`.tflite` bundles (`face_landmarker.task`, `gesture_recognizer.task`,
+`pose_landmarker_full.task`, `selfie_multiclass.tflite`) are the ones
+`fetch-models` writes; host and fetch the ones your lenses use.
+
+Feed the tracked results back to the session for a lens to anchor to. The
+single-face path is `setFaceLandmarks`. To fan a face-anchored lens out across
+every face in frame, hand the engine the faces you tracked this frame:
+
+```typescript
+session.submitFaces(faces);      // GossFaceInput[], up to GOSS_FACE_MAX; [] clears
+for (let i = 0; i < session.faceCount(); i++) {
+  const face = session.faceResultAt(i);
+}
+```
+
+`submitBodies` is the multi-person equivalent, reaching every tracked figure;
+`setPoseUpperBody(true)` drops the skeleton's legs (knees down) for selfie
+framing where they are out of shot:
+
+```typescript
+session.submitBodies(bodies);    // GossPoseInput[], up to GOSS_BODY_MAX; [] clears
+for (let i = 0; i < session.bodyCount(); i++) {
+  const body = session.bodyResultAt(i);
+}
+```
+
+`faceRegion` returns the tracked point of a named attach point for pinning
+content - forehead, glabella, nose tip, chin, an eye, a cheek, an ear, or a mouth
+corner - reading the face landmarks you last submitted:
+
+```typescript
+const p = session.faceRegion(GossFaceRegion.Forehead);
+```
+
+For a body or hand joint, read the point off the tracker result directly: the
+pose result carries all 33 landmarks, each hand its 21. The `bodyJoint` and
+`handJoint` session pins read the engine's own internal trackers, which this
+build does not stand up, so they stay empty here; on native they return the named
+joint's point.
+
+The face tracker also drives the `gaze.*` and `head.nod`/`head.shake`/`head.tilt`
+lens triggers; `camera.*` follow the camera controls and `geo.in_region` the
+geofence below. The full grammar is in [the lens spec](../../lenses/SPEC.md).
+
+## Segmentation
+
+`GossSegmenter` runs a single `.tflite` model through the same tracking module
+and returns a subject mask, `GOSS_SEGMENTATION_MASK_SIDE` squared floats, plus the
+model's own class channels. Feed the subject mask back as the texture the lens's
+blend and mask channels sample:
+
+```typescript
+const segmenter = await GossSegmenter.create(moduleBytes, selfieMulticlassBytes);
+const subject = segmenter.process(rgba, width, height);
+session.setSegmentationMask(subject);      // null clears it
+```
+
+`selfie_multiclass.tflite` publishes the seven channels in
+`GOSS_SEGMENTATION_CHANNELS` (person, background, hair, body_skin, face_skin,
+clothes, others); `hair_segmenter.tflite` and `deeplab_v3.tflite` are the other
+shipped models, a single hair mask and another multiclass label set. A lens that
+names class channels reports them as a bitmask from `segmentationChannels`; upload
+exactly those each frame with `setSegmentationClassMask`, after the subject mask,
+since setting the subject clears the classes:
+
+```typescript
+const wanted = session.segmentationChannels();
+for (let channel = 1; channel < GOSS_SEGMENTATION_CHANNELS.length; channel++) {
+  if (wanted & (1 << channel)) {
+    session.setSegmentationClassMask(channel, segmenter.classMask(channel - 1));
+  }
+}
+```
+
+## World and AR
+
+`GossWebXRWorldSource` feeds WebXR frames into the session's world tracker, so a
+world-anchored lens or AR brush stroke stays fixed in the scene. Drive it from the
+XR animation loop:
+
+```typescript
+import { GossWebXRWorldSource } from "@myzonerocks/gosslens";
 
 const world = new GossWebXRWorldSource(session);
 // in the XR animation loop:
 world.onFrame(xrFrame, referenceSpace, timestampUs);
 ```
 
-On web, recording and photo capture run through the browser, used by both
-this SDK and the demo: `captureFrame()` exports a PNG off the composited
-canvas, and `captureStream()` drives a `MediaRecorder` with the
-engine-normalized recording policy. The engine's own hardware encoder and
-the platform-native HEIC format stay native-only, since the browser owns
-encoding on web; see [docs/PARITY.md](../../docs/PARITY.md). The
-cross-platform SDK overview is in the [root README](../../README.md).
+## Depth
+
+If the XR session was granted `depth-sensing`, feed each frame's depth so a
+depth-aware lens can occlude content behind real geometry. The map is metres per
+pixel, row major, with the near and far range it spans:
+
+```typescript
+const info = frame.getDepthInformation(view);   // WebXR depth-sensing
+// read info into a Float32Array of metres, then:
+session.submitDepth(depth, info.width, info.height, 0.1, 5.0);
+```
+
+An empty array clears it. The engine keeps the latest map for the occlusion pass.
+
+## Camera intrinsics
+
+If the camera reports its calibration, feed it once so an `undistort.pass` can
+straighten wide-angle lens distortion. The focal lengths and principal point are
+in pixels of the submitted frame, followed by the radial coefficients (k1, k2):
+
+```typescript
+// fx, fy, cx, cy from the platform's camera calibration, then:
+session.submitCameraIntrinsics(fx, fy, cx, cy, new Float32Array([k1, k2]));
+```
+
+An empty array clears them, leaving an `undistort.pass` inert.
+
+## Geofilters
+
+A lens can gate on place. Feed a location fix from the Geolocation API and
+describe the region the lens belongs to; the engine decides membership in wasm
+and the fix never leaves the page:
+
+```typescript
+navigator.geolocation.watchPosition((pos) => {
+  session.submitLocation(pos.coords.latitude, pos.coords.longitude,
+                         pos.coords.accuracy, pos.timestamp * 1000);
+});
+session.setGeofence(lat, lon, 150);
+```
+
+`setGeofenceBBox` and `setGeofencePolygon` describe a box or a ring instead;
+`setGeoAccuracy` sets the worst fix that still counts as inside, and
+`clearGeofence` drops the gate. Membership drives the lens grammar's
+`geo.in_region` trigger.
+
+## Brush
+
+Freehand strokes composite over the frame. Open a stroke, push normalized points,
+and close it; the engine keeps the undo/redo stack and hands back the ribbon
+(x, y, r, g, b, a per vertex):
+
+```typescript
+session.setBrushStyle(1, 0.4, 0.6, 1, 0.01);
+session.setBrushMode(3);              // 0 pen, 1 highlighter, 2 marker, 3 neon
+session.beginStroke();
+session.addStrokePoint(nx, ny);
+session.endStroke();
+const ribbon = session.brushVertices();
+```
+
+`setARBrushStyle`/`beginARStroke`/`addARStrokePoint(x, y, z)`/`endARStroke` are the
+world-anchored twin: points are pushed in the world frame world tracking reports,
+so a stroke stays fixed in the scene.
+
+## Capture and recording
+
+The browser owns encoding on the web, so photo and video capture run through it.
+`captureFrame` reads the composited canvas back and returns a PNG data URL:
+
+```typescript
+const png = await engine.captureFrame(session);   // or preview.captureFrame()
+```
+
+`captureStill` is the high-resolution path, decoupled from the preview size: it
+renders the composite at its own or a requested resolution, supersamples, and
+returns the encoded bytes with the format, gamut, and bit depth you tag. It needs
+the wasm renderer, so it is a no-op on the pure WebGL2 fallback:
+
+```typescript
+const jpeg = await engine.captureStill(session, { width: 4032, height: 3024, format: 1, quality: 90 });
+```
+
+For video, drive a `MediaRecorder` off `canvas.captureStream()` yourself.
+`setRecordingPolicy`/`recordingPolicy` round-trips the clip cap, segment and loop
+mode, speed preset, mic mute, save-original, and stabilization the engine
+normalizes, so the recorder and chrome you build read one validated policy. The
+still encoder is the core's own (PNG and JPEG, wide-gamut and 16-bit PNG tagged);
+the live video encoder stays native-only, which is why recording runs through the
+browser. See [PARITY.md](../../docs/PARITY.md).
+
+## Compositing
+
+Beyond the camera, a session composites named RGBA sources into one frame, for a
+duet, a stitch, or a live grid. The camera is the implicit source 0; define others
+by name and push a frame into each:
+
+```typescript
+session.defineSource("guest");
+session.submitSourceFrameRgba("guest", rgba, width, height, width * 4);
+session.setLayout(3);   // 0 custom, 1 side-by-side, 2 top-bottom, 3 pip, 4 grid, 5 overlay
+```
+
+`setSourceComposite` sets a source's opacity and key mode (none, matte, or a
+chroma key with its color and similarity), so a keyed guest drops onto the base:
+
+```typescript
+session.setSourceComposite("guest", 1, 2, [0, 1, 0], 0.4);   // chroma-key green
+```
+
+`defineScreenShare` registers a source whose frame letterboxes to fit its cell,
+`removeSource` drops one, and `clearLayout` returns to the camera alone.
+
+## Lives and calls
+
+The web is the easy case: the rendered canvas is already a live video source.
+`canvas.captureStream()` hands you a `MediaStreamTrack` of the composited,
+lens-baked output with no readback and no copy - publish it straight to LiveKit or
+any WebRTC peer:
+
+```typescript
+const track = canvas.captureStream(30).getVideoTracks()[0];
+// publish `track` through your LiveKit room or RTCPeerConnection
+```
+
+Keep the render loop running (`renderFrame` per frame) and the track carries every
+composited frame. Reach for `captureLiveFrame` only when you need the raw pixels
+(BGRA by default) rather than a track. For audio, `mixOutputAudio` folds the
+lens's own sound into the mic block you are about to publish and returns the mixed
+interleaved s16 for your outgoing WebRTC audio track: it resamples the lens sound
+to your track's rate and sums it in, so there is nothing to hand-mix (pass `null`
+for the mic to send the lens sound over silence). `pullAudio` still pulls the lens
+sound alone for local WebAudio playback with no call in progress; `GossAudioOutput`
+wraps that playback (an `AudioWorklet` it owns, `start()` from a gesture, `pump()`
+each frame beside `tickLens`). `GossMicInput` captures the microphone into
+`submitAudio` so level and beat triggers fire in the browser, and
+`GossVideoTexture` plays an MP4 through the browser's decoder into a named
+source a lens composites.
+
+When the lens carries an `audio.infer` node with a caption binding, the engine
+runs on-device ASR over the mic and `captionText` reads the decoded text by the
+node's id, for the page to draw as a live subtitle:
+
+```typescript
+const line = session.captionText("caption");
+if (line) subtitleEl.textContent = line;
+```
+
+## Method names
+
+The operation names match the other SDKs: `GossEngine.create(gosslens)`,
+`GossSession.create(engine)`, `submitFrameRgbaCopy`, `renderFrame`. The full table
+is in [API.md](../../docs/API.md).
 
 ## Demo app
 
@@ -89,8 +461,3 @@ cross-platform SDK overview is in the [root README](../../README.md).
 parsers and the WebGPU/WebGL2 pick. The browser end-to-end proofs live in
 [`demo/prove.ts`](demo/prove.ts) and [`demo/track-prove.ts`](demo/track-prove.ts),
 with the host conformance in [`harness/`](../../harness/).
-
-## TODO
-
-- Publish `@myzonerocks/gosslens` to npm; until then `bun add` resolves
-  only inside the workspace.

package/dist/src/audio-output.d.ts

@@ -0,0 +1,12 @@
+import type { GossSession } from "./index";
+export declare class GossAudioOutput {
+    static readonly sampleRate = 48000;
+    private session;
+    private context;
+    private node;
+    constructor(session: GossSession);
+    start(): Promise<void>;
+    pump(frames?: number): void;
+    stop(): Promise<void>;
+}
+//# sourceMappingURL=audio-output.d.ts.map

package/dist/src/audio-output.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"audio-output.d.ts","sourceRoot":"","sources":["../../src/audio-output.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,SAAS,CAAC;AAgD3C,qBAAa,eAAe;IAE1B,MAAM,CAAC,QAAQ,CAAC,UAAU,SAAU;IAEpC,OAAO,CAAC,OAAO,CAAc;IAC7B,OAAO,CAAC,OAAO,CAA6B;IAC5C,OAAO,CAAC,IAAI,CAAiC;IAE7C,YAAY,OAAO,EAAE,WAAW,EAE/B;IAIK,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC,CAkB3B;IAMD,IAAI,CAAC,MAAM,SAAM,GAAG,IAAI,CAKvB;IAGK,IAAI,IAAI,OAAO,CAAC,IAAI,CAAC,CAM1B;CACF"}

package/dist/src/audio-output.js

@@ -0,0 +1,99 @@
+/// The worklet drains chunks the page posts it, so the audio thread
+/// never touches the wasm heap and the engine's graph-thread pull
+/// contract holds; an underrun renders silence.
+const processorSource = `
+class GossLensAudioProcessor extends AudioWorkletProcessor {
+  constructor() {
+    super();
+    this.queue = [];
+    this.offset = 0;
+    this.queued = 0;
+    this.port.onmessage = (event) => {
+      this.queue.push(event.data);
+      this.queued += event.data.length;
+      while (this.queue.length > 1 && this.queued - this.queue[0].length + this.offset > 16384) {
+        this.queued -= this.queue[0].length;
+        this.queue.shift();
+        this.offset = 0;
+      }
+    };
+  }
+  process(_inputs, outputs) {
+    const out = outputs[0][0];
+    if (!out) return true;
+    for (let i = 0; i < out.length; i += 1) {
+      const head = this.queue[0];
+      if (!head) {
+        out[i] = 0;
+        continue;
+      }
+      out[i] = head[this.offset] / 32768;
+      this.offset += 1;
+      if (this.offset >= head.length) {
+        this.queued -= head.length;
+        this.queue.shift();
+        this.offset = 0;
+      }
+    }
+    return true;
+  }
+}
+registerProcessor("goss-lens-audio", GossLensAudioProcessor);
+`;
+/// Routes the lens mixer to the page's speakers. The engine pull is
+/// graph-thread only, so pump() runs in the frame loop on the main
+/// thread and posts each block to an AudioWorklet that plays it.
+export class GossAudioOutput {
+    /// The lens mixer's fixed output format.
+    static sampleRate = 48_000;
+    session;
+    context = null;
+    node = null;
+    constructor(session) {
+        this.session = session;
+    }
+    /// Stands the worklet up; browsers require a user gesture before an
+    /// AudioContext runs, so call this from one and await it.
+    async start() {
+        if (this.context)
+            return;
+        const context = new AudioContext({ sampleRate: GossAudioOutput.sampleRate });
+        const moduleUrl = URL.createObjectURL(new Blob([processorSource], { type: "text/javascript" }));
+        try {
+            await context.audioWorklet.addModule(moduleUrl);
+        }
+        finally {
+            URL.revokeObjectURL(moduleUrl);
+        }
+        const node = new AudioWorkletNode(context, "goss-lens-audio", {
+            numberOfInputs: 0,
+            numberOfOutputs: 1,
+            outputChannelCount: [1],
+        });
+        node.connect(context.destination);
+        await context.resume();
+        this.context = context;
+        this.node = node;
+    }
+    /// Pulls the next mixer block and hands it to the worklet. Call once
+    /// per frame from the same loop that ticks the lens. The post
+    /// transfers the block's buffer, which detaches it, so each pump is
+    /// one fresh small block by design rather than a reusable buffer.
+    pump(frames = 800) {
+        const node = this.node;
+        if (!node)
+            return;
+        const block = this.session.pullAudio(frames);
+        node.port.postMessage(block, [block.buffer]);
+    }
+    /// Tears the worklet and context down.
+    async stop() {
+        this.node?.disconnect();
+        this.node = null;
+        const context = this.context;
+        this.context = null;
+        if (context)
+            await context.close();
+    }
+}
+//# sourceMappingURL=audio-output.js.map

package/dist/src/audio-output.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"audio-output.js","sourceRoot":"","sources":["../../src/audio-output.ts"],"names":[],"mappings":"AAEA,oEAAoE;AACpE,kEAAkE;AAClE,gDAAgD;AAChD,MAAM,eAAe,GAAG;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;CAsCvB,CAAC;AAEF,oEAAoE;AACpE,mEAAmE;AACnE,iEAAiE;AACjE,MAAM,OAAO,eAAe;IAC1B,yCAAyC;IACzC,MAAM,CAAU,UAAU,GAAG,MAAM,CAAC;IAE5B,OAAO,CAAc;IACrB,OAAO,GAAwB,IAAI,CAAC;IACpC,IAAI,GAA4B,IAAI,CAAC;IAE7C,YAAY,OAAoB;QAC9B,IAAI,CAAC,OAAO,GAAG,OAAO,CAAC;IACzB,CAAC;IAED,oEAAoE;IACpE,0DAA0D;IAC1D,KAAK,CAAC,KAAK;QACT,IAAI,IAAI,CAAC,OAAO;YAAE,OAAO;QACzB,MAAM,OAAO,GAAG,IAAI,YAAY,CAAC,EAAE,UAAU,EAAE,eAAe,CAAC,UAAU,EAAE,CAAC,CAAC;QAC7E,MAAM,SAAS,GAAG,GAAG,CAAC,eAAe,CAAC,IAAI,IAAI,CAAC,CAAC,eAAe,CAAC,EAAE,EAAE,IAAI,EAAE,iBAAiB,EAAE,CAAC,CAAC,CAAC;QAChG,IAAI,CAAC;YACH,MAAM,OAAO,CAAC,YAAY,CAAC,SAAS,CAAC,SAAS,CAAC,CAAC;QAClD,CAAC;gBAAS,CAAC;YACT,GAAG,CAAC,eAAe,CAAC,SAAS,CAAC,CAAC;QACjC,CAAC;QACD,MAAM,IAAI,GAAG,IAAI,gBAAgB,CAAC,OAAO,EAAE,iBAAiB,EAAE;YAC5D,cAAc,EAAE,CAAC;YACjB,eAAe,EAAE,CAAC;YAClB,kBAAkB,EAAE,CAAC,CAAC,CAAC;SACxB,CAAC,CAAC;QACH,IAAI,CAAC,OAAO,CAAC,OAAO,CAAC,WAAW,CAAC,CAAC;QAClC,MAAM,OAAO,CAAC,MAAM,EAAE,CAAC;QACvB,IAAI,CAAC,OAAO,GAAG,OAAO,CAAC;QACvB,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC;IACnB,CAAC;IAED,qEAAqE;IACrE,8DAA8D;IAC9D,oEAAoE;IACpE,kEAAkE;IAClE,IAAI,CAAC,MAAM,GAAG,GAAG;QACf,MAAM,IAAI,GAAG,IAAI,CAAC,IAAI,CAAC;QACvB,IAAI,CAAC,IAAI;YAAE,OAAO;QAClB,MAAM,KAAK,GAAG,IAAI,CAAC,OAAO,CAAC,SAAS,CAAC,MAAM,CAAC,CAAC;QAC7C,IAAI,CAAC,IAAI,CAAC,WAAW,CAAC,KAAK,EAAE,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,CAAC;IAC/C,CAAC;IAED,uCAAuC;IACvC,KAAK,CAAC,IAAI;QACR,IAAI,CAAC,IAAI,EAAE,UAAU,EAAE,CAAC;QACxB,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC;QACjB,MAAM,OAAO,GAAG,IAAI,CAAC,OAAO,CAAC;QAC7B,IAAI,CAAC,OAAO,GAAG,IAAI,CAAC;QACpB,IAAI,OAAO;YAAE,MAAM,OAAO,CAAC,KAAK,EAAE,CAAC;IACrC,CAAC;CACF"}

package/dist/src/index.d.ts

@@ -526,4 +526,7 @@ export declare class GossPreviewSession {
 }
 export { GossWebXRWorldSource } from "./world";
 export type { GossXRFrameLike } from "./world";
+export { GossAudioOutput } from "./audio-output";
+export { GossMicInput } from "./mic-input";
+export { GossVideoTexture } from "./video-texture";
 //# sourceMappingURL=index.d.ts.map