react-native-webrtc-kaleidoscope 1.1.0 → 2.0.0

package/NOTICE.md

@@ -6,11 +6,22 @@ react-native-webrtc-kaleidoscope ships under the MIT license (see [LICENSE](./LI
 
 - **react-native-webrtc** — MIT license. <https://github.com/react-native-webrtc/react-native-webrtc>. The `track._setVideoEffects(...)` JS surface, the Android `ProcessorProvider` registry, and the iOS `RTCVideoFrameProcessor` protocol all originate upstream. This package is a consumer of, not a fork of, that codebase.
 
-## Native segmentation backends (used by the `blur` effect)
+## Segmentation backends (used by the `blur` and `background-image` effects)
 
-- **Apple Vision** (iOS) — system framework. `VNGeneratePersonSegmentationRequest`. Subject to the Apple SDK License Agreement.
-- **MLKit Selfie Segmentation** (Android) — Apache License 2.0. Distributed by Google as a managed AAR. <https://developers.google.com/ml-kit/vision/selfie-segmentation>.
-- **MediaPipe Selfie Segmentation** (web) — Apache License 2.0. Loaded as an `optionalDependency` so native consumers do not bundle the WASM payload. <https://github.com/google-ai-edge/mediapipe>.
+All three platforms run Google's MediaPipe selfie segmentation. The native targets use the MediaPipe Tasks Image Segmenter with a model bundled in this package; the web target uses the legacy MediaPipe Selfie Segmentation Solution loaded at runtime from a CDN.
+
+- **MediaPipe Tasks Vision** (Android) — Apache License 2.0. The `com.google.mediapipe:tasks-vision` AAR, `ImageSegmenter` API. <https://github.com/google-ai-edge/mediapipe>.
+- **MediaPipe Tasks Vision** (iOS) — Apache License 2.0. The `MediaPipeTasksVision` CocoaPod, `ImageSegmenter` API. <https://github.com/google-ai-edge/mediapipe>.
+- **MediaPipe Selfie Segmentation** (web) — Apache License 2.0. The `@mediapipe/selfie_segmentation` Solution, loaded at runtime from `cdn.jsdelivr.net` (not bundled in this package). <https://github.com/google-ai-edge/mediapipe>.
+
+### Bundled model
+
+The native targets redistribute a TensorFlow Lite model inside the published package; it ships in `android/src/main/assets/selfie_segmenter.tflite` and in the iOS `Kaleidoscope.bundle`.
+
+- **MediaPipe SelfieSegmenter** (`selfie_segmenter.tflite`) — distributed by Google as part of MediaPipe Solutions.
+  - Source: <https://storage.googleapis.com/mediapipe-models/image_segmenter/selfie_segmenter/float16/latest/selfie_segmenter.tflite>
+  - Model card: <https://storage.googleapis.com/mediapipe-assets/Model%20Card%20MediaPipe%20Selfie%20Segmentation.pdf>
+  - The MediaPipe project and SDK are published by Google under the Apache License 2.0. The `selfie_segmenter.tflite` weights are redistributed here as published by Google as part of MediaPipe Solutions; Google does not attach a separate license to the hosted model file, so its use is governed by the model card above. The web target does not bundle this file; it fetches its model from the CDN at runtime.
 
 ## Algorithmic and architectural references
 
@@ -19,8 +30,8 @@ react-native-webrtc-kaleidoscope ships under the MIT license (see [LICENSE](./LI
 
 ## Trademarks
 
-- "Apple", "Vision", "Core Image", and related Apple marks are trademarks of Apple Inc. Used here as nominative references only.
-- "Google", "MLKit", and "MediaPipe" are trademarks of Google LLC. Used here as nominative references only.
+- "Apple", "Core Image", "Metal", and related Apple marks are trademarks of Apple Inc. Used here as nominative references only.
+- "Google", "MediaPipe", and "TensorFlow Lite" are trademarks of Google LLC. Used here as nominative references only.
 - "WebRTC" is a project of the W3C and IETF.
 
 This package is not affiliated with, endorsed by, or certified by Apple, Google, the W3C, or the `react-native-webrtc` project.

package/README.md

@@ -8,9 +8,11 @@
 [![npm version](https://img.shields.io/npm/v/react-native-webrtc-kaleidoscope?color=cb3837&logo=npm)](https://www.npmjs.com/package/react-native-webrtc-kaleidoscope)
 [![Types: included](https://img.shields.io/npm/types/react-native-webrtc-kaleidoscope?color=3178c6&logo=typescript)](https://www.npmjs.com/package/react-native-webrtc-kaleidoscope)
 [![CI](https://github.com/simiancraft/react-native-webrtc-kaleidoscope/actions/workflows/ci.yml/badge.svg)](https://github.com/simiancraft/react-native-webrtc-kaleidoscope/actions/workflows/ci.yml)
+[![codecov](https://codecov.io/gh/simiancraft/react-native-webrtc-kaleidoscope/graph/badge.svg)](https://codecov.io/gh/simiancraft/react-native-webrtc-kaleidoscope)
+[![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/simiancraft/react-native-webrtc-kaleidoscope/badge)](https://securityscorecards.dev/viewer/?uri=github.com/simiancraft/react-native-webrtc-kaleidoscope)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](./LICENSE)
 
-> Live video effects for `react-native-webrtc`, packaged as a managed-Expo-friendly Expo Module.
+> Creative, shader-based video effects for React Native video calls: blur or replace a person's background, with more camera effects to come. Works with `react-native-webrtc` and LiveKit, managed-Expo-friendly.
 
 ## Status
 
@@ -18,17 +20,17 @@
 
 ### What works today
 
-- **Mirror** (horizontal flip).
+- **Transform** (flip X / flip Y / rotate 90° CW / CCW) — orientation utilities, identical across platforms.
 - **Blur** (background blur, person stays sharp).
-- **Background replacement** (composite a still PNG behind the segmented person; two bundled office presets, library-side asset pipeline).
+- **Background replacement** (composite a still image behind the segmented person; eleven bundled presets, plus arbitrary URLs on web; library-side asset pipeline).
 - **Runtime tuning** of the GLSL effects; see the [Use](#use) section.
 
-| Platform | Mirror | Blur | Background replacement | Notes |
+| Platform | Transform | Blur | Background replacement | Notes |
 |---|---|---|---|---|
 | Web (Chrome / Edge) | ✓ | ✓ | ✓ | MediaStreamTrackProcessor + MediaPipe Selfie Segmentation (WASM, CDN) |
-| Android (API 24+) | ✓ | ✓ | ✓ | OpenGL ES 3.0 + MLKit Selfie Segmentation |
+| Android (API 24+) | ✓ | ✓ | ✓ | OpenGL ES 3.0 + MediaPipe Selfie Segmentation (Tasks) |
 | iOS (≥ 15) | ✓ | ✓ | ✓ | Metal + Vision (person segmentation), verified on device. Older A11 devices (iPhone X) run at a lower frame rate |
-| Safari / Firefox | — | — | — | No Insertable Streams; `applyVideoEffects` throws a typed error |
+| Safari / Firefox | — | — | — | No Insertable Streams; `applyVideoEffects` throws a clear capability error |
 
 ### Coming soon
 
@@ -101,31 +103,50 @@ import {
   setMaskHardness,
   setMaskThreshold,
 } from 'react-native-webrtc-kaleidoscope';
+import { darkOffice } from 'react-native-webrtc-kaleidoscope/backgrounds/dark-office';
 
 const stream = await mediaDevices.getUserMedia({ video: true });
 const [track] = stream.getVideoTracks();
 
-applyVideoEffects(track, ['mirror']);
+applyVideoEffects(track, ['flip-x']); // also flip-y, rotate-cw, rotate-ccw
 applyVideoEffects(track, ['blur']);
-applyVideoEffects(track, [{ name: 'background-image', source: 'office-1' }]);
+applyVideoEffects(track, [{ name: 'background-image', source: darkOffice }]);
 applyVideoEffects(track, []); // clear all effects
 
 // Runtime tuning (effects pick up the new values on the next frame):
-setBlurSigma(25);        // Gaussian σ; clamped to [0.5, 64], default 8.
-setMaskHardness(0.2);    // smoothstep transition width; clamped to [0, 1]. 0 = soft halo, 1 = near-step. Default 0.5.
-setMaskThreshold(0.7);   // smoothstep center; clamped to [0.05, 0.95]. Higher rejects low-confidence pixels. Default 0.5.
+setBlurSigma(5);         // Gaussian σ; clamped to [0.5, 7], default 5.
+setMaskHardness(0.5);    // smoothstep transition width; clamped to [0, 1]. 0 = soft halo, 1 = near-step. Default 0.5.
+setMaskThreshold(0.7);   // smoothstep center; clamped to [0.05, 0.95]. Higher rejects low-confidence pixels. Default 0.7.
 ```
 
 Effects chain in array order.
 
-**Tuning note:** optimal values are platform-specific because each segmentation model (MediaPipe on web, MLKit on Android, Vision when iOS lands) produces a different confidence distribution. Working defaults on a typical well-lit scene:
+**Tuning note:** all three platforms run MediaPipe selfie segmentation (Tasks Image Segmenter on native, the Selfie Segmentation Solution on web), but optimal values still differ slightly because the input downscale and the API variant differ per platform, producing different confidence distributions. Working defaults on a typical well-lit scene:
 
 | Platform | Blur sigma | Mask hardness | Mask threshold |
 |---|---|---|---|
 | Web (MediaPipe) | 25 | 0.2 | 0.85 |
-| Android (MLKit) | 30 | 0.2 | 0.6 |
+| Android (MediaPipe) | 30 | 0.2 | 0.6 |
 
-The library ships neutral defaults (8, 0.5, 0.5) and consumers tune at runtime via the API above; whether to ship the dialed-in values as platform-specific defaults is an open question waiting on iOS data.
+The library ships defaults (5, 0.5, 0.7) and consumers tune at runtime via the API above; whether to ship the dialed-in per-platform values as defaults is an open question.
+
+## Background presets
+
+Eleven backgrounds ship for the `background-image` effect, imported per preset (e.g. `import { darkOffice } from 'react-native-webrtc-kaleidoscope/backgrounds/dark-office'`). On web a preset can also be any image URL or data URI; native resolves bundled preset names only.
+
+| Theme | Light | Dark |
+|---|---|---|
+| Office | <img src="src/backgrounds/light-office.webp" width="220" alt="light-office" /> | <img src="src/backgrounds/dark-office.webp" width="220" alt="dark-office" /> |
+| Home | <img src="src/backgrounds/home-light.webp" width="220" alt="home-light" /> | <img src="src/backgrounds/home-dark.webp" width="220" alt="home-dark" /> |
+| Nature | <img src="src/backgrounds/nature-light.webp" width="220" alt="nature-light" /> | <img src="src/backgrounds/nature-dark.webp" width="220" alt="nature-dark" /> |
+| Stylized | <img src="src/backgrounds/stylized-light.webp" width="220" alt="stylized-light" /> | <img src="src/backgrounds/stylized-dark.webp" width="220" alt="stylized-dark" /> |
+| Simiancraft | <img src="src/backgrounds/simiancraft-light.webp" width="220" alt="simiancraft-light" /> | <img src="src/backgrounds/simiancraft-dark.webp" width="220" alt="simiancraft-dark" /> |
+
+Plus **`debug-resolutions`**, a viewport/resolution calibration grid for verifying background cover-fit:
+
+<img src="src/backgrounds/debug-resolutions.webp" width="220" alt="debug-resolutions" />
+
+See [`src/backgrounds/README.md`](./src/backgrounds/README.md) for sizing and how to add a preset.
 
 ## Web and native differences
 
@@ -133,9 +154,9 @@ The API surface is the same across platforms, but the runtimes differ in ways wo
 
 - **Effect parameters.** Web reads tuning from the global setters (`setBlurSigma`, `setMaskHardness`, `setMaskThreshold`) on the next frame. Native currently ignores per-call `EffectSpec` parameters such as `{ name: 'blur', sigma: 12 }`; tuning is global through the same setters. Per-call uniforms through the native registry are a follow-up.
 - **Background source.** `background-image.source` is a bundled preset name on native (the upstream `_setVideoEffects` registry is keyed by flat strings, not URIs), but on web it accepts either a preset name or an arbitrary image URL or data URI.
-- **Background presets ship as tree-shakeable files.** Two example backgrounds (`office-1`, `office-2`) are importable per preset: `import { office1 } from 'react-native-webrtc-kaleidoscope/backgrounds/office-1'`. Each preset is its own file behind its own subpath export, and the package sets `sideEffects: false`, so an unused preset is dropped by web bundlers — and, since Metro doesn't tree-shake, simply never imported on native. Web resolves the bundled WebP to a URL; native loads its own bundled copy by name. Web also still accepts an arbitrary image URL or data URI. See [`src/backgrounds/README.md`](./src/backgrounds/README.md).
+- **Background presets ship as tree-shakeable files.** The bundled backgrounds (see [Background presets](#background-presets)) are importable per preset: `import { darkOffice } from 'react-native-webrtc-kaleidoscope/backgrounds/dark-office'`. Each preset is its own file behind its own subpath export, and the package sets `sideEffects: false`, so an unused preset is dropped by web bundlers — and, since Metro doesn't tree-shake, simply never imported on native. Web resolves the bundled WebP to a URL; native loads its own bundled copy by name. Web also still accepts an arbitrary image URL or data URI. See [`src/backgrounds/README.md`](./src/backgrounds/README.md).
 - **Segmentation model on web.** The web blur and background-image effects load MediaPipe Selfie Segmentation from the jsdelivr CDN (`cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation`) on first use. A strict Content-Security-Policy must allow that origin for `script-src`, `connect-src`, and the WASM fetch, and the effects do not work offline. Mirror needs no model.
-- **Browser support on web.** Effects use Insertable Streams (`MediaStreamTrackProcessor` and `MediaStreamTrackGenerator`), which ship in Chromium-based browsers; Safari and Firefox throw a typed capability error.
+- **Browser support on web.** Effects use Insertable Streams (`MediaStreamTrackProcessor` and `MediaStreamTrackGenerator`), which ship in Chromium-based browsers (Chrome, Edge); Safari and Firefox lack the API, so `applyVideoEffects` throws a clear capability error and the demo falls back to the unprocessed track.
 
 ## What this isn't
 
@@ -150,8 +171,8 @@ The codebase lives across four surfaces:
 
 - `src/` — JS facade and shared types. `applyVideoEffects(track, effects)` plus runtime tuning setters.
 - `src/web/` — WebGL2 pipeline. MediaPipe segmentation + GLSL composite. One shader file per stage in `src/web/shaders.ts`.
-- `android/` — OpenGL ES 3.0 pipeline. MLKit segmentation (async, worker-thread, last-known-mask cache) + GLSL composite. Shaders inline in `gpu/Shaders.kt` as `const val` strings.
-- `ios/` — Metal pipeline (Swift) with Vision person segmentation. The canonical GLSL in `shaders/` transpiles to Metal Shading Language via `scripts/build-shaders.ts`. Implemented and verified on device.
+- `android/` — OpenGL ES 3.0 pipeline. MediaPipe Tasks segmentation (async, worker-thread, last-known-mask cache) + GLSL composite. Shaders inline in `gpu/Shaders.kt` as `const val` strings.
+- `ios/` — Metal pipeline (Swift) with MediaPipe Tasks segmentation (`selfie_segmenter.tflite`, the same model the Android target bundles). The canonical GLSL in `shaders/` transpiles to Metal Shading Language via `scripts/build-shaders.ts`. Implemented and verified on device.
 
 The composite shader (`shaders/composite.frag`) is the same GLSL source for every effect category (blur, background-image, future procedural backgrounds). Per-effect difference is upstream of the composite: how the `uBackground` texture gets produced.
 

package/android/build.gradle

@@ -55,10 +55,9 @@ dependencies {
   }
   compileOnly project(rnWebrtcProject)
 
-  // Person/background mask for the blur effect. The Google artifact ID is
-  // `segmentation-selfie`, not `selfie-segmentation` (the plan and earlier
-  // notes had it inverted; that is why the pin would not resolve).
-  implementation 'com.google.mlkit:segmentation-selfie:16.0.0-beta6'
+  // Person/background mask: MediaPipe Tasks ImageSegmenter (Mask.kt), the same
+  // model family the web and iOS sides run.
+  implementation 'com.google.mediapipe:tasks-vision:0.10.14'
 }
 
 // Helper: gracefully read root project ext properties with a default.

package/android/src/main/java/com/simiancraft/kaleidoscope/EffectTuning.kt

@@ -13,14 +13,14 @@ package com.simiancraft.kaleidoscope
 
 internal object EffectTuning {
   /**
-   * Gaussian sigma for the blur effect; higher = softer blur. Default
-   * matches the v0.1 hardcoded value. Custom setter clamps to a sane
-   * range; setting the property is the public mutation API.
+   * Gaussian sigma for the blur effect; higher = softer blur. Default 5;
+   * clamped to [0.5, 7] (the useful range before the linear-sampled kernel
+   * truncates and bands). Setting the property is the public mutation API.
    */
   @Volatile
-  var blurSigma: Float = 8f
+  var blurSigma: Float = 5f
     set(value) {
-      field = value.coerceIn(0.5f, 64f)
+      field = value.coerceIn(0.5f, 7f)
     }
 
   /**
@@ -37,21 +37,52 @@ internal object EffectTuning {
   /**
    * Mask smoothstep center for blur and background-image composites,
    * in [0, 1]; the threshold at which a pixel's raw confidence flips from
-   * "background" to "person". Default 0.5 reproduces the historical
-   * smoothstep centered on the confidence midpoint. Higher values reject
+   * "background" to "person". Default 0.7 (dialed in post-optimization).
+   * Higher values reject
    * low-confidence edges (chair backs, hair flyaway); lower values are
    * more inclusive. Clamped to a workable range below to keep the
    * smoothstep transition non-degenerate.
    */
   @Volatile
-  var maskThreshold: Float = 0.5f
+  var maskThreshold: Float = 0.7f
     set(value) {
       field = value.coerceIn(0.05f, 0.95f)
     }
 
+  /**
+   * Debug GPU timing. When true, the GLES effect factories log per-frame
+   * GPU-completion latency under the "Perf" logcat tag (read a frame late via
+   * the frame-pipeline fence). Off by default; toggled from JS via the Expo
+   * Module's setDebugTiming so a tester can capture ground-truth numbers on a
+   * device build without rebuilding. Mirrors the iOS timing flag being added
+   * in parallel.
+   */
+  @Volatile
+  var debugTiming: Boolean = false
+    set(value) {
+      field = value
+    }
+
+  /**
+   * Segmentation input short-side (px). The mask is produced from an input
+   * downscaled to this; lower = cheaper segmentation, softer mask edge.
+   * Default 384; clamped [128, 1080]. The floor matches iOS and web
+   * (EffectTuning.swift, tuning.ts) so the JS setSegmentationTargetShortSide
+   * contract is identical across platforms; MediaPipe resizes the input to the
+   * model's own 256x256 internally, so a smaller input still segments. Tuned
+   * live from JS via setSegmentationTargetShortSide.
+   */
+  @Volatile
+  var targetShortSide: Int = 384
+    set(value) {
+      field = value.coerceIn(128, 1080)
+    }
+
   fun reset() {
-    blurSigma = 8f
+    blurSigma = 5f
     maskHardness = 0.5f
-    maskThreshold = 0.5f
+    maskThreshold = 0.7f
+    debugTiming = false
+    targetShortSide = 384
   }
 }

package/android/src/main/java/com/simiancraft/kaleidoscope/KaleidoscopeModule.kt

@@ -36,6 +36,14 @@ class KaleidoscopeModule : Module() {
       EffectTuning.maskThreshold = value
     }
 
+    Function("setDebugTiming") { value: Boolean ->
+      EffectTuning.debugTiming = value
+    }
+
+    Function("setSegmentationTargetShortSide") { value: Int ->
+      EffectTuning.targetShortSide = value
+    }
+
     Function("resetEffectTuning") {
       EffectTuning.reset()
     }

package/android/src/main/java/com/simiancraft/kaleidoscope/Registration.kt

@@ -1,8 +1,8 @@
 // Frame-processor registration for Android. Called from
 // KaleidoscopeModule.OnCreate at Expo Module init time, before any track
 // requests an effect by name. The Context is needed by GPU effects so they
-// can read PNG assets for background-image and create RenderScript-style
-// resources where applicable.
+// can read bundled assets: the background-image WebP presets and the
+// selfie_segmenter.tflite model (loaded via SegmentationEngine).
 
 package com.simiancraft.kaleidoscope
 
@@ -10,30 +10,69 @@ import android.content.Context
 import com.oney.WebRTCModule.videoEffects.ProcessorProvider
 import com.simiancraft.kaleidoscope.effects.BackgroundImageFactory
 import com.simiancraft.kaleidoscope.effects.BlurFactory
-import com.simiancraft.kaleidoscope.effects.MirrorFactory
-import com.simiancraft.kaleidoscope.gpu.GpuEffectFactory
+import com.simiancraft.kaleidoscope.effects.TransformFactory
+import com.simiancraft.kaleidoscope.gpu.Orientation
 
 object Registration {
   @JvmStatic
   fun registerAll(context: Context) {
-    ProcessorProvider.addProcessor("mirror", MirrorFactory())
     ProcessorProvider.addProcessor("blur", BlurFactory(context))
 
+    // Geometric reorientation effects. flip-x is the corrected screen-horizontal
+    // mirror (replaces the old "mirror" CPU effect). The rotation correction
+    // lives entirely in Orientation.kt; each registration just names its op.
+    ProcessorProvider.addProcessor("flip-x", TransformFactory(Orientation.Op.FLIP_X))
+    ProcessorProvider.addProcessor("flip-y", TransformFactory(Orientation.Op.FLIP_Y))
+    ProcessorProvider.addProcessor("rotate-cw", TransformFactory(Orientation.Op.ROTATE_CW))
+    ProcessorProvider.addProcessor("rotate-ccw", TransformFactory(Orientation.Op.ROTATE_CCW))
+
     // Background-image variants — one factory per source preset. JS side
     // emits "background-image-{source}" so each preset gets its own
     // ProcessorProvider entry. Parameterized dispatch via uniforms lands
     // when we extend the upstream rn-webrtc API surface.
     ProcessorProvider.addProcessor(
-      "background-image-office-1",
-      BackgroundImageFactory(context, "office-1"),
+      "background-image-debug-resolutions",
+      BackgroundImageFactory(context, "debug-resolutions"),
     )
     ProcessorProvider.addProcessor(
-      "background-image-office-2",
-      BackgroundImageFactory(context, "office-2"),
+      "background-image-dark-office",
+      BackgroundImageFactory(context, "dark-office"),
+    )
+    ProcessorProvider.addProcessor(
+      "background-image-light-office",
+      BackgroundImageFactory(context, "light-office"),
+    )
+    ProcessorProvider.addProcessor(
+      "background-image-home-light",
+      BackgroundImageFactory(context, "home-light"),
+    )
+    ProcessorProvider.addProcessor(
+      "background-image-home-dark",
+      BackgroundImageFactory(context, "home-dark"),
+    )
+    ProcessorProvider.addProcessor(
+      "background-image-nature-light",
+      BackgroundImageFactory(context, "nature-light"),
+    )
+    ProcessorProvider.addProcessor(
+      "background-image-nature-dark",
+      BackgroundImageFactory(context, "nature-dark"),
+    )
+    ProcessorProvider.addProcessor(
+      "background-image-stylized-light",
+      BackgroundImageFactory(context, "stylized-light"),
+    )
+    ProcessorProvider.addProcessor(
+      "background-image-stylized-dark",
+      BackgroundImageFactory(context, "stylized-dark"),
+    )
+    ProcessorProvider.addProcessor(
+      "background-image-simiancraft-light",
+      BackgroundImageFactory(context, "simiancraft-light"),
+    )
+    ProcessorProvider.addProcessor(
+      "background-image-simiancraft-dark",
+      BackgroundImageFactory(context, "simiancraft-dark"),
     )
-
-    // Temporary architecture-proof passthrough hook from the GPU bring-up.
-    // Removed in the cleanup pass before v0.1 ships.
-    ProcessorProvider.addProcessor("gpu-passthrough", GpuEffectFactory())
   }
 }

package/android/src/main/java/com/simiancraft/kaleidoscope/effects/BackgroundImageFactory.kt

@@ -2,16 +2,16 @@
 //
 // Per frame:
 //   1. Render the input OES camera texture into a cached "original 2D" FBO.
-//   2. Lazy-load the named PNG asset (e.g. "office-1") from the library's
+//   2. Lazy-load the named WebP asset (e.g. "dark-office") from the library's
 //      android/src/main/assets/backgrounds/ on first frame; upload as a 2D
 //      GL texture; cache for subsequent frames.
-//   3. Produce a mask via Mask.produce (downsample, MLKit, upload).
+//   3. Produce a mask via Mask.produce (downsample, MediaPipe, upload).
 //   4. Composite original + image + mask into a fresh output texture via
 //      COMPOSITE_FRAG.
 //   5. Wrap the fresh output texture in a TextureBufferImpl and return a
 //      VideoFrame.
 //
-// Each registered name (e.g. "background-image-office-1") gets its own
+// Each registered name (e.g. "background-image-dark-office") gets its own
 // factory keyed by an asset name; multiple factories can coexist if the
 // consumer registers multiple variants.
 //
@@ -31,7 +31,9 @@ import com.oney.WebRTCModule.videoEffects.VideoFrameProcessorFactoryInterface
 import com.simiancraft.kaleidoscope.EffectTuning
 import com.simiancraft.kaleidoscope.gpu.Egl
 import com.simiancraft.kaleidoscope.gpu.Fbo
+import com.simiancraft.kaleidoscope.gpu.FramePipeline
 import com.simiancraft.kaleidoscope.gpu.GlDebug
+import com.simiancraft.kaleidoscope.gpu.Ingest
 import com.simiancraft.kaleidoscope.gpu.GlProgram
 import com.simiancraft.kaleidoscope.gpu.Shaders
 import com.simiancraft.kaleidoscope.segmentation.Mask
@@ -43,8 +45,9 @@ import org.webrtc.YuvConverter
 
 /**
  * @param context Used to read the PNG asset.
- * @param assetName Filename (without `.png`) under `assets/backgrounds/`.
- *                   E.g. "office-1" -> assets/backgrounds/office-1.png.
+ * @param assetName Filename (without `.webp`) under `assets/backgrounds/`.
+ *                   E.g. "dark-office" -> assets/backgrounds/dark-office.webp.
+ *                   BitmapFactory decodes WebP natively (supported since API 14).
  *
  * maskHardness is read per frame from com.simiancraft.kaleidoscope.EffectTuning
  * so JS callers can tune the smoothstep edge via the Expo Module's
@@ -62,6 +65,10 @@ private class BackgroundImageProcessor(
   private val context: Context,
   private val assetName: String,
 ) : VideoFrameProcessor {
+  // process() is only ever invoked on the single SurfaceTextureHelper capture
+  // thread (VideoEffectProcessor.onFrameCaptured), so this never actually
+  // contends. Retained as cheap uncontended insurance and as an explicit marker
+  // that the GL state below is single-threaded; it is NOT a cross-thread guard.
   private val lock = Any()
 
   private var oesToTwoD: GlProgram? = null
@@ -71,9 +78,13 @@ private class BackgroundImageProcessor(
   private var cachedWidth = 0
   private var cachedHeight = 0
 
-  private val mask = Mask()
+  private val mask = Mask(context)
   private var yuvConverter: YuvConverter? = null
 
+  // R3: one-frame GPU pipeline (see FramePipeline). Replaces the per-frame
+  // glFinish so the capture thread does not block on this frame's GPU work.
+  private val pipeline = FramePipeline()
+
   // Background image cached as a 2D GL texture; loaded lazily on the first
   // successful frame to ensure GL setup is ready. Aspect ratio captured at
   // load time so the composite shader can center-crop into the output.
@@ -116,13 +127,19 @@ private class BackgroundImageProcessor(
       )
       return null
     }
-    val width = inputBuffer.width
-    val height = inputBuffer.height
-    if (width <= 0 || height <= 0) {
-      Log.w(TAG, "Degenerate dims ${width}x${height}; forwarding.")
+    val bufW = inputBuffer.width
+    val bufH = inputBuffer.height
+    if (bufW <= 0 || bufH <= 0) {
+      Log.w(TAG, "Degenerate dims ${bufW}x${bufH}; forwarding.")
       return null
     }
 
+    // Ingest normalization: the OES->2D pass lands a DISPLAY-UPRIGHT frame, so
+    // the original FBO and output are sized in DISPLAY dims, the cover-fit runs
+    // against the display aspect, and the frame goes out rotation 0.
+    val width = Ingest.displayWidth(bufW, bufH, frame.rotation)
+    val height = Ingest.displayHeight(bufW, bufH, frame.rotation)
+
     GlDebug.check("bgImage entry")
     val saved = Egl.save()
     var outputTextureId = 0
@@ -145,7 +162,8 @@ private class BackgroundImageProcessor(
       origFbo.bind()
       oes.use()
       oes.setInt("uTex", 0)
-      val texMatrix = Egl.matrixToGl(inputBuffer.transformMatrix)
+      // Compose transformMatrix with the display rotation so the FBO lands upright.
+      val texMatrix = Ingest.composedTexMatrix(inputBuffer.transformMatrix, frame.rotation)
       GLES30.glUniformMatrix4fv(oes.uniformLocation("uTexMatrix"), 1, false, texMatrix, 0)
       GLES30.glDisable(GLES30.GL_DEPTH_TEST)
       GLES30.glDisable(GLES30.GL_BLEND)
@@ -208,8 +226,6 @@ private class BackgroundImageProcessor(
       GLES30.glDrawArrays(GLES30.GL_TRIANGLE_STRIP, 0, 4)
       GlDebug.check("bgImage composite pass")
 
-      GLES30.glFinish()
-
       GLES30.glFramebufferTexture2D(
         GLES30.GL_FRAMEBUFFER,
         GLES30.GL_COLOR_ATTACHMENT0,
@@ -222,28 +238,42 @@ private class BackgroundImageProcessor(
       outputFboHandle = 0
       GlDebug.check("bgImage output cleanup")
 
+      // R3: fence this frame and hand the previous GPU-complete frame off; the
+      // pipeline now owns outputTextureId, so zero the local handle.
+      val ready = pipeline.enqueue(
+        outputTextureId,
+        width,
+        height,
+        // Pixels are already display-upright after ingest; emit rotation 0.
+        0,
+        frame.timestampNs,
+        EffectTuning.debugTiming,
+        TAG,
+      )
+      outputTextureId = 0
+      ready ?: return null
+
       val yc = yuvConverter ?: run {
         val c = YuvConverter()
         yuvConverter = c
         c
       }
 
-      val capturedTextureId = outputTextureId
+      val readyTextureId = ready.textureId
       val outputBuffer = TextureBufferImpl(
-        width,
-        height,
+        ready.width,
+        ready.height,
         VideoFrame.TextureBuffer.Type.RGB,
-        capturedTextureId,
+        readyTextureId,
         Matrix(),
         textureHelper.handler,
         yc,
         Runnable {
-          GLES30.glDeleteTextures(1, intArrayOf(capturedTextureId), 0)
+          GLES30.glDeleteTextures(1, intArrayOf(readyTextureId), 0)
         },
       )
-      outputTextureId = 0
 
-      VideoFrame(outputBuffer, frame.rotation, frame.timestampNs)
+      VideoFrame(outputBuffer, ready.rotation, ready.timestampNs)
     } catch (t: Throwable) {
       if (outputTextureId != 0) {
         try {
@@ -288,15 +318,15 @@ private class BackgroundImageProcessor(
   private fun ensureBackgroundTexture() {
     if (backgroundTextureId != 0) return
     val bmp = try {
-      context.assets.open("backgrounds/$assetName.png").use { stream ->
+      context.assets.open("backgrounds/$assetName.webp").use { stream ->
         BitmapFactory.decodeStream(stream)
       }
     } catch (t: Throwable) {
-      Log.e(TAG, "failed to load asset backgrounds/$assetName.png", t)
+      Log.e(TAG, "failed to load asset backgrounds/$assetName.webp", t)
       return
     }
     if (bmp == null) {
-      Log.e(TAG, "BitmapFactory returned null for backgrounds/$assetName.png")
+      Log.e(TAG, "BitmapFactory returned null for backgrounds/$assetName.webp")
       return
     }
     // Pre-flip vertically so the texture lands in the shared convention