react-native-webrtc-kaleidoscope 0.0.1 → 1.1.0

package/README.md

@@ -1,12 +1,39 @@
+<p align="center">
+  <img src="./docs/kaleidoscope-logo.png" alt="react-native-webrtc-kaleidoscope logo" width="180" />
+</p>
+
 # react-native-webrtc-kaleidoscope
 
-> Live video effects for `react-native-webrtc`, packaged as a managed-Expo-friendly Expo Module. v0.1 ships **mirror** and **blur** on iOS, Android, and web.
+[![status: alpha](https://img.shields.io/badge/status-alpha-orange)](#status)
+[![npm version](https://img.shields.io/npm/v/react-native-webrtc-kaleidoscope?color=cb3837&logo=npm)](https://www.npmjs.com/package/react-native-webrtc-kaleidoscope)
+[![Types: included](https://img.shields.io/npm/types/react-native-webrtc-kaleidoscope?color=3178c6&logo=typescript)](https://www.npmjs.com/package/react-native-webrtc-kaleidoscope)
+[![CI](https://github.com/simiancraft/react-native-webrtc-kaleidoscope/actions/workflows/ci.yml/badge.svg)](https://github.com/simiancraft/react-native-webrtc-kaleidoscope/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](./LICENSE)
+
+> Live video effects for `react-native-webrtc`, packaged as a managed-Expo-friendly Expo Module.
 
 ## Status
 
-Pre-1.0. Active development. The bootstrap-and-ship plan in [`bootstrap-and-ship-v0-1.md`](./bootstrap-and-ship-v0-1.md) (root, while it exists) is the source of truth for v0.1.
+**Active development; not yet production-ready.** Published to npm at `1.0.0` (semantic-release cut the first tag at 1.0.0; the number reflects release automation, not a maturity claim). The npm presentation, marketing, and release-quality polish will come in a later pass; right now the README's job is to tell the truth about what works.
+
+### What works today
+
+- **Mirror** (horizontal flip).
+- **Blur** (background blur, person stays sharp).
+- **Background replacement** (composite a still PNG behind the segmented person; two bundled office presets, library-side asset pipeline).
+- **Runtime tuning** of the GLSL effects; see the [Use](#use) section.
+
+| Platform | Mirror | Blur | Background replacement | Notes |
+|---|---|---|---|---|
+| Web (Chrome / Edge) | ✓ | ✓ | ✓ | MediaStreamTrackProcessor + MediaPipe Selfie Segmentation (WASM, CDN) |
+| Android (API 24+) | ✓ | ✓ | ✓ | OpenGL ES 3.0 + MLKit Selfie Segmentation |
+| iOS (≥ 15) | ✓ | ✓ | ✓ | Metal + Vision (person segmentation), verified on device. Older A11 devices (iPhone X) run at a lower frame rate |
+| Safari / Firefox | — | — | — | No Insertable Streams; `applyVideoEffects` throws a typed error |
 
-The first usable release will be `v0.1.0`, shipping `mirror` and `blur` on all three platforms. Until then, the package is published as `0.1.0-alpha.x` for name reservation and integration testing.
+### Coming soon
+
+- **Procedural backgrounds** (animated shaders behind the person, not just still images). Same composite path; the only new piece is each effect's background producer.
+- A careful pass over the npm presentation, install docs, and demo polish before any "we recommend you use this" framing.
 
 ## Install
 
@@ -16,6 +43,34 @@ bun add react-native-webrtc react-native-webrtc-kaleidoscope
 
 `react-native-webrtc` is a peer dependency. Install it explicitly.
 
+### Using LiveKit?
+
+If your project uses `@livekit/react-native` it pulls in `@livekit/react-native-webrtc`, a fork of upstream `react-native-webrtc` that preserves the same `videoEffects` native classes and the `_setVideoEffects` JS API. Kaleidoscope works against either fork; the Android Gradle script picks whichever one your autolinking surfaced.
+
+```sh
+bun add @livekit/react-native @livekit/react-native-webrtc react-native-webrtc-kaleidoscope
+```
+
+Pick one fork. Installing both upstream `react-native-webrtc` and `@livekit/react-native-webrtc` in the same app will cause native class collisions; that's the consumer's problem to resolve.
+
+**Native wiring.** `@livekit/react-native` hands you a `LocalVideoTrack`; apply effects to its underlying `MediaStreamTrack`:
+
+```ts
+import { applyVideoEffects } from 'react-native-webrtc-kaleidoscope';
+
+applyVideoEffects(localCameraTrack.mediaStreamTrack, ['blur']);
+```
+
+**Web wiring.** On web, LiveKit owns the `RTCRtpSender`, so you cannot swap the track yourself; go through LiveKit's processor API instead. The opt-in `/livekit` subpath ships a ready-made processor (it needs `livekit-client`, which a LiveKit app already has):
+
+```ts
+import { KaleidoscopeProcessor } from 'react-native-webrtc-kaleidoscope/livekit';
+
+await localVideoTrack.setProcessor(new KaleidoscopeProcessor(['blur']), true);
+```
+
+The second argument shows the processed stream in your local preview. The processor tears down its Insertable-Streams pipeline on camera flip (`restart`) and unpublish (`destroy`), so repeated flips do not leak generators.
+
 ## Configure
 
 Add the config plugin to `app.config.ts`:
@@ -23,11 +78,13 @@ Add the config plugin to `app.config.ts`:
 ```ts
 export default {
   expo: {
-    plugins: ['react-native-webrtc', 'react-native-webrtc-kaleidoscope'],
+    plugins: ['react-native-webrtc-kaleidoscope'],
   },
 };
 ```
 
+(`react-native-webrtc` 124.x does not ship a config plugin upstream; do not list it in `plugins`. If you are on a fork that adds one, add it explicitly.)
+
 Then rebuild native code:
 
 ```sh
@@ -38,32 +95,77 @@ bunx expo prebuild
 
 ```ts
 import { mediaDevices } from 'react-native-webrtc';
-import { applyVideoEffects } from 'react-native-webrtc-kaleidoscope';
+import {
+  applyVideoEffects,
+  setBlurSigma,
+  setMaskHardness,
+  setMaskThreshold,
+} from 'react-native-webrtc-kaleidoscope';
 
 const stream = await mediaDevices.getUserMedia({ video: true });
 const [track] = stream.getVideoTracks();
 
-applyVideoEffects(track, ['blur']);   // background blur
-applyVideoEffects(track, ['mirror']); // horizontal flip
-applyVideoEffects(track, []);          // clear all effects
+applyVideoEffects(track, ['mirror']);
+applyVideoEffects(track, ['blur']);
+applyVideoEffects(track, [{ name: 'background-image', source: 'office-1' }]);
+applyVideoEffects(track, []); // clear all effects
+
+// Runtime tuning (effects pick up the new values on the next frame):
+setBlurSigma(25);        // Gaussian σ; clamped to [0.5, 64], default 8.
+setMaskHardness(0.2);    // smoothstep transition width; clamped to [0, 1]. 0 = soft halo, 1 = near-step. Default 0.5.
+setMaskThreshold(0.7);   // smoothstep center; clamped to [0.05, 0.95]. Higher rejects low-confidence pixels. Default 0.5.
 ```
 
 Effects chain in array order.
 
-## Platform support
+**Tuning note:** optimal values are platform-specific because each segmentation model (MediaPipe on web, MLKit on Android, Vision when iOS lands) produces a different confidence distribution. Working defaults on a typical well-lit scene:
 
-| Platform | Mirror | Blur | Backend |
+| Platform | Blur sigma | Mask hardness | Mask threshold |
 |---|---|---|---|
-| iOS ≥ 15 | ✓ | ✓ | Apple Vision + Core Image |
-| Android ≥ API 24 | ✓ | ✓ | MLKit Selfie Segmentation + RenderScript / RenderEffect |
-| Chrome / Edge | ✓ | ✓ | MediaStreamTrackProcessor + MediaPipe Selfie Segmentation (WASM) |
-| Safari | ✓ | ⚠ | No Insertable Streams; throws a typed error |
-| Firefox | ✓ | ⚠ | Same; capability-check before calling |
+| Web (MediaPipe) | 25 | 0.2 | 0.85 |
+| Android (MLKit) | 30 | 0.2 | 0.6 |
+
+The library ships neutral defaults (8, 0.5, 0.5) and consumers tune at runtime via the API above; whether to ship the dialed-in values as platform-specific defaults is an open question waiting on iOS data.
+
+## Web and native differences
+
+The API surface is the same across platforms, but the runtimes differ in ways worth knowing before you wire effects in:
+
+- **Effect parameters.** Web reads tuning from the global setters (`setBlurSigma`, `setMaskHardness`, `setMaskThreshold`) on the next frame. Native currently ignores per-call `EffectSpec` parameters such as `{ name: 'blur', sigma: 12 }`; tuning is global through the same setters. Per-call uniforms through the native registry are a follow-up.
+- **Background source.** `background-image.source` is a bundled preset name on native (the upstream `_setVideoEffects` registry is keyed by flat strings, not URIs), but on web it accepts either a preset name or an arbitrary image URL or data URI.
+- **Background presets ship as tree-shakeable files.** Two example backgrounds (`office-1`, `office-2`) are importable per preset: `import { office1 } from 'react-native-webrtc-kaleidoscope/backgrounds/office-1'`. Each preset is its own file behind its own subpath export, and the package sets `sideEffects: false`, so an unused preset is dropped by web bundlers — and, since Metro doesn't tree-shake, simply never imported on native. Web resolves the bundled WebP to a URL; native loads its own bundled copy by name. Web also still accepts an arbitrary image URL or data URI. See [`src/backgrounds/README.md`](./src/backgrounds/README.md).
+- **Segmentation model on web.** The web blur and background-image effects load MediaPipe Selfie Segmentation from the jsdelivr CDN (`cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation`) on first use. A strict Content-Security-Policy must allow that origin for `script-src`, `connect-src`, and the WASM fetch, and the effects do not work offline. Mirror needs no model.
+- **Browser support on web.** Effects use Insertable Streams (`MediaStreamTrackProcessor` and `MediaStreamTrackGenerator`), which ship in Chromium-based browsers; Safari and Firefox throw a typed capability error.
+
+## What this isn't
+
+- **Not a fork of `react-native-webrtc`.** A thin layer over its undocumented `_setVideoEffects` registry on native, and `MediaStreamTrackProcessor` on web. Install alongside `react-native-webrtc`.
+- **Not a managed cloud SaaS.** Effects run locally on the device; the track stays peer-to-peer. No service, no API key, no per-minute billing.
+- **Not a face-filter SDK.** Effects are background segmentation and frame transforms, not facial AR.
+- **Not a streaming protocol replacement.** The transformed track plugs into the consumer's existing `RTCPeerConnection` pipeline.
+
+## Architecture
+
+The codebase lives across four surfaces:
+
+- `src/` — JS facade and shared types. `applyVideoEffects(track, effects)` plus runtime tuning setters.
+- `src/web/` — WebGL2 pipeline. MediaPipe segmentation + GLSL composite. One shader file per stage in `src/web/shaders.ts`.
+- `android/` — OpenGL ES 3.0 pipeline. MLKit segmentation (async, worker-thread, last-known-mask cache) + GLSL composite. Shaders inline in `gpu/Shaders.kt` as `const val` strings.
+- `ios/` — Metal pipeline (Swift) with Vision person segmentation. The canonical GLSL in `shaders/` transpiles to Metal Shading Language via `scripts/build-shaders.ts`. Implemented and verified on device.
+
+The composite shader (`shaders/composite.frag`) is the same GLSL source for every effect category (blur, background-image, future procedural backgrounds). Per-effect difference is upstream of the composite: how the `uBackground` texture gets produced.
+
+See [`PATTERNS.md`](./PATTERNS.md) for the file-layout conventions, texture-orientation contract, and recipe for adding new effects, shaders, presets, or tunable parameters.
 
 ## Reference
 
-- [CONTRIBUTING.md](./CONTRIBUTING.md) — setup, scripts, commit conventions
-- [AGENTS.md](./AGENTS.md) — agent / contributor orientation
-- [SECURITY.md](./SECURITY.md) — security policy and reporting
-- [NOTICE.md](./NOTICE.md) — third-party attributions
-- Sibling projects: [chromonym](https://github.com/simiancraft/chromonym), [unitforge](https://github.com/simiancraft/unitforge) — same OSS-hygiene template
+- [CONTRIBUTING.md](./CONTRIBUTING.md): setup, scripts, commit conventions.
+- [AGENTS.md](./AGENTS.md): agent and contributor orientation.
+- [PATTERNS.md](./PATTERNS.md): codebase conventions and how-to-extend.
+- [SECURITY.md](./SECURITY.md): security policy and reporting.
+- [NOTICE.md](./NOTICE.md): third-party attributions.
+- Sibling projects: [chromonym](https://github.com/simiancraft/chromonym) and [unitforge](https://github.com/simiancraft/unitforge); same OSS-hygiene template.
+
+---
+
+MIT licensed. © 2026 Jesse Harlin / [Simiancraft](https://github.com/simiancraft).

package/android/build.gradle

@@ -0,0 +1,67 @@
+// Android build script for react-native-webrtc-kaleidoscope.
+
+apply plugin: 'com.android.library'
+apply plugin: 'kotlin-android'
+
+group = 'com.simiancraft.kaleidoscope'
+// Single source of truth for the version is package.json (the iOS podspec reads
+// it the same way), so the three platforms cannot drift apart.
+version = new groovy.json.JsonSlurper().parseText(file("../package.json").text).version
+
+def expoModulesCorePlugin = new File(project(":expo-modules-core").projectDir.absolutePath, "ExpoModulesCorePlugin.gradle")
+if (expoModulesCorePlugin.exists()) {
+  apply from: expoModulesCorePlugin
+  applyKotlinExpoModulesCorePlugin()
+}
+
+android {
+  namespace 'com.simiancraft.kaleidoscope'
+  compileSdkVersion safeExtGet('compileSdkVersion', 34)
+
+  defaultConfig {
+    minSdkVersion safeExtGet('minSdkVersion', 24)
+    targetSdkVersion safeExtGet('targetSdkVersion', 34)
+  }
+
+  compileOptions {
+    sourceCompatibility JavaVersion.VERSION_17
+    targetCompatibility JavaVersion.VERSION_17
+  }
+
+  kotlinOptions {
+    jvmTarget = '17'
+  }
+}
+
+dependencies {
+  implementation project(':expo-modules-core')
+
+  // react-native-webrtc is a peer dependency: link against its types
+  // (ProcessorProvider, VideoFrameProcessor, VideoFrameProcessorFactoryInterface,
+  // and the org.webrtc.* surface it bundles) at compile time only; the consumer
+  // app provides the runtime via autolinking.
+  //
+  // Two forks ship the same classes under `com.oney.WebRTCModule.videoEffects.*`:
+  //   - react-native-webrtc/react-native-webrtc (upstream)
+  //   - livekit/react-native-webrtc (LiveKit fork, required by @livekit/react-native)
+  // Pick whichever the consumer's autolinking surfaced; throw if neither is present.
+  def rnWebrtcCandidates = [':react-native-webrtc', ':livekit_react-native-webrtc']
+  def rnWebrtcProject = rnWebrtcCandidates.find { rootProject.findProject(it) != null }
+  if (rnWebrtcProject == null) {
+    throw new GradleException(
+      'react-native-webrtc-kaleidoscope: install either react-native-webrtc ' +
+      'or @livekit/react-native-webrtc as a peer dependency.'
+    )
+  }
+  compileOnly project(rnWebrtcProject)
+
+  // Person/background mask for the blur effect. The Google artifact ID is
+  // `segmentation-selfie`, not `selfie-segmentation` (the plan and earlier
+  // notes had it inverted; that is why the pin would not resolve).
+  implementation 'com.google.mlkit:segmentation-selfie:16.0.0-beta6'
+}
+
+// Helper: gracefully read root project ext properties with a default.
+def safeExtGet(prop, fallback) {
+  rootProject.ext.has(prop) ? rootProject.ext.get(prop) : fallback
+}

package/android/src/main/AndroidManifest.xml

@@ -0,0 +1,2 @@
+<?xml version="1.0" encoding="utf-8"?>
+<manifest xmlns:android="http://schemas.android.com/apk/res/android" />

package/android/src/main/java/com/simiancraft/kaleidoscope/EffectTuning.kt

@@ -0,0 +1,57 @@
+// Mutable runtime parameters for the GLSL effects on Android. The Expo
+// Module's setBlurSigma / setMaskHardness JS functions update these values;
+// per-frame processors read them on every frame so changes take effect
+// without re-registering processors.
+//
+// This is the parameter-passing side-channel that lets us tune effect
+// uniforms without touching the upstream react-native-webrtc registry,
+// which only accepts flat-string effect names. The web side mirrors this
+// shape in src/web/tuning.ts; iOS mirrors it in
+// ios/KaleidoscopeModule/EffectTuning.swift.
+
+package com.simiancraft.kaleidoscope
+
+internal object EffectTuning {
+  /**
+   * Gaussian sigma for the blur effect; higher = softer blur. Default
+   * matches the v0.1 hardcoded value. Custom setter clamps to a sane
+   * range; setting the property is the public mutation API.
+   */
+  @Volatile
+  var blurSigma: Float = 8f
+    set(value) {
+      field = value.coerceIn(0.5f, 64f)
+    }
+
+  /**
+   * Mask smoothstep hardness for blur and background-image composites,
+   * in [0, 1]; 0 = soft halo, 1 = near-step. Default reproduces the
+   * historical smoothstep(0.34, 0.66) edge.
+   */
+  @Volatile
+  var maskHardness: Float = 0.5f
+    set(value) {
+      field = value.coerceIn(0f, 1f)
+    }
+
+  /**
+   * Mask smoothstep center for blur and background-image composites,
+   * in [0, 1]; the threshold at which a pixel's raw confidence flips from
+   * "background" to "person". Default 0.5 reproduces the historical
+   * smoothstep centered on the confidence midpoint. Higher values reject
+   * low-confidence edges (chair backs, hair flyaway); lower values are
+   * more inclusive. Clamped to a workable range below to keep the
+   * smoothstep transition non-degenerate.
+   */
+  @Volatile
+  var maskThreshold: Float = 0.5f
+    set(value) {
+      field = value.coerceIn(0.05f, 0.95f)
+    }
+
+  fun reset() {
+    blurSigma = 8f
+    maskHardness = 0.5f
+    maskThreshold = 0.5f
+  }
+}

package/android/src/main/java/com/simiancraft/kaleidoscope/KaleidoscopeModule.kt

@@ -0,0 +1,43 @@
+// Expo Module entry point for react-native-webrtc-kaleidoscope on Android.
+// Calls Registration.registerAll(context) at module init so frame-processor
+// factories land in ProcessorProvider before any track requests them.
+//
+// Also exposes setBlurSigma / setMaskHardness / resetEffectTuning JS
+// functions that mutate com.simiancraft.kaleidoscope.EffectTuning at
+// runtime; the per-frame processors read those values each frame, so
+// changes take effect without re-registering or rebuilding processors.
+// This side-channels the upstream react-native-webrtc registry, which only
+// accepts flat-string effect names; spec parameters flow through here.
+
+package com.simiancraft.kaleidoscope
+
+import expo.modules.kotlin.modules.Module
+import expo.modules.kotlin.modules.ModuleDefinition
+
+class KaleidoscopeModule : Module() {
+  override fun definition() = ModuleDefinition {
+    Name("RnWebrtcKaleidoscope")
+
+    OnCreate {
+      val ctx = appContext.reactContext
+        ?: error("Kaleidoscope: no react context at OnCreate; cannot register Android effects")
+      Registration.registerAll(ctx)
+    }
+
+    Function("setBlurSigma") { value: Float ->
+      EffectTuning.blurSigma = value
+    }
+
+    Function("setMaskHardness") { value: Float ->
+      EffectTuning.maskHardness = value
+    }
+
+    Function("setMaskThreshold") { value: Float ->
+      EffectTuning.maskThreshold = value
+    }
+
+    Function("resetEffectTuning") {
+      EffectTuning.reset()
+    }
+  }
+}

package/android/src/main/java/com/simiancraft/kaleidoscope/Registration.kt

@@ -0,0 +1,39 @@
+// Frame-processor registration for Android. Called from
+// KaleidoscopeModule.OnCreate at Expo Module init time, before any track
+// requests an effect by name. The Context is needed by GPU effects so they
+// can read PNG assets for background-image and create RenderScript-style
+// resources where applicable.
+
+package com.simiancraft.kaleidoscope
+
+import android.content.Context
+import com.oney.WebRTCModule.videoEffects.ProcessorProvider
+import com.simiancraft.kaleidoscope.effects.BackgroundImageFactory
+import com.simiancraft.kaleidoscope.effects.BlurFactory
+import com.simiancraft.kaleidoscope.effects.MirrorFactory
+import com.simiancraft.kaleidoscope.gpu.GpuEffectFactory
+
+object Registration {
+  @JvmStatic
+  fun registerAll(context: Context) {
+    ProcessorProvider.addProcessor("mirror", MirrorFactory())
+    ProcessorProvider.addProcessor("blur", BlurFactory(context))
+
+    // Background-image variants — one factory per source preset. JS side
+    // emits "background-image-{source}" so each preset gets its own
+    // ProcessorProvider entry. Parameterized dispatch via uniforms lands
+    // when we extend the upstream rn-webrtc API surface.
+    ProcessorProvider.addProcessor(
+      "background-image-office-1",
+      BackgroundImageFactory(context, "office-1"),
+    )
+    ProcessorProvider.addProcessor(
+      "background-image-office-2",
+      BackgroundImageFactory(context, "office-2"),
+    )
+
+    // Temporary architecture-proof passthrough hook from the GPU bring-up.
+    // Removed in the cleanup pass before v0.1 ships.
+    ProcessorProvider.addProcessor("gpu-passthrough", GpuEffectFactory())
+  }
+}