react-native-webrtc-kaleidoscope 0.0.0 → 1.0.0

package/NOTICE.md

@@ -0,0 +1,35 @@
+# Notices and Attributions
+
+react-native-webrtc-kaleidoscope ships under the MIT license (see [LICENSE](./LICENSE)). This document tracks third-party sources, standards, and trademarks referenced by the library and the effects it ships.
+
+## Runtime peer dependency
+
+- **react-native-webrtc** — MIT license. <https://github.com/react-native-webrtc/react-native-webrtc>. The `track._setVideoEffects(...)` JS surface, the Android `ProcessorProvider` registry, and the iOS `RTCVideoFrameProcessor` protocol all originate upstream. This package is a consumer of, not a fork of, that codebase.
+
+## Native segmentation backends (used by the `blur` effect)
+
+- **Apple Vision** (iOS) — system framework. `VNGeneratePersonSegmentationRequest`. Subject to the Apple SDK License Agreement.
+- **MLKit Selfie Segmentation** (Android) — Apache License 2.0. Distributed by Google as a managed AAR. <https://developers.google.com/ml-kit/vision/selfie-segmentation>.
+- **MediaPipe Selfie Segmentation** (web) — Apache License 2.0. Loaded as an `optionalDependency` so native consumers do not bundle the WASM payload. <https://github.com/google-ai-edge/mediapipe>.
+
+## Algorithmic and architectural references
+
+- **mrousavy/FaceBlurApp** — segmentation + GPU composite at 60–120 FPS. Different plumbing (camera-preview pipeline, not a WebRTC track), but the algorithmic shape of "mask + blurred copy + composite" is the same.
+- **Volcomix/virtual-background** — TFLite WASM reference for the web blur composite.
+
+## Trademarks
+
+- "Apple", "Vision", "Core Image", and related Apple marks are trademarks of Apple Inc. Used here as nominative references only.
+- "Google", "MLKit", and "MediaPipe" are trademarks of Google LLC. Used here as nominative references only.
+- "WebRTC" is a project of the W3C and IETF.
+
+This package is not affiliated with, endorsed by, or certified by Apple, Google, the W3C, or the `react-native-webrtc` project.
+
+## Forward-looking discipline
+
+Effects added after v0.1 that depend on new third-party code or registered marks must:
+
+1. Cite the source (with a stable URL and version) in the effect's source code.
+2. Add an attribution section to this document.
+3. Disclaim affiliation, endorsement, or certification where appropriate.
+4. Treat trademarked names as **nominative references** only.

package/README.md

@@ -1,7 +1,144 @@
+<p align="center">
+  <img src="./docs/kaleidoscope-logo.png" alt="react-native-webrtc-kaleidoscope logo" width="180" />
+</p>
+
 # react-native-webrtc-kaleidoscope
 
-Live video effects (mirror, blur, more) for `react-native-webrtc`, packaged as a managed-Expo-friendly Expo Module.
+[![status: alpha](https://img.shields.io/badge/status-alpha-orange)](#status)
+[![npm version](https://img.shields.io/npm/v/react-native-webrtc-kaleidoscope?color=cb3837&logo=npm)](https://www.npmjs.com/package/react-native-webrtc-kaleidoscope)
+[![Types: included](https://img.shields.io/npm/types/react-native-webrtc-kaleidoscope?color=3178c6&logo=typescript)](https://www.npmjs.com/package/react-native-webrtc-kaleidoscope)
+[![CI](https://github.com/simiancraft/react-native-webrtc-kaleidoscope/actions/workflows/ci.yml/badge.svg)](https://github.com/simiancraft/react-native-webrtc-kaleidoscope/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](./LICENSE)
+
+> Live video effects for `react-native-webrtc`, packaged as a managed-Expo-friendly Expo Module.
+
+## Status
+
+**Active development; not yet production-ready.** Published to npm as `0.1.0-alpha.x` for name reservation and integration testing. The npm presentation, marketing, and release-quality polish will come in a later pass; right now the README's job is to tell the truth about what works.
+
+### What works today
+
+- **Mirror** (horizontal flip).
+- **Blur** (background blur, person stays sharp).
+- **Background replacement** (composite a still PNG behind the segmented person; two bundled office presets, library-side asset pipeline).
+- **Runtime tuning** of the GLSL effects; see the [Use](#use) section.
+
+| Platform | Mirror | Blur | Background replacement | Notes |
+|---|---|---|---|---|
+| Web (Chrome / Edge) | ✓ | ✓ | ✓ | MediaStreamTrackProcessor + MediaPipe Selfie Segmentation (WASM, CDN) |
+| Android (API 24+) | ✓ | ✓ | ✓ | OpenGL ES 3.0 + MLKit Selfie Segmentation |
+| iOS (≥ 15) | — | — | — | Coming soon; transpilation pipeline in place, host implementation pending |
+| Safari / Firefox | — | — | — | No Insertable Streams; `applyVideoEffects` throws a typed error |
+
+### Coming soon
+
+- **iOS support**, via the canonical GLSL transpiled to Metal Shading Language (`glslangValidator` → `spirv-cross --msl`). The pipeline is in the repo at `scripts/transpile-shaders.ts` and the GLSL canonical source lives in `shaders/`. The Swift host code that loads the metallib and runs the Metal pipeline is the next chunk of work.
+- **Procedural backgrounds** (animated shaders behind the person, not just still images). Same composite path; the only new piece is each effect's background producer.
+- A careful pass over the npm presentation, install docs, and demo polish before any "we recommend you use this" framing.
+
+## Install
+
+```sh
+bun add react-native-webrtc react-native-webrtc-kaleidoscope
+```
+
+`react-native-webrtc` is a peer dependency. Install it explicitly.
+
+### Using LiveKit?
+
+If your project uses `@livekit/react-native` it pulls in `@livekit/react-native-webrtc`, a fork of upstream `react-native-webrtc` that preserves the same `videoEffects` native classes and the `_setVideoEffects` JS API. Kaleidoscope works against either fork; the Android Gradle script picks whichever one your autolinking surfaced.
+
+```sh
+bun add @livekit/react-native @livekit/react-native-webrtc react-native-webrtc-kaleidoscope
+```
+
+Pick one fork. Installing both upstream `react-native-webrtc` and `@livekit/react-native-webrtc` in the same app will cause native class collisions; that's the consumer's problem to resolve.
+
+## Configure
+
+Add the config plugin to `app.config.ts`:
+
+```ts
+export default {
+  expo: {
+    plugins: ['react-native-webrtc-kaleidoscope'],
+  },
+};
+```
+
+(`react-native-webrtc` 124.x does not ship a config plugin upstream; do not list it in `plugins`. If you are on a fork that adds one, add it explicitly.)
+
+Then rebuild native code:
+
+```sh
+bunx expo prebuild
+```
+
+## Use
+
+```ts
+import { mediaDevices } from 'react-native-webrtc';
+import {
+  applyVideoEffects,
+  setBlurSigma,
+  setMaskHardness,
+  setMaskThreshold,
+} from 'react-native-webrtc-kaleidoscope';
+
+const stream = await mediaDevices.getUserMedia({ video: true });
+const [track] = stream.getVideoTracks();
+
+applyVideoEffects(track, ['mirror']);
+applyVideoEffects(track, ['blur']);
+applyVideoEffects(track, [{ name: 'background-image', source: 'office-1' }]);
+applyVideoEffects(track, []); // clear all effects
+
+// Runtime tuning (effects pick up the new values on the next frame):
+setBlurSigma(25);        // Gaussian σ; clamped to [0.5, 64], default 8.
+setMaskHardness(0.2);    // smoothstep transition width; clamped to [0, 1]. 0 = soft halo, 1 = near-step. Default 0.5.
+setMaskThreshold(0.7);   // smoothstep center; clamped to [0.05, 0.95]. Higher rejects low-confidence pixels. Default 0.5.
+```
+
+Effects chain in array order.
+
+**Tuning note:** optimal values are platform-specific because each segmentation model (MediaPipe on web, MLKit on Android, Vision when iOS lands) produces a different confidence distribution. Working defaults on a typical well-lit scene:
+
+| Platform | Blur sigma | Mask hardness | Mask threshold |
+|---|---|---|---|
+| Web (MediaPipe) | 25 | 0.2 | 0.85 |
+| Android (MLKit) | 30 | 0.2 | 0.6 |
+
+The library ships neutral defaults (8, 0.5, 0.5) and consumers tune at runtime via the API above; whether to ship the dialed-in values as platform-specific defaults is an open question waiting on iOS data.
+
+## What this isn't
+
+- **Not a fork of `react-native-webrtc`.** A thin layer over its undocumented `_setVideoEffects` registry on native, and `MediaStreamTrackProcessor` on web. Install alongside `react-native-webrtc`.
+- **Not a managed cloud SaaS.** Effects run locally on the device; the track stays peer-to-peer. No service, no API key, no per-minute billing.
+- **Not a face-filter SDK.** Effects are background segmentation and frame transforms, not facial AR.
+- **Not a streaming protocol replacement.** The transformed track plugs into the consumer's existing `RTCPeerConnection` pipeline.
+
+## Architecture
+
+The codebase lives across four surfaces:
+
+- `src/` — JS facade and shared types. `applyVideoEffects(track, effects)` plus runtime tuning setters.
+- `src/web/` — WebGL2 pipeline. MediaPipe segmentation + GLSL composite. One shader file per stage in `src/web/shaders.ts`.
+- `android/` — OpenGL ES 3.0 pipeline. MLKit segmentation (async, worker-thread, last-known-mask cache) + GLSL composite. Shaders inline in `gpu/Shaders.kt` as `const val` strings.
+- `ios/` — Scaffold only; the canonical GLSL in `shaders/` will transpile to Metal Shading Language for the iOS path.
+
+The composite shader (`shaders/composite.frag`) is the same GLSL source for every effect category (blur, background-image, future procedural backgrounds). Per-effect difference is upstream of the composite: how the `uBackground` texture gets produced.
+
+See [`PATTERNS.md`](./PATTERNS.md) for the file-layout conventions, texture-orientation contract, and recipe for adding new effects, shaders, presets, or tunable parameters.
+
+## Reference
+
+- [CONTRIBUTING.md](./CONTRIBUTING.md): setup, scripts, commit conventions.
+- [AGENTS.md](./AGENTS.md): agent and contributor orientation.
+- [PATTERNS.md](./PATTERNS.md): codebase conventions and how-to-extend.
+- [SECURITY.md](./SECURITY.md): security policy and reporting.
+- [NOTICE.md](./NOTICE.md): third-party attributions.
+- Sibling projects: [chromonym](https://github.com/simiancraft/chromonym) and [unitforge](https://github.com/simiancraft/unitforge); same OSS-hygiene template.
 
-**This is a placeholder.** The package name is reserved while active development is in progress. The first usable release will be `v0.1` shipping the `mirror` and `blur` effects on iOS, Android, and web.
+---
 
-Source: <https://github.com/simiancraft/react-native-webrtc-kaleidoscope>
+MIT licensed. © 2026 Jesse Harlin / [Simiancraft](https://github.com/simiancraft).

package/android/build.gradle

@@ -0,0 +1,65 @@
+// Android build script for react-native-webrtc-kaleidoscope.
+
+apply plugin: 'com.android.library'
+apply plugin: 'kotlin-android'
+
+group = 'com.simiancraft.kaleidoscope'
+version = '0.1.0-alpha.1'
+
+def expoModulesCorePlugin = new File(project(":expo-modules-core").projectDir.absolutePath, "ExpoModulesCorePlugin.gradle")
+if (expoModulesCorePlugin.exists()) {
+  apply from: expoModulesCorePlugin
+  applyKotlinExpoModulesCorePlugin()
+}
+
+android {
+  namespace 'com.simiancraft.kaleidoscope'
+  compileSdkVersion safeExtGet('compileSdkVersion', 34)
+
+  defaultConfig {
+    minSdkVersion safeExtGet('minSdkVersion', 24)
+    targetSdkVersion safeExtGet('targetSdkVersion', 34)
+  }
+
+  compileOptions {
+    sourceCompatibility JavaVersion.VERSION_17
+    targetCompatibility JavaVersion.VERSION_17
+  }
+
+  kotlinOptions {
+    jvmTarget = '17'
+  }
+}
+
+dependencies {
+  implementation project(':expo-modules-core')
+
+  // react-native-webrtc is a peer dependency: link against its types
+  // (ProcessorProvider, VideoFrameProcessor, VideoFrameProcessorFactoryInterface,
+  // and the org.webrtc.* surface it bundles) at compile time only; the consumer
+  // app provides the runtime via autolinking.
+  //
+  // Two forks ship the same classes under `com.oney.WebRTCModule.videoEffects.*`:
+  //   - react-native-webrtc/react-native-webrtc (upstream)
+  //   - livekit/react-native-webrtc (LiveKit fork, required by @livekit/react-native)
+  // Pick whichever the consumer's autolinking surfaced; throw if neither is present.
+  def rnWebrtcCandidates = [':react-native-webrtc', ':livekit_react-native-webrtc']
+  def rnWebrtcProject = rnWebrtcCandidates.find { rootProject.findProject(it) != null }
+  if (rnWebrtcProject == null) {
+    throw new GradleException(
+      'react-native-webrtc-kaleidoscope: install either react-native-webrtc ' +
+      'or @livekit/react-native-webrtc as a peer dependency.'
+    )
+  }
+  compileOnly project(rnWebrtcProject)
+
+  // Person/background mask for the blur effect. The Google artifact ID is
+  // `segmentation-selfie`, not `selfie-segmentation` (the plan and earlier
+  // notes had it inverted; that is why the pin would not resolve).
+  implementation 'com.google.mlkit:segmentation-selfie:16.0.0-beta6'
+}
+
+// Helper: gracefully read root project ext properties with a default.
+def safeExtGet(prop, fallback) {
+  rootProject.ext.has(prop) ? rootProject.ext.get(prop) : fallback
+}

package/android/src/main/AndroidManifest.xml

@@ -0,0 +1,2 @@
+<?xml version="1.0" encoding="utf-8"?>
+<manifest xmlns:android="http://schemas.android.com/apk/res/android" />

package/android/src/main/java/com/simiancraft/kaleidoscope/EffectTuning.kt

@@ -0,0 +1,57 @@
+// Mutable runtime parameters for the GLSL effects on Android. The Expo
+// Module's setBlurSigma / setMaskHardness JS functions update these values;
+// per-frame processors read them on every frame so changes take effect
+// without re-registering processors.
+//
+// This is the parameter-passing side-channel that lets us tune effect
+// uniforms without touching the upstream react-native-webrtc registry,
+// which only accepts flat-string effect names. The web side mirrors this
+// shape in src/web/tuning.ts; iOS mirrors it in
+// ios/KaleidoscopeModule/EffectTuning.swift.
+
+package com.simiancraft.kaleidoscope
+
+internal object EffectTuning {
+  /**
+   * Gaussian sigma for the blur effect; higher = softer blur. Default
+   * matches the v0.1 hardcoded value. Custom setter clamps to a sane
+   * range; setting the property is the public mutation API.
+   */
+  @Volatile
+  var blurSigma: Float = 8f
+    set(value) {
+      field = value.coerceIn(0.5f, 64f)
+    }
+
+  /**
+   * Mask smoothstep hardness for blur and background-image composites,
+   * in [0, 1]; 0 = soft halo, 1 = near-step. Default reproduces the
+   * historical smoothstep(0.34, 0.66) edge.
+   */
+  @Volatile
+  var maskHardness: Float = 0.5f
+    set(value) {
+      field = value.coerceIn(0f, 1f)
+    }
+
+  /**
+   * Mask smoothstep center for blur and background-image composites,
+   * in [0, 1]; the threshold at which a pixel's raw confidence flips from
+   * "background" to "person". Default 0.5 reproduces the historical
+   * smoothstep centered on the confidence midpoint. Higher values reject
+   * low-confidence edges (chair backs, hair flyaway); lower values are
+   * more inclusive. Clamped to a workable range below to keep the
+   * smoothstep transition non-degenerate.
+   */
+  @Volatile
+  var maskThreshold: Float = 0.5f
+    set(value) {
+      field = value.coerceIn(0.05f, 0.95f)
+    }
+
+  fun reset() {
+    blurSigma = 8f
+    maskHardness = 0.5f
+    maskThreshold = 0.5f
+  }
+}

package/android/src/main/java/com/simiancraft/kaleidoscope/KaleidoscopeModule.kt

@@ -0,0 +1,43 @@
+// Expo Module entry point for react-native-webrtc-kaleidoscope on Android.
+// Calls Registration.registerAll(context) at module init so frame-processor
+// factories land in ProcessorProvider before any track requests them.
+//
+// Also exposes setBlurSigma / setMaskHardness / resetEffectTuning JS
+// functions that mutate com.simiancraft.kaleidoscope.EffectTuning at
+// runtime; the per-frame processors read those values each frame, so
+// changes take effect without re-registering or rebuilding processors.
+// This side-channels the upstream react-native-webrtc registry, which only
+// accepts flat-string effect names; spec parameters flow through here.
+
+package com.simiancraft.kaleidoscope
+
+import expo.modules.kotlin.modules.Module
+import expo.modules.kotlin.modules.ModuleDefinition
+
+class KaleidoscopeModule : Module() {
+  override fun definition() = ModuleDefinition {
+    Name("RnWebrtcKaleidoscope")
+
+    OnCreate {
+      val ctx = appContext.reactContext
+        ?: error("Kaleidoscope: no react context at OnCreate; cannot register Android effects")
+      Registration.registerAll(ctx)
+    }
+
+    Function("setBlurSigma") { value: Float ->
+      EffectTuning.blurSigma = value
+    }
+
+    Function("setMaskHardness") { value: Float ->
+      EffectTuning.maskHardness = value
+    }
+
+    Function("setMaskThreshold") { value: Float ->
+      EffectTuning.maskThreshold = value
+    }
+
+    Function("resetEffectTuning") {
+      EffectTuning.reset()
+    }
+  }
+}

package/android/src/main/java/com/simiancraft/kaleidoscope/Registration.kt

@@ -0,0 +1,39 @@
+// Frame-processor registration for Android. Called from
+// KaleidoscopeModule.OnCreate at Expo Module init time, before any track
+// requests an effect by name. The Context is needed by GPU effects so they
+// can read PNG assets for background-image and create RenderScript-style
+// resources where applicable.
+
+package com.simiancraft.kaleidoscope
+
+import android.content.Context
+import com.oney.WebRTCModule.videoEffects.ProcessorProvider
+import com.simiancraft.kaleidoscope.effects.BackgroundImageFactory
+import com.simiancraft.kaleidoscope.effects.BlurFactory
+import com.simiancraft.kaleidoscope.effects.MirrorFactory
+import com.simiancraft.kaleidoscope.gpu.GpuEffectFactory
+
+object Registration {
+  @JvmStatic
+  fun registerAll(context: Context) {
+    ProcessorProvider.addProcessor("mirror", MirrorFactory())
+    ProcessorProvider.addProcessor("blur", BlurFactory(context))
+
+    // Background-image variants — one factory per source preset. JS side
+    // emits "background-image-{source}" so each preset gets its own
+    // ProcessorProvider entry. Parameterized dispatch via uniforms lands
+    // when we extend the upstream rn-webrtc API surface.
+    ProcessorProvider.addProcessor(
+      "background-image-office-1",
+      BackgroundImageFactory(context, "office-1"),
+    )
+    ProcessorProvider.addProcessor(
+      "background-image-office-2",
+      BackgroundImageFactory(context, "office-2"),
+    )
+
+    // Architecture-proof hook from PLAN.md Commit 3. Removed in the cleanup
+    // pass before v0.1 ships.
+    ProcessorProvider.addProcessor("gpu-passthrough", GpuEffectFactory())
+  }
+}