react-native-webrtc-kaleidoscope 1.1.0 → 2.1.0

package/NOTICE.md

@@ -4,23 +4,34 @@ react-native-webrtc-kaleidoscope ships under the MIT license (see [LICENSE](./LI
 
 ## Runtime peer dependency
 
-- **react-native-webrtc** — MIT license. <https://github.com/react-native-webrtc/react-native-webrtc>. The `track._setVideoEffects(...)` JS surface, the Android `ProcessorProvider` registry, and the iOS `RTCVideoFrameProcessor` protocol all originate upstream. This package is a consumer of, not a fork of, that codebase.
+- **react-native-webrtc**: MIT license. <https://github.com/react-native-webrtc/react-native-webrtc>. The `track._setVideoEffects(...)` JS surface, the Android `ProcessorProvider` registry, and the iOS `RTCVideoFrameProcessor` protocol all originate upstream. This package is a consumer of, not a fork of, that codebase.
 
-## Native segmentation backends (used by the `blur` effect)
+## Segmentation backends (used by the segmentation mask that drives the `blur` and image layers)
 
-- **Apple Vision** (iOS) — system framework. `VNGeneratePersonSegmentationRequest`. Subject to the Apple SDK License Agreement.
-- **MLKit Selfie Segmentation** (Android) — Apache License 2.0. Distributed by Google as a managed AAR. <https://developers.google.com/ml-kit/vision/selfie-segmentation>.
-- **MediaPipe Selfie Segmentation** (web) — Apache License 2.0. Loaded as an `optionalDependency` so native consumers do not bundle the WASM payload. <https://github.com/google-ai-edge/mediapipe>.
+All three platforms run Google's MediaPipe selfie segmentation. The native targets use the MediaPipe Tasks Image Segmenter with a model bundled in this package; the web target uses the legacy MediaPipe Selfie Segmentation Solution loaded at runtime from a CDN.
+
+- **MediaPipe Tasks Vision** (Android): Apache License 2.0. The `com.google.mediapipe:tasks-vision` AAR, `ImageSegmenter` API. <https://github.com/google-ai-edge/mediapipe>.
+- **MediaPipe Tasks Vision** (iOS): Apache License 2.0. The `MediaPipeTasksVision` CocoaPod, `ImageSegmenter` API. <https://github.com/google-ai-edge/mediapipe>.
+- **MediaPipe Selfie Segmentation** (web): Apache License 2.0. The `@mediapipe/selfie_segmentation` Solution, loaded at runtime from `cdn.jsdelivr.net` (not bundled in this package). <https://github.com/google-ai-edge/mediapipe>.
+
+### Bundled model
+
+The native targets redistribute a TensorFlow Lite model inside the published package; it ships in `android/src/main/assets/selfie_segmenter.tflite` and in the iOS `Kaleidoscope.bundle`.
+
+- **MediaPipe SelfieSegmenter** (`selfie_segmenter.tflite`): distributed by Google as part of MediaPipe Solutions.
+  - Source: <https://storage.googleapis.com/mediapipe-models/image_segmenter/selfie_segmenter/float16/latest/selfie_segmenter.tflite>
+  - Model card: <https://storage.googleapis.com/mediapipe-assets/Model%20Card%20MediaPipe%20Selfie%20Segmentation.pdf>
+  - The MediaPipe project and SDK are published by Google under the Apache License 2.0. The `selfie_segmenter.tflite` weights are redistributed here as published by Google as part of MediaPipe Solutions; Google does not attach a separate license to the hosted model file, so its use is governed by the model card above. The web target does not bundle this file; it fetches its model from the CDN at runtime.
 
 ## Algorithmic and architectural references
 
-- **mrousavy/FaceBlurApp** — segmentation + GPU composite at 60–120 FPS. Different plumbing (camera-preview pipeline, not a WebRTC track), but the algorithmic shape of "mask + blurred copy + composite" is the same.
-- **Volcomix/virtual-background** — TFLite WASM reference for the web blur composite.
+- **mrousavy/FaceBlurApp**: segmentation + GPU composite at 60–120 FPS. Different plumbing (camera-preview pipeline, not a WebRTC track), but the algorithmic shape of "mask + blurred copy + composite" is the same.
+- **Volcomix/virtual-background**: TFLite WASM reference for the web blur composite.
 
 ## Trademarks
 
-- "Apple", "Vision", "Core Image", and related Apple marks are trademarks of Apple Inc. Used here as nominative references only.
-- "Google", "MLKit", and "MediaPipe" are trademarks of Google LLC. Used here as nominative references only.
+- "Apple", "Core Image", "Metal", and related Apple marks are trademarks of Apple Inc. Used here as nominative references only.
+- "Google", "MediaPipe", and "TensorFlow Lite" are trademarks of Google LLC. Used here as nominative references only.
 - "WebRTC" is a project of the W3C and IETF.
 
 This package is not affiliated with, endorsed by, or certified by Apple, Google, the W3C, or the `react-native-webrtc` project.

package/README.md

@@ -8,31 +8,40 @@
 [![npm version](https://img.shields.io/npm/v/react-native-webrtc-kaleidoscope?color=cb3837&logo=npm)](https://www.npmjs.com/package/react-native-webrtc-kaleidoscope)
 [![Types: included](https://img.shields.io/npm/types/react-native-webrtc-kaleidoscope?color=3178c6&logo=typescript)](https://www.npmjs.com/package/react-native-webrtc-kaleidoscope)
 [![CI](https://github.com/simiancraft/react-native-webrtc-kaleidoscope/actions/workflows/ci.yml/badge.svg)](https://github.com/simiancraft/react-native-webrtc-kaleidoscope/actions/workflows/ci.yml)
+[![codecov](https://codecov.io/gh/simiancraft/react-native-webrtc-kaleidoscope/graph/badge.svg)](https://codecov.io/gh/simiancraft/react-native-webrtc-kaleidoscope)
+[![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/simiancraft/react-native-webrtc-kaleidoscope/badge)](https://securityscorecards.dev/viewer/?uri=github.com/simiancraft/react-native-webrtc-kaleidoscope)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](./LICENSE)
 
-> Live video effects for `react-native-webrtc`, packaged as a managed-Expo-friendly Expo Module.
+<p align="center">
+  <code>kaleidoscope</code> &nbsp;•&nbsp; <code>transform</code> &nbsp;•&nbsp; <code>mask</code>
+</p>
+
+> Creative, shader-based video effects for React Native video calls: blur or replace a person's background in a teleconference. Works with `react-native-webrtc` and LiveKit, managed-Expo-friendly.
+
+**Bind a track once, then drive it with three verbs.** `kaleidoscope` swaps the background effect (blur, a still image, or a procedural shader), `transform` reorients the frame, and `mask` tunes the segmentation edge. The effects you can command are a typed preset book you declare in your own project; at `expo prebuild`, only the assets you actually reference ship in your native bundle.
 
 ## Status
 
-**Active development; not yet production-ready.** Published to npm at `1.0.0` (semantic-release cut the first tag at 1.0.0; the number reflects release automation, not a maturity claim). The npm presentation, marketing, and release-quality polish will come in a later pass; right now the README's job is to tell the truth about what works.
+**Active development; not yet production-ready.** Published to npm (the badge shows the current version, which tracks release automation rather than maturity). Presentation and install-doc polish come later; for now the README documents what works.
 
 ### What works today
 
-- **Mirror** (horizontal flip).
-- **Blur** (background blur, person stays sharp).
-- **Background replacement** (composite a still PNG behind the segmented person; two bundled office presets, library-side asset pipeline).
-- **Runtime tuning** of the GLSL effects; see the [Use](#use) section.
+- **`kaleidoscope`**: the art axis. Blur, a bundled background image (ten themed backgrounds plus a calibration grid, or your own assets), or a procedural shader, commanded by preset name. The generative shaders that ship are `plasma`, `clouds`, `godrays`, `fireflies`, `nebula`, `simianlights`, `anamorphic-lensflare`, `light-beams-and-motes`, and `corporate-blobs`.
+- **`transform`**: absolute flips and rotation (snapped to 90°), reapplied as full state each call.
+- **`mask`**: the segmentation edge (hardness, threshold) shared by every art effect.
+- **Tree-shaking**: declare a preset book; only the assets you reference ship in your native bundle (web tree-shakes by import).
+- **Drop-in UI (optional)**: a preset-driven `PresetBookMenu` (the menu) under `react-native-webrtc-kaleidoscope/preset-book-menu`, plus a headless live-editor kit (`PresetControlPanel`, mask and transform controls, a theme provider) under `react-native-webrtc-kaleidoscope/preset-control-panel`. Controlled and NativeWind-ready. See [Drop-in UI](#drop-in-ui-optional).
 
-| Platform | Mirror | Blur | Background replacement | Notes |
+| Platform | Transform | Blur | Background replacement | Notes |
 |---|---|---|---|---|
 | Web (Chrome / Edge) | ✓ | ✓ | ✓ | MediaStreamTrackProcessor + MediaPipe Selfie Segmentation (WASM, CDN) |
-| Android (API 24+) | ✓ | ✓ | ✓ | OpenGL ES 3.0 + MLKit Selfie Segmentation |
-| iOS (≥ 15) | ✓ | ✓ | ✓ | Metal + Vision (person segmentation), verified on device. Older A11 devices (iPhone X) run at a lower frame rate |
-| Safari / Firefox | — | — | — | No Insertable Streams; `applyVideoEffects` throws a typed error |
+| Android (API 24+) | ✓ | ✓ | ✓ | OpenGL ES 3.0 + MediaPipe Selfie Segmentation (Tasks) |
+| iOS (≥ 15) | ✓ | ✓ | ✓ | Metal + MediaPipe Selfie Segmentation (Tasks), verified on device. Older A11 devices (iPhone X) run at a lower frame rate |
+| Safari / Firefox | n/a | n/a | n/a | No Insertable Streams; the effects throw a clear capability error |
 
 ### Coming soon
 
-- **Procedural backgrounds** (animated shaders behind the person, not just still images). Same composite path; the only new piece is each effect's background producer.
+- **Animated image backgrounds**: a bundled still that moves (beyond the procedural shaders, which already render behind the person today). Same composite path; the new piece is a per-effect background producer for animated images.
 - A careful pass over the npm presentation, install docs, and demo polish before any "we recommend you use this" framing.
 
 ## Install
@@ -53,12 +62,14 @@ bun add @livekit/react-native @livekit/react-native-webrtc react-native-webrtc-k
 
 Pick one fork. Installing both upstream `react-native-webrtc` and `@livekit/react-native-webrtc` in the same app will cause native class collisions; that's the consumer's problem to resolve.
 
-**Native wiring.** `@livekit/react-native` hands you a `LocalVideoTrack`; apply effects to its underlying `MediaStreamTrack`:
+**Native wiring.** `@livekit/react-native` hands you a `LocalVideoTrack`; bind effects to its underlying `MediaStreamTrack`:
 
 ```ts
-import { applyVideoEffects } from 'react-native-webrtc-kaleidoscope';
+import { bindKaleidoscope } from 'react-native-webrtc-kaleidoscope';
+import { presets } from './kaleidoscope.preset-book';
 
-applyVideoEffects(localCameraTrack.mediaStreamTrack, ['blur']);
+const { kaleidoscope } = bindKaleidoscope(localCameraTrack.mediaStreamTrack, { presets });
+kaleidoscope('blur-soft');
 ```
 
 **Web wiring.** On web, LiveKit owns the `RTCRtpSender`, so you cannot swap the track yourself; go through LiveKit's processor API instead. The opt-in `/livekit` subpath ships a ready-made processor (it needs `livekit-client`, which a LiveKit app already has):
@@ -69,7 +80,7 @@ import { KaleidoscopeProcessor } from 'react-native-webrtc-kaleidoscope/livekit'
 await localVideoTrack.setProcessor(new KaleidoscopeProcessor(['blur']), true);
 ```
 
-The second argument shows the processed stream in your local preview. The processor tears down its Insertable-Streams pipeline on camera flip (`restart`) and unpublish (`destroy`), so repeated flips do not leak generators.
+The `true` second argument to `setProcessor` shows the processed stream in your local preview. The processor tears down its Insertable-Streams pipeline on camera flip (`restart`) and unpublish (`destroy`), so repeated flips do not leak generators.
 
 ## Configure
 
@@ -93,49 +104,181 @@ bunx expo prebuild
 
 ## Use
 
+First declare a **preset book** in your project: a flat catalog of **composites**, the only things you can command. A composite is `{ name, taxonomy, thumbnail?, layers }`, where `taxonomy` is the picker's grouping path, root first (`[group]` or `[group, category]`, e.g. `['Backgrounds', 'Office']`); everything is a layer stack, painted back to front. A layer is `{ id, shader, target?, blend? }` plus the shader's fields: `image` takes a `source`; `direct` samples the ingest-normalized (upright, non-mirrored) camera frame for its target (`target: 'subject'` is the masked person, `target: 'background'` the full frame); `blur` and the generative shaders (`plasma`, `clouds`, …) take `uniforms`. `target` defaults to `'background'` (fullscreen); `'subject'` stencils to the segmented person. Each layer's `id` is unique within its composite. Declare the book `as const satisfies PresetBook` for per-layer typing.
+
+```ts
+// kaleidoscope.preset-book.ts
+import type { PresetBook } from 'react-native-webrtc-kaleidoscope';
+import { officeDark } from 'react-native-webrtc-kaleidoscope/images/office/office-dark';
+// Packaged composites ship ready to use; import and spread them in.
+import { wizardTower } from 'react-native-webrtc-kaleidoscope/composites/wizard-tower';
+
+export const presets = {
+  // Replace your background with an image, you composited over it.
+  'office-dark': {
+    name: 'Dark Office',
+    taxonomy: ['Backgrounds', 'Office'],
+    thumbnail: officeDark,
+    layers: [
+      { id: 'office-dark', shader: 'image', source: officeDark },
+      { id: 'you', shader: 'direct', target: 'subject' },
+    ],
+  },
+  // Blur the background, stay sharp.
+  'blur-heavy': {
+    name: 'Heavy',
+    taxonomy: ['Effects', 'Blur'],
+    layers: [
+      { id: 'bg', shader: 'blur', target: 'background', uniforms: { sigma: 7 } },
+      { id: 'you', shader: 'direct', target: 'subject' },
+    ],
+  },
+  // A packaged multi-layer composite (clouds + a cut-out image + you).
+  'wizard-tower': wizardTower,
+} as const satisfies PresetBook;
+```
+
+Then bind a track once and drive it with the three verbs:
+
 ```ts
 import { mediaDevices } from 'react-native-webrtc';
-import {
-  applyVideoEffects,
-  setBlurSigma,
-  setMaskHardness,
-  setMaskThreshold,
-} from 'react-native-webrtc-kaleidoscope';
+import { bindKaleidoscope } from 'react-native-webrtc-kaleidoscope';
+import { presets } from './kaleidoscope.preset-book';
 
 const stream = await mediaDevices.getUserMedia({ video: true });
 const [track] = stream.getVideoTracks();
 
-applyVideoEffects(track, ['mirror']);
-applyVideoEffects(track, ['blur']);
-applyVideoEffects(track, [{ name: 'background-image', source: 'office-1' }]);
-applyVideoEffects(track, []); // clear all effects
+const { kaleidoscope, transform, mask } = bindKaleidoscope(track, {
+  presets,
+  // Web rebuilds the pipeline per command and yields a NEW track; read it here
+  // (attach to <video> or replaceTrack). Native mutates the bound track in place.
+  onTrack: (out) => {
+    /* setPreviewTrack(out) */
+  },
+});
+
+// kaleidoscope, the art axis. Pass a preset id (autocompletes from your book):
+kaleidoscope('wizard-tower');
+// Override a layer's uniforms live, addressed by layer id, while the patched
+// preset is active (merged, no pipeline rebuild). `shader` types the uniforms:
+kaleidoscope('blur-heavy', [{ id: 'bg', shader: 'blur', uniforms: { sigma: 5 } }]);
+kaleidoscope(null);                          // clear the art
+
+// transform, absolute geometry. Every call is the full state from identity;
+// re-pass what you want to keep. rotate snaps to the nearest 90°.
+transform({ flip: { x: true }, rotate: 90 });
+transform();                                 // reset to identity
+
+// mask, the segmentation edge shared by every art effect. Both required, 0..1.
+mask({ hardness: 0.5, threshold: 0.5 });
+```
+
+That is the whole runtime surface: `kaleidoscope`, `transform`, `mask`.
+
+Numeric shader uniforms are normalized 0..1 by convention where practical; JSDoc documents each option's expected range as an IntelliSense hint (ranges are not enforced at runtime; validate in your own layer if you forward them to end users).
+
+**Tuning note:** all three platforms run MediaPipe selfie segmentation (Tasks Image Segmenter on native, the Selfie Segmentation Solution on web), so the mask edge that suits one may differ slightly from another. `mask({ hardness, threshold })` defaults to `0.5 / 0.5`; nudge it to match your camera and lighting.
 
-// Runtime tuning (effects pick up the new values on the next frame):
-setBlurSigma(25);        // Gaussian σ; clamped to [0.5, 64], default 8.
-setMaskHardness(0.2);    // smoothstep transition width; clamped to [0, 1]. 0 = soft halo, 1 = near-step. Default 0.5.
-setMaskThreshold(0.7);   // smoothstep center; clamped to [0.05, 0.95]. Higher rejects low-confidence pixels. Default 0.5.
+## Drop-in UI (optional)
+
+Build your own controls against the three verbs, or import a ready-made, headless picker from `react-native-webrtc-kaleidoscope/preset-book-menu` that reads your preset book directly:
+
+```tsx
+import { useEffect, useState } from 'react';
+import { PresetBookMenu } from 'react-native-webrtc-kaleidoscope/preset-book-menu';
+import { presets } from './kaleidoscope.preset-book';
+
+// `kaleidoscope` is the verb returned by bindKaleidoscope(track, { presets }).
+function BackgroundControls({ kaleidoscope }) {
+  const [art, setArt] = useState<keyof typeof presets | null>(null);
+  useEffect(() => {
+    if (art) kaleidoscope(art);
+    else kaleidoscope(null);
+  }, [art, kaleidoscope]);
+
+  return <PresetBookMenu presets={presets} value={art} onSelect={setArt} />;
+}
 ```
 
-Effects chain in array order.
+`PresetBookMenu` is a two-level browser driven by each composite's `taxonomy`: a tab row across the top, one tab per **group** (`taxonomy[0]`, e.g. Effects, Worlds, Backgrounds, Shaders), and a left-hand menu of **categories** (`taxonomy[1]`) under the active group; the tile grid is filtered by both. A flat (depth-1) group shows no category menu. Every preset renders as a uniform tile: a wallpaper when the composite has a `thumbnail`, a recessed button of the same footprint when it does not, so a thumbnail-less preset never breaks the grid. The same pieces are exported as standalone primitives (`PresetGrid`, `PresetTile`, plus the `usePresetBookMenu` hook and `PresetBookMenuLayout`), so you can lay out your own. Selection is controlled (`value` + `onSelect(id)`, narrowed to your book's keys); the components are presentational: they emit the selected id, you apply it via `kaleidoscope`.
 
-**Tuning note:** optimal values are platform-specific because each segmentation model (MediaPipe on web, MLKit on Android, Vision when iOS lands) produces a different confidence distribution. Working defaults on a typical well-lit scene:
+**Styling, three tiers.** Sensible defaults out of the box; override with an RN `style` prop, a `className` prop, or a `renderTile` render-prop slot for full control.
 
-| Platform | Blur sigma | Mask hardness | Mask threshold |
-|---|---|---|---|
-| Web (MediaPipe) | 25 | 0.2 | 0.85 |
-| Android (MLKit) | 30 | 0.2 | 0.6 |
+**NativeWind-ready.** The components accept `className`. To turn it on, import the opt-in registration once in your NativeWind interop setup (`nativewind` is an optional peer dependency; the core `./preset-book-menu` import never pulls it in):
 
-The library ships neutral defaults (8, 0.5, 0.5) and consumers tune at runtime via the API above; whether to ship the dialed-in values as platform-specific defaults is an open question waiting on iOS data.
+```ts
+import { registerKaleidoscopeNativeWind } from 'react-native-webrtc-kaleidoscope/nativewind';
+registerKaleidoscopeNativeWind();
+```
+
+### Live controls (the editor)
+
+For a tuning or admin panel, `react-native-webrtc-kaleidoscope/preset-control-panel` ships a headless editor that reads the active preset and renders a control per tunable uniform, plus the mask and transform panels:
+
+```tsx
+import {
+  KaleidoscopeThemeProvider,
+  PresetControlPanel,
+  MaskControlPanel,
+  TransformControlPanel,
+} from 'react-native-webrtc-kaleidoscope/preset-control-panel';
+
+// `controls` is the object from bindKaleidoscope(track, { presets }).
+<KaleidoscopeThemeProvider>
+  <PresetControlPanel presets={presets} value={art} onPatch={(p) => controls.kaleidoscope(art, [p])} />
+  <MaskControlPanel hardness={h} threshold={t} onChange={setMask} />
+  <TransformControlPanel flip={flip} rotate={rotate} onChange={setTransform} />
+</KaleidoscopeThemeProvider>
+```
+
+Each preset supplies its editor as a `controls` component on the book entry. The packaged composites export theirs at `react-native-webrtc-kaleidoscope/composites/<name>/controls`; for your own presets, compose `CompositeLayerControlPanel` over a shader's control descriptor (or `makeControls` for a custom widget). See [catalog/shaders/README.md](./catalog/shaders/README.md).
+
+Like the picker, the editor is controlled and presentational: it emits patches and you apply them. `KaleidoscopeThemeProvider` themes every control at once (a slot bank; `style` works everywhere, `className` via the same opt-in NativeWind interop). The sliders need `@react-native-community/slider` (an optional peer; a native module, so installing it needs a dev-client rebuild).
+
+Live per-layer tuning runs on web today; on native the editor renders but the live per-layer uniform channel is in progress. Mask and transform are live on every platform.
+
+## Worlds
+
+Packaged composites: a multi-layer stack (a generative shader or a cut-out image, the masked person on top), imported and spread into your book (e.g. `import { wizardTower } from 'react-native-webrtc-kaleidoscope/composites/wizard-tower'`). They carry their own `taxonomy: ['Worlds', <group>]`, so the menu groups them under the Worlds tab.
+
+| Wizard Tower | Observation Deck | Fairy Cave |
+|---|---|---|
+| <img src="composites/wizard-tower/wizard-tower.thumb.webp" width="220" alt="wizard-tower" /> | <img src="composites/observation-deck/observation-deck.thumb.webp" width="220" alt="observation-deck" /> | <img src="composites/fairy-cave/fairy-cave.thumb.webp" width="220" alt="fairy-cave" /> |
+| Underwater | Nebula | Simianlights |
+| <img src="composites/underwater/underwater.thumb.webp" width="220" alt="underwater" /> | <img src="composites/nebula/nebula.thumb.webp" width="220" alt="nebula" /> | <img src="composites/simianlights/simianlights.thumb.webp" width="220" alt="simianlights" /> |
+| Corporate Blobs | | |
+| <img src="composites/corporate-blobs/corporate-blobs.thumb.webp" width="220" alt="corporate-blobs" /> | | |
+
+The `clouds` (Sky) composite also ships; its preview tile is pending.
+
+## Background presets
+
+The bundled backgrounds ship as `image` layers, filed by category and imported per image (e.g. `import { officeDark } from 'react-native-webrtc-kaleidoscope/images/office/office-dark'`). On web an image can also be any image URL or data URI; native resolves bundled image ids only.
+
+| Category | Light | Dark |
+|---|---|---|
+| Office | <img src="images/office/office-light.thumb.webp" width="220" alt="office-light" /> | <img src="images/office/office-dark.thumb.webp" width="220" alt="office-dark" /> |
+| Home | <img src="images/home/home-light.thumb.webp" width="220" alt="home-light" /> | <img src="images/home/home-dark.thumb.webp" width="220" alt="home-dark" /> |
+| Nature | <img src="images/nature/landscape-light.thumb.webp" width="220" alt="landscape-light" /> | <img src="images/nature/landscape-dark.thumb.webp" width="220" alt="landscape-dark" /> |
+| Sci-Fi | <img src="images/sci-fi/sci-fi-light.thumb.webp" width="220" alt="sci-fi-light" /> | |
+| Underwater | | <img src="images/underwater/oceanscape-dark.thumb.webp" width="220" alt="oceanscape-dark" /> |
+| Simiancraft | <img src="images/simiancraft/simiancraft-light.thumb.webp" width="220" alt="simiancraft-light" /> | <img src="images/simiancraft/simiancraft-dark.thumb.webp" width="220" alt="simiancraft-dark" /> |
+
+The `simiancraft` category also ships two transparent brand images, `simiancraft-light-transparency` and `simiancraft-dark-transparency` (alpha preserved). Plus **`debug-resolutions`**, a viewport/resolution calibration grid for verifying background cover-fit:
+
+<img src="images/debug/debug-resolutions.thumb.webp" width="220" alt="debug-resolutions" />
+
+See [`catalog/images/README.md`](./catalog/images/README.md) for the folder layout, the two image formats, and how to add one.
 
 ## Web and native differences
 
 The API surface is the same across platforms, but the runtimes differ in ways worth knowing before you wire effects in:
 
-- **Effect parameters.** Web reads tuning from the global setters (`setBlurSigma`, `setMaskHardness`, `setMaskThreshold`) on the next frame. Native currently ignores per-call `EffectSpec` parameters such as `{ name: 'blur', sigma: 12 }`; tuning is global through the same setters. Per-call uniforms through the native registry are a follow-up.
-- **Background source.** `background-image.source` is a bundled preset name on native (the upstream `_setVideoEffects` registry is keyed by flat strings, not URIs), but on web it accepts either a preset name or an arbitrary image URL or data URI.
-- **Background presets ship as tree-shakeable files.** Two example backgrounds (`office-1`, `office-2`) are importable per preset: `import { office1 } from 'react-native-webrtc-kaleidoscope/backgrounds/office-1'`. Each preset is its own file behind its own subpath export, and the package sets `sideEffects: false`, so an unused preset is dropped by web bundlers — and, since Metro doesn't tree-shake, simply never imported on native. Web resolves the bundled WebP to a URL; native loads its own bundled copy by name. Web also still accepts an arbitrary image URL or data URI. See [`src/backgrounds/README.md`](./src/backgrounds/README.md).
-- **Segmentation model on web.** The web blur and background-image effects load MediaPipe Selfie Segmentation from the jsdelivr CDN (`cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation`) on first use. A strict Content-Security-Policy must allow that origin for `script-src`, `connect-src`, and the WASM fetch, and the effects do not work offline. Mirror needs no model.
-- **Browser support on web.** Effects use Insertable Streams (`MediaStreamTrackProcessor` and `MediaStreamTrackGenerator`), which ship in Chromium-based browsers; Safari and Firefox throw a typed capability error.
+- **Output track.** On web each `kaleidoscope`/`transform` command rebuilds the Insertable-Streams pipeline and yields a NEW `MediaStreamTrack`, surfaced via `onTrack`; on native the bound track is mutated in place. `mask` updates the segmentation edge the running composite reads each frame, with no rebuild on either platform.
+- **Image source.** An `image` layer's `source` is a bundled preset name on native (the upstream `_setVideoEffects` registry is keyed by flat strings, not URIs), but on web it accepts either a preset name or an arbitrary image URL or data URI.
+- **Background presets ship as tree-shakeable files.** The bundled backgrounds (see [Background presets](#background-presets)) are importable per image: `import { officeDark } from 'react-native-webrtc-kaleidoscope/images/office/office-dark'`. Each image is its own file behind its own subpath export, and the package sets `sideEffects: false`, so an unused preset is dropped by web bundlers; since Metro doesn't tree-shake, it is simply never imported on native. Web resolves the bundled WebP to a URL; native loads its own bundled copy by name. Web also still accepts an arbitrary image URL or data URI. See [`catalog/images/README.md`](./catalog/images/README.md).
+- **Segmentation model on web.** The web compositor loads MediaPipe Selfie Segmentation from the jsdelivr CDN (`cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation`) on first use. A strict Content-Security-Policy must allow that origin for `script-src`, `connect-src`, and the WASM fetch, and the effects do not work offline. The `transform` ops need no model.
+- **Browser support on web.** Effects use Insertable Streams (`MediaStreamTrackProcessor` and `MediaStreamTrackGenerator`), which ship in Chromium-based browsers (Chrome, Edge); Safari and Firefox lack the API, so the effects throw a clear capability error and the demo falls back to the unprocessed track.
 
 ## What this isn't
 
@@ -146,14 +289,20 @@ The API surface is the same across platforms, but the runtimes differ in ways wo
 
 ## Architecture
 
-The codebase lives across four surfaces:
+The canonical assets live in three root, folder-per-item directories, out of the TypeScript build path; the build and the prebuild copy read from them:
+
+- `catalog/shaders/<name>/`: each shader's `.frag` plus its typed `.ts` (uniforms + control descriptor). All shaders share one vertex stage, `catalog/shaders/_shared/passthrough.vert`; there is no per-shader `.vert`. `catalog/shaders/_shared/` also holds the per-layer composite frags (`composite-camera.frag`, `composite-image.frag`, `composite-subject.frag`, `composite-masked.frag`, `composite-blit.frag`) and `transform.frag`. `bun run build:shaders` codegens the web and Android sources and transpiles the iOS Metal from these. See [catalog/shaders/README.md](./catalog/shaders/README.md) to add or extend a shader.
+- `catalog/images/<category>/`: images filed by category, several per folder. Each image is a quad: `<leaf>.webp`, its `<leaf>.thumb.webp`, and the `<leaf>.ts` / `<leaf>.web.ts` loader pair, behind a `./images/<category>/<leaf>` subpath export.
+- `catalog/composites/<name>/`: each packaged composite (a `KaleidoscopePreset`), behind a `./composites/<name>` subpath export.
+
+The code lives across the platform surfaces:
 
-- `src/` — JS facade and shared types. `applyVideoEffects(track, effects)` plus runtime tuning setters.
-- `src/web/` — WebGL2 pipeline. MediaPipe segmentation + GLSL composite. One shader file per stage in `src/web/shaders.ts`.
-- `android/` — OpenGL ES 3.0 pipeline. MLKit segmentation (async, worker-thread, last-known-mask cache) + GLSL composite. Shaders inline in `gpu/Shaders.kt` as `const val` strings.
-- `ios/` — Metal pipeline (Swift) with Vision person segmentation. The canonical GLSL in `shaders/` transpiles to Metal Shading Language via `scripts/build-shaders.ts`. Implemented and verified on device.
+- `src/`: JS facade and shared types. `bindKaleidoscope` returns the `kaleidoscope` / `transform` / `mask` verbs; the preset-book types live in `src/kaleidoscope/`.
+- `web-driver/`: WebGL2 pipeline. MediaPipe segmentation + the layered compositor (`web-driver/effects/composite.ts`).
+- `android/`: OpenGL ES 3.0 pipeline. MediaPipe Tasks segmentation + the layered compositor (`effects/CompositeFactory.kt`); codegen lands in `gpu/ShadersGenerated.kt`, the hand-written layer GLSL in `effects/LayerShaders.kt`.
+- `ios/`: Metal pipeline (Swift) with MediaPipe Tasks segmentation (`selfie_segmenter.tflite`, the same model Android bundles); the canonical GLSL transpiles to Metal via `scripts/build-shaders.ts`.
 
-The composite shader (`shaders/composite.frag`) is the same GLSL source for every effect category (blur, background-image, future procedural backgrounds). Per-effect difference is upstream of the composite: how the `uBackground` texture gets produced.
+Every effect is a LAYER in one compositor: an `image` image, a `direct` passthrough (the masked person, or the raw camera), a camera-sampling `blur`, or a generative shader, composited back to front with per-layer blend. There is one registered native effect, `composite`; its layer stack is delivered out of band and reconciled each command.
 
 See [`PATTERNS.md`](./PATTERNS.md) for the file-layout conventions, texture-orientation contract, and recipe for adding new effects, shaders, presets, or tunable parameters.
 
@@ -162,6 +311,7 @@ See [`PATTERNS.md`](./PATTERNS.md) for the file-layout conventions, texture-orie
 - [CONTRIBUTING.md](./CONTRIBUTING.md): setup, scripts, commit conventions.
 - [AGENTS.md](./AGENTS.md): agent and contributor orientation.
 - [PATTERNS.md](./PATTERNS.md): codebase conventions and how-to-extend.
+- [catalog/shaders/README.md](./catalog/shaders/README.md): adding and extending shaders.
 - [SECURITY.md](./SECURITY.md): security policy and reporting.
 - [NOTICE.md](./NOTICE.md): third-party attributions.
 - Sibling projects: [chromonym](https://github.com/simiancraft/chromonym) and [unitforge](https://github.com/simiancraft/unitforge); same OSS-hygiene template.

package/android/build.gradle

@@ -31,6 +31,14 @@ android {
   kotlinOptions {
     jvmTarget = '17'
   }
+
+  // JVM unit tests (testDebugUnitTest) for pure-logic units like the composite
+  // layer-stack parser. returnDefaultValues makes android.util.Log a no-op stub
+  // instead of throwing; the real org.json on the test classpath (below) shadows
+  // the stubbed android.jar one so JSON parsing actually runs.
+  testOptions {
+    unitTests.returnDefaultValues = true
+  }
 }
 
 dependencies {
@@ -55,10 +63,14 @@ dependencies {
   }
   compileOnly project(rnWebrtcProject)
 
-  // Person/background mask for the blur effect. The Google artifact ID is
-  // `segmentation-selfie`, not `selfie-segmentation` (the plan and earlier
-  // notes had it inverted; that is why the pin would not resolve).
-  implementation 'com.google.mlkit:segmentation-selfie:16.0.0-beta6'
+  // Person/background mask: MediaPipe Tasks ImageSegmenter (Mask.kt), the same
+  // model family the web and iOS sides run.
+  implementation 'com.google.mediapipe:tasks-vision:0.10.14'
+
+  // Unit tests (JVM, testDebugUnitTest). The real org.json shadows the stubbed
+  // android.jar copy so CompositeLayers' JSONArray/JSONObject parsing runs for real.
+  testImplementation 'junit:junit:4.13.2'
+  testImplementation 'org.json:json:20231013'
 }
 
 // Helper: gracefully read root project ext properties with a default.

package/android/src/main/java/com/simiancraft/kaleidoscope/CompositeLayers.kt

@@ -0,0 +1,127 @@
+// Composite-layer spec side-channel for the native composite compositor.
+//
+// The Expo Module's setCompositeLayers(json) JS function writes here; the
+// CompositeFactory reads the current layer stack each frame and composites it.
+// This delivers the spec without re-registering: it carries the whole ordered
+// layer stack (the composite is one registered effect name, "composite", whose
+// contents JS swaps as the active composite changes).
+//
+// The wire shape is a JSON array of layer objects (see parse()). JS sends it as a
+// String across the Expo bridge; we parse it once at set() time into immutable
+// Kotlin models the GL thread can read without locking. @Volatile on the parsed
+// reference is the memory barrier; the JS-thread write publishes, the GL-thread
+// read observes the latest stable snapshot.
+//
+// A malformed layer is skipped with a log rather than crashing the render thread;
+// an unparseable whole payload leaves the previous composite in place.
+
+package com.simiancraft.kaleidoscope
+
+import android.util.Log
+import org.json.JSONArray
+import org.json.JSONObject
+
+/** One parsed composite layer. The `shader` discriminant decides which fields matter. */
+internal data class CompositeLayer(
+    val id: String, // unique within a composite; the live-tuning / patch address
+    val shader: String,
+    val target: String, // "background" | "subject"
+    val blend: String?, // "normal" | "additive" | null (base = opaque)
+    val source: String?, // image id for an `image` layer; null otherwise
+    val uniforms: Map<String, FloatArray>, // generative-layer uniforms, by name
+)
+
+internal object CompositeLayers {
+    private const val TAG = "Kaleidoscope.CompositeLayers"
+
+    @Volatile
+    private var layers: List<CompositeLayer> = emptyList()
+
+    /** The current parsed layer stack; a stable snapshot safe to read on the GL thread. */
+    fun get(): List<CompositeLayer> = layers
+
+    /**
+     * Parse and store the composite layer stack from the JS-supplied JSON string.
+     * Called on the Expo module thread (JS-driven), not the GL thread. Leaves the
+     * previous composite in place on a whole-payload parse failure.
+     */
+    fun set(json: String) {
+        val parsed =
+            try {
+                parse(json)
+            } catch (t: Throwable) {
+                Log.e(TAG, "failed to parse composite layers; keeping previous composite", t)
+                return
+            }
+        layers = parsed
+        Log.i(TAG, "composite layers set: ${parsed.size} layer(s) [${parsed.joinToString(",") { it.shader }}]")
+    }
+
+    /** Clear the active composite (used when a non-composite effect takes over). */
+    fun clear() {
+        layers = emptyList()
+    }
+
+    private fun parse(json: String): List<CompositeLayer> {
+        val arr = JSONArray(json)
+        val out = ArrayList<CompositeLayer>(arr.length())
+        for (i in 0 until arr.length()) {
+            val obj = arr.optJSONObject(i) ?: continue
+            val shader = obj.optString("shader", "")
+            if (shader.isEmpty()) {
+                Log.w(TAG, "layer $i has no shader; skipping")
+                continue
+            }
+            // `id` is always present on the wire now (serializeCompositeLayers emits it);
+            // fall back to the array index so a malformed payload missing it still
+            // yields a stable, unique-per-stack address rather than a collision.
+            val id = if (obj.has("id") && !obj.isNull("id")) obj.getString("id") else i.toString()
+            val target = obj.optString("target", "background")
+            val blend = if (obj.has("blend") && !obj.isNull("blend")) obj.getString("blend") else null
+            val source = if (obj.has("source") && !obj.isNull("source")) obj.getString("source") else null
+            val uniforms = parseUniforms(obj.optJSONObject("uniforms"))
+            out.add(CompositeLayer(id, shader, target, blend, source, uniforms))
+        }
+        return out
+    }
+
+    // name -> FloatArray. A JSON number -> [f]; a JSON array of numbers -> FloatArray.
+    // A non-numeric or malformed value is skipped with a log (the shader keeps its
+    // GLSL default for that name).
+    private fun parseUniforms(obj: JSONObject?): Map<String, FloatArray> {
+        if (obj == null) return emptyMap()
+        val out = HashMap<String, FloatArray>(obj.length())
+        val keys = obj.keys()
+        while (keys.hasNext()) {
+            val key = keys.next()
+            val value = obj.opt(key)
+            val floats = normalize(value)
+            if (floats == null) {
+                Log.w(TAG, "skipping uniform '$key': unsupported type ${value?.javaClass?.simpleName}")
+                continue
+            }
+            out[key] = floats
+        }
+        return out
+    }
+
+    private fun normalize(value: Any?): FloatArray? =
+        when (value) {
+            is Number -> floatArrayOf(value.toFloat())
+            is JSONArray -> {
+                val out = FloatArray(value.length())
+                var ok = true
+                for (i in 0 until value.length()) {
+                    val n = value.opt(i)
+                    if (n is Number) {
+                        out[i] = n.toFloat()
+                    } else {
+                        ok = false
+                        break
+                    }
+                }
+                if (ok) out else null
+            }
+            else -> null
+        }
+}