@voltkit-rtc/client 0.8.0 → 0.9.0

package/CHANGELOG.md

@@ -1,5 +1,34 @@
 # Changelog
 
+## 0.9.0
+
+- **AI noise suppression: `@voltkit-rtc/client/noise-suppression`** (Agora AI Denoiser, LiveKit Krisp, ZEGOCLOUD
+  AI noise suppression). `createNoiseSuppressor({ workletUrl?, wasmUrl?, simdWasmUrl? })` is RNNoise in an
+  AudioWorklet (`@sapphi-red/web-noise-suppressor` 0.4.1, MIT, loaded only when used); files from jsDelivr at the
+  pinned version unless self-hosted. In Chromium, rumble-like noise went down by more than 60 dB.
+- **`setAudioProcessor(processor | null)` and `audioProcessorInUse`** (LiveKit `setProcessor`): the microphone goes
+  through an `AudioTrackProcessor` (`{ name, process(track), destroy() }`) before it is sent. Swapped in place while
+  live; kept across device switches, a recovered microphone, reconnects and audio-quality changes. A failing
+  processor sends the microphone unprocessed and emits `error`. `mic-silent` is not checked while one is on.
+- **Virtual background and beauty: `@voltkit-rtc/client/video-effects`** (LiveKit track processors, Agora/ZEGOCLOUD
+  virtual background and beauty, TRTC video effects). `createVideoEffects({ background: 'none' | 'blur' | { image },
+  blur, beauty, wasmPath?, modelUrl?, maxFrameRate?, onOverload? })`, `update()` live, `isVideoEffectsSupported()`.
+  WebGL2: mask-weighted blur (no halo), edges refined against the picture, cover-fitted images, beauty without a
+  face model. MediaPipe's selfie segmenter (landscape model for wide pictures, square for tall), from the optional
+  peer `@mediapipe/tasks-vision` at exactly 0.10.35 (later versions report usage to Google); nothing behind the
+  person shows until it is found. Insertable streams in Chrome/Edge, a hidden video on a worker timer elsewhere;
+  Chime-style adaptive segmentation on slow devices.
+- **`setVideoProcessor(processor | null)` and `videoProcessorInUse`**: the camera goes through a `VideoTrackProcessor`
+  (`{ name, process(track), destroy() }`, same size out) before it is sent; the processed picture is also what `local`
+  shows. Swapped in place while live; kept across `switchDevice`, `replaceVideoTrack`, reconnects and quality changes,
+  without restarting the processor. The camera behind it is the one switched, constrained, watched and released.
+  A failing processor sends the camera unprocessed and emits `error`. `camera-fps-low` is not checked while one is on.
+- **A faster first picture** (Agora/TRTC fast first frame). The join asks for the media transports and uses the ones
+  the server made with it (`join {transports}`, controllers from 2026-10-20; older ones are asked as before). The
+  DTLS parameters are sent without waiting for the reply, since mediasoup takes them after its handshake has begun:
+  ICE starts a round trip sooner, and a refusal rejoins. A video's first layers no longer wait before its resume.
+  Up to three round trips fewer before the first publish or the first video.
+
 ## 0.8.0
 
 - **Media recovery:** send and receive transports restart ICE after failure or

package/README.md

@@ -245,6 +245,102 @@ off only for headphones or line-in music, or listeners hear their own echo.
 Music (`publishMusic`) is a separate 128 kbps stereo track and is not affected:
 it is HD and free whatever the microphone's quality (see [Music](#music)).
 
+### Noise suppression (0.9.0)
+
+AI noise suppression for the microphone, as Agora's AI Denoiser, LiveKit's
+Krisp filter and ZEGOCLOUD's AI noise suppression do: RNNoise, a small neural
+network, runs in the browser's audio thread and takes fans, traffic, air
+conditioning and room rumble out of the voice. It is a separate import, so apps
+that do not use it never download it:
+
+```ts
+import { createNoiseSuppressor } from '@voltkit-rtc/client/noise-suppression';
+
+await client.setAudioProcessor(createNoiseSuppressor());   // before or after publish
+client.audioProcessorInUse;                                 // the processor, or null
+await client.setAudioProcessor(null);                       // send the microphone as it is again
+```
+
+- It applies to the microphone published now (swapped in place: listeners hear
+  no gap) and to every one after it: `switchDevice('audioinput', …)`, a
+  microphone recovered after unplugging, a reconnect, `setAudioQuality`.
+  Music and screen-share audio are not processed.
+- The worklet (64 kB) and the model (150 kB) load the first time it runs, from
+  jsDelivr at the exact version this SDK was tested with. Under a strict
+  Content-Security-Policy, copy `rnnoise/workletProcessor.js`, `rnnoise.wasm`
+  and `rnnoise_simd.wasm` from `node_modules/@sapphi-red/web-noise-suppressor/dist`
+  to your site and pass their URLs:
+  `createNoiseSuppressor({ workletUrl, wasmUrl, simdWasmUrl })`.
+- If it cannot start (the files are blocked, an old browser), the microphone
+  still goes out, unprocessed, and `error` says why. The next microphone tries
+  again.
+- Turn it on from a click (a toggle, the join button): browsers start audio
+  processing only after the page has been touched, and the SDK resumes it on
+  the first tap or key press.
+- `mic-silent` is not reported while a processor is on: a quiet room is
+  near-silence after denoising. Your own level in `audioLevels` is what
+  listeners hear.
+- Measured in Chromium: rumble-like noise (fans, traffic, AC) went down by more
+  than 60 dB; steady white hiss, which it was not trained on, by 7-12 dB. Keep
+  the browser's own `echoCancellation`; its `noiseSuppression` can stay on too.
+
+Your own processor works the same way: anything with `name`,
+`process(track) → Promise<MediaStreamTrack>` and `destroy()`
+(`AudioTrackProcessor`), such as a licensed Krisp build.
+
+### Virtual background and beauty (0.9.0)
+
+A blurred or replaced background and beauty for the camera, as LiveKit's track
+processors, Agora's and ZEGOCLOUD's virtual background and beauty, and TRTC's
+video effects do: in the browser, on the GPU. A separate import, so apps that do
+not use it never download it:
+
+```ts
+import { createVideoEffects, isVideoEffectsSupported } from '@voltkit-rtc/client/video-effects';
+
+if (isVideoEffectsSupported()) {
+  const effects = createVideoEffects({ background: 'blur', beauty: true });
+  await client.setVideoProcessor(effects);                           // before or after publish
+  client.videoProcessorInUse;                                        // the processor, or null
+
+  await effects.update({ background: { image: '/office.jpg' } });   // live: no pause for viewers
+  await effects.update({ blur: 8, beauty: { smooth: 0.7, brighten: 0.4 } });
+  await client.setVideoProcessor(null);                              // the camera as it is again
+}
+```
+
+- `background`: `'none'`, `'blur'` (`blur` 1-10, default 5) or `{ image }` (a
+  URL, Blob, image, canvas or ImageBitmap), fitted like `object-fit: cover`.
+- `beauty`: `true`, or `smooth`, `brighten`, `redness`, `sharpen` (0-1;
+  defaults 0.5, 0.3, 0.1, 0.3). Skin is smoothed with an edge-preserving filter
+  (eyes, lips and teeth stay sharp) and lightly brightened.
+- What you see in `local` is what viewers get. Switching cameras
+  (`switchDevice`), `replaceVideoTrack`, reconnects and quality changes keep the
+  effects without restarting them. A paused camera costs no processing.
+- **Background effects need MediaPipe**: `npm install @mediapipe/tasks-vision@0.10.35`
+  (exactly that version: later ones report usage to Google). Its wasm (3 MB
+  compressed) and the person model (250 kB) load the first time a background is
+  used, from jsDelivr and Google. To serve them yourself (a strict CSP), copy
+  `node_modules/@mediapipe/tasks-vision/wasm` to your site and pass `wasmPath`,
+  and `modelUrl` for the model. Beauty alone needs none of this.
+- Nothing behind you shows while the model loads: the background is fully
+  blurred or replaced until you are found. If the effects cannot start, the
+  camera goes out as it is and `error` says why, so watch `error` when privacy
+  matters.
+- Webcams (wide pictures) use MediaPipe's landscape model and phones (tall
+  pictures) the square one, which is what each traces best.
+- A slow device finds you only every few frames and reuses the outline (the
+  picture keeps the camera's frame rate). `onOverload` tells you when even that
+  is not enough: turn the background off or lower the camera's size.
+- Chrome and Edge process each frame as it arrives and keep going in a
+  background tab. Firefox and Safari draw from a hidden video on a timer, which
+  a background tab can slow down.
+
+Your own processor works the same way: anything with `name`,
+`process(track) → Promise<MediaStreamTrack>` (same size and frame rate) and
+`destroy()` (`VideoTrackProcessor`). The SDK stops a picture it no longer sends;
+stop drawing it once its `readyState` is `'ended'`.
+
 ### Project media settings
 
 A project's owner chooses what it may publish: audio and video on or off, the
@@ -677,6 +773,12 @@ StreamClient.prepareConnection(wsUrl); // on the room screen; never rejects
 await client.connect();
 ```
 
+From 0.9.0 the first picture comes sooner, with nothing to change in your app:
+the join brings the media transports with it (controllers from 2026-10-20), and
+neither the DTLS handshake nor a video's first layer waits for an
+acknowledgement any more. Each is a round trip saved, 50-150 ms on a mobile
+network, for the first publish and the first video alike.
+
 ### Check before going live (0.8.0)
 
 ```ts

package/dist/cjs/diagnostics.d.ts

@@ -6,7 +6,7 @@
  * attach to a support ticket: the SDK's recent events (a ring buffer) and its state now.
  */
 /** This package's version; kept equal to package.json by a test. */
-export declare const SDK_VERSION = "0.8.0";
+export declare const SDK_VERSION = "0.9.0";
 /** How many recent events the log keeps. */
 export declare const DIAGNOSTICS_LOG_SIZE = 200;
 export interface DiagnosticsEntry {

package/dist/cjs/diagnostics.js

@@ -9,7 +9,7 @@
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.DiagnosticsLog = exports.DIAGNOSTICS_LOG_SIZE = exports.SDK_VERSION = void 0;
 /** This package's version; kept equal to package.json by a test. */
-exports.SDK_VERSION = '0.8.0';
+exports.SDK_VERSION = '0.9.0';
 /** How many recent events the log keeps. */
 exports.DIAGNOSTICS_LOG_SIZE = 200;
 /** The events worth keeping for support, and how to summarise each. */

package/dist/cjs/effects-core.d.ts

@@ -0,0 +1,95 @@
+/**
+ * Video effects, the parts that need no browser (docs/implementation-progress/RTC_PERF_PROGRESS.md P3-5): settings,
+ * fitting a background image, the size segmentation runs at, and pacing it on a slow device. Kept apart from the
+ * WebGL and MediaPipe code so they are tested in Node.
+ */
+/** A background picture: a URL, or anything the browser can draw. */
+export type BackgroundImageSource = string | Blob | ImageBitmap | HTMLImageElement | HTMLCanvasElement | OffscreenCanvas;
+/** What happens behind the person: nothing, a blur, or a picture. */
+export type BackgroundSetting = 'none' | 'blur' | {
+    image: BackgroundImageSource;
+};
+/** Beauty, each 0-1 (Agora's beauty effect has the same four). */
+export interface BeautyOptions {
+    /** Skin smoothing. Default 0.5. */
+    smooth?: number;
+    /** Brightening. Default 0.3. */
+    brighten?: number;
+    /** A rosier skin. Default 0.1. */
+    redness?: number;
+    /** Sharper eyes, hair and edges (not skin). Default 0.3. */
+    sharpen?: number;
+}
+export interface BeautySettings {
+    smooth: number;
+    brighten: number;
+    redness: number;
+    sharpen: number;
+}
+export declare const DEFAULT_BEAUTY: Readonly<BeautySettings>;
+/** The blur behind the person, 1 (light) to 10 (strong). */
+export declare const DEFAULT_BLUR = 5;
+/** Beauty settings from what the app passed: `true` is the defaults, `false`/`null` is off. */
+export declare function beautySettings(options: BeautyOptions | boolean | null | undefined): BeautySettings | null;
+/** A blur level from what the app passed: 1-10, rounded; anything else is the default. */
+export declare function blurLevel(value: unknown): number;
+/**
+ * The blur's Gaussian sigma, in texels of the quarter-size picture it is computed on: a share of the picture's height,
+ * so a level looks the same at 480p and 1080p.
+ */
+export declare function blurSigma(level: number, blurHeight: number): number;
+/** What the background setting asks of the pipeline. */
+export declare function backgroundMode(setting: BackgroundSetting | undefined | null): 'none' | 'blur' | 'image';
+/**
+ * Fit a background picture to the frame the way CSS `object-fit: cover` does: it fills the frame, cropped evenly on
+ * the long side, never stretched. Returns the texture transform `imageUv = frameUv * scale + offset`.
+ */
+export declare function coverFit(imageWidth: number, imageHeight: number, frameWidth: number, frameHeight: number): {
+    sx: number;
+    sy: number;
+    ox: number;
+    oy: number;
+};
+/** The size the person is found at: the frame scaled so its longer side is `longSide` (the model's input). */
+export declare function segmentationSize(width: number, height: number, longSide?: number): {
+    width: number;
+    height: number;
+};
+/** How fast the mask follows a new one: lower is steadier between frames, higher follows movement sooner. */
+export declare const MASK_FOLLOW = 0.65;
+/**
+ * Smooth the person mask between frames (Stream's temporal smoothing), against flicker at the edges, and write it as
+ * bytes for the GPU. `steady` keeps the running mask; a new size starts it over.
+ */
+export declare function smoothMask(confidence: Float32Array, steady: Float32Array | null, out: Uint8Array, follow?: number): Float32Array;
+/** The most frames a mask is reused for before the device is called overloaded. */
+export declare const MAX_SEGMENT_EVERY = 6;
+/**
+ * Amazon Chime's adaptive background filter: finding the person is the expensive part, so when it takes too much of
+ * each frame's time it runs only every Nth frame and the last mask is reused -- the picture still runs at the camera's
+ * rate. N comes back down once it is cheap again. `overloaded` is the point where even that is not enough.
+ */
+export declare class SegmentationPacer {
+    private readonly maxEvery;
+    private average;
+    private changedAt;
+    private every;
+    constructor(maxEvery?: number);
+    /** Run segmentation on every this-many frames. */
+    get interval(): number;
+    /** Average milliseconds one segmentation takes. */
+    get segmentMs(): number;
+    shouldSegment(frameIndex: number): boolean;
+    /** The share of the frame time segmentation takes now, given how often it runs. */
+    load(frameIntervalMs: number): number;
+    /** Whether segmenting even this rarely still takes too long. */
+    overloaded(frameIntervalMs: number): boolean;
+    /** One segmentation took `ms`, at a frame interval of `frameIntervalMs`. */
+    report(ms: number, frameIntervalMs: number, now: number): void;
+}
+/** Timestamps MediaPipe's video mode accepts: in milliseconds and always increasing. */
+export declare class MonotonicClock {
+    private last;
+    next(now: number): number;
+}
+//# sourceMappingURL=effects-core.d.ts.map

package/dist/cjs/effects-core.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"effects-core.d.ts","sourceRoot":"","sources":["../../src/effects-core.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAEH,qEAAqE;AACrE,MAAM,MAAM,qBAAqB,GAAG,MAAM,GAAG,IAAI,GAAG,WAAW,GAAG,gBAAgB,GAAG,iBAAiB,GAAG,eAAe,CAAC;AAEzH,qEAAqE;AACrE,MAAM,MAAM,iBAAiB,GAAG,MAAM,GAAG,MAAM,GAAG;IAAE,KAAK,EAAE,qBAAqB,CAAA;CAAE,CAAC;AAEnF,kEAAkE;AAClE,MAAM,WAAW,aAAa;IAC5B,mCAAmC;IACnC,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,gCAAgC;IAChC,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,kCAAkC;IAClC,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,4DAA4D;IAC5D,OAAO,CAAC,EAAE,MAAM,CAAC;CAClB;AAED,MAAM,WAAW,cAAc;IAC7B,MAAM,EAAE,MAAM,CAAC;IACf,QAAQ,EAAE,MAAM,CAAC;IACjB,OAAO,EAAE,MAAM,CAAC;IAChB,OAAO,EAAE,MAAM,CAAC;CACjB;AAED,eAAO,MAAM,cAAc,EAAE,QAAQ,CAAC,cAAc,CAA6E,CAAC;AAElI,4DAA4D;AAC5D,eAAO,MAAM,YAAY,IAAI,CAAC;AAK9B,+FAA+F;AAC/F,wBAAgB,cAAc,CAAC,OAAO,EAAE,aAAa,GAAG,OAAO,GAAG,IAAI,GAAG,SAAS,GAAG,cAAc,GAAG,IAAI,CASzG;AAED,0FAA0F;AAC1F,wBAAgB,SAAS,CAAC,KAAK,EAAE,OAAO,GAAG,MAAM,CAEhD;AAED;;;GAGG;AACH,wBAAgB,SAAS,CAAC,KAAK,EAAE,MAAM,EAAE,UAAU,EAAE,MAAM,GAAG,MAAM,CAEnE;AAED,wDAAwD;AACxD,wBAAgB,cAAc,CAAC,OAAO,EAAE,iBAAiB,GAAG,SAAS,GAAG,IAAI,GAAG,MAAM,GAAG,MAAM,GAAG,OAAO,CAIvG;AAED;;;GAGG;AACH,wBAAgB,QAAQ,CAAC,UAAU,EAAE,MAAM,EAAE,WAAW,EAAE,MAAM,EAAE,UAAU,EAAE,MAAM,EAAE,WAAW,EAAE,MAAM,GAAG;IAAE,EAAE,EAAE,MAAM,CAAC;IAAC,EAAE,EAAE,MAAM,CAAC;IAAC,EAAE,EAAE,MAAM,CAAC;IAAC,EAAE,EAAE,MAAM,CAAA;CAAE,CAW7J;AAED,8GAA8G;AAC9G,wBAAgB,gBAAgB,CAAC,KAAK,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,EAAE,QAAQ,SAAM,GAAG;IAAE,KAAK,EAAE,MAAM,CAAC;IAAC,MAAM,EAAE,MAAM,CAAA;CAAE,CAIjH;AAED,6GAA6G;AAC7G,eAAO,MAAM,WAAW,OAAO,CAAC;AAEhC;;;GAGG;AACH,wBAAgB,UAAU,CAAC,UAAU,EAAE,YAAY,EAAE,MAAM,EAAE,YAAY,GAAG,IAAI,EAAE,GAAG,EAAE,UAAU,EAAE,MAAM,SAAc,GAAG,YAAY,CAUrI;AAED,mFAAmF;AACnF,eAAO,MAAM,iBAAiB,IAAI,CAAC;AAEnC;;;;GAIG;AACH,qBAAa,iBAAiB;IAKhB,OAAO,CAAC,QAAQ,CAAC,QAAQ;IAJrC,OAAO,CAAC,OAAO,CAAuB;IACtC,OAAO,CAAC,SAAS,CAA4B;IAC7C,OAAO,CAAC,KAAK,CAAK;gBAEW,QAAQ,SAAoB;IAEzD,kDAAkD;IAClD,IAAI,QAAQ,IAAI,MAAM,CAErB;IAED,mDAAmD;IACnD,IAAI,SAAS,IAAI,MAAM,CAEtB;IAED,aAAa,CAAC,UAAU,EAAE,MAAM,GAAG,OAAO;IAI1C,mFAAmF;IACnF,IAAI,CAAC,eAAe,EAAE,MAAM,GAAG,MAAM;IAKrC,gEAAgE;IAChE,UAAU,CAAC,eAAe,EAAE,MAAM,GAAG,OAAO;IAI5C,4EAA4E;IAC5E,MAAM,CAAC,EAAE,EAAE,MAAM,EAAE,eAAe,EAAE,MAAM,EAAE,GAAG,EAAE,MAAM,GAAG,IAAI;CAY/D;AAED,wFAAwF;AACxF,qBAAa,cAAc;IACzB,OAAO,CAAC,IAAI,CAAK;IACjB,IAAI,CAAC,GAAG,EAAE,MAAM,GAAG,MAAM;CAI1B"}

package/dist/cjs/effects-core.js

@@ -0,0 +1,154 @@
+"use strict";
+/**
+ * Video effects, the parts that need no browser (docs/implementation-progress/RTC_PERF_PROGRESS.md P3-5): settings,
+ * fitting a background image, the size segmentation runs at, and pacing it on a slow device. Kept apart from the
+ * WebGL and MediaPipe code so they are tested in Node.
+ */
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.MonotonicClock = exports.SegmentationPacer = exports.MAX_SEGMENT_EVERY = exports.MASK_FOLLOW = exports.DEFAULT_BLUR = exports.DEFAULT_BEAUTY = void 0;
+exports.beautySettings = beautySettings;
+exports.blurLevel = blurLevel;
+exports.blurSigma = blurSigma;
+exports.backgroundMode = backgroundMode;
+exports.coverFit = coverFit;
+exports.segmentationSize = segmentationSize;
+exports.smoothMask = smoothMask;
+exports.DEFAULT_BEAUTY = Object.freeze({ smooth: 0.5, brighten: 0.3, redness: 0.1, sharpen: 0.3 });
+/** The blur behind the person, 1 (light) to 10 (strong). */
+exports.DEFAULT_BLUR = 5;
+const unit = (value, fallback) => typeof value === 'number' && Number.isFinite(value) ? Math.min(1, Math.max(0, value)) : fallback;
+/** Beauty settings from what the app passed: `true` is the defaults, `false`/`null` is off. */
+function beautySettings(options) {
+    if (!options)
+        return null;
+    const o = options === true ? {} : options;
+    return {
+        smooth: unit(o.smooth, exports.DEFAULT_BEAUTY.smooth),
+        brighten: unit(o.brighten, exports.DEFAULT_BEAUTY.brighten),
+        redness: unit(o.redness, exports.DEFAULT_BEAUTY.redness),
+        sharpen: unit(o.sharpen, exports.DEFAULT_BEAUTY.sharpen),
+    };
+}
+/** A blur level from what the app passed: 1-10, rounded; anything else is the default. */
+function blurLevel(value) {
+    return typeof value === 'number' && Number.isFinite(value) ? Math.min(10, Math.max(1, Math.round(value))) : exports.DEFAULT_BLUR;
+}
+/**
+ * The blur's Gaussian sigma, in texels of the quarter-size picture it is computed on: a share of the picture's height,
+ * so a level looks the same at 480p and 1080p.
+ */
+function blurSigma(level, blurHeight) {
+    return Math.max(1, (blurLevel(level) / 10) * (blurHeight / 14));
+}
+/** What the background setting asks of the pipeline. */
+function backgroundMode(setting) {
+    if (setting === 'blur')
+        return 'blur';
+    if (setting && typeof setting === 'object' && 'image' in setting && setting.image)
+        return 'image';
+    return 'none';
+}
+/**
+ * Fit a background picture to the frame the way CSS `object-fit: cover` does: it fills the frame, cropped evenly on
+ * the long side, never stretched. Returns the texture transform `imageUv = frameUv * scale + offset`.
+ */
+function coverFit(imageWidth, imageHeight, frameWidth, frameHeight) {
+    if (!(imageWidth > 0 && imageHeight > 0 && frameWidth > 0 && frameHeight > 0))
+        return { sx: 1, sy: 1, ox: 0, oy: 0 };
+    const image = imageWidth / imageHeight;
+    const frame = frameWidth / frameHeight;
+    if (image > frame) {
+        // Wider than the frame: its sides are cropped.
+        const sx = frame / image;
+        return { sx, sy: 1, ox: (1 - sx) / 2, oy: 0 };
+    }
+    const sy = image / frame;
+    return { sx: 1, sy, ox: 0, oy: (1 - sy) / 2 };
+}
+/** The size the person is found at: the frame scaled so its longer side is `longSide` (the model's input). */
+function segmentationSize(width, height, longSide = 256) {
+    if (!(width > 0 && height > 0))
+        return { width: longSide, height: longSide };
+    const scale = longSide / Math.max(width, height);
+    return { width: Math.max(1, Math.round(width * scale)), height: Math.max(1, Math.round(height * scale)) };
+}
+/** How fast the mask follows a new one: lower is steadier between frames, higher follows movement sooner. */
+exports.MASK_FOLLOW = 0.65;
+/**
+ * Smooth the person mask between frames (Stream's temporal smoothing), against flicker at the edges, and write it as
+ * bytes for the GPU. `steady` keeps the running mask; a new size starts it over.
+ */
+function smoothMask(confidence, steady, out, follow = exports.MASK_FOLLOW) {
+    const fresh = steady === null || steady.length !== confidence.length;
+    const running = fresh ? new Float32Array(confidence) : steady;
+    for (let i = 0; i < confidence.length; i += 1) {
+        const value = Math.min(1, Math.max(0, confidence[i]));
+        const next = fresh ? value : running[i] + (value - running[i]) * follow;
+        running[i] = next;
+        out[i] = Math.round(next * 255);
+    }
+    return running;
+}
+/** The most frames a mask is reused for before the device is called overloaded. */
+exports.MAX_SEGMENT_EVERY = 6;
+/**
+ * Amazon Chime's adaptive background filter: finding the person is the expensive part, so when it takes too much of
+ * each frame's time it runs only every Nth frame and the last mask is reused -- the picture still runs at the camera's
+ * rate. N comes back down once it is cheap again. `overloaded` is the point where even that is not enough.
+ */
+class SegmentationPacer {
+    maxEvery;
+    average = null;
+    changedAt = Number.NEGATIVE_INFINITY;
+    every = 1;
+    constructor(maxEvery = exports.MAX_SEGMENT_EVERY) {
+        this.maxEvery = maxEvery;
+    }
+    /** Run segmentation on every this-many frames. */
+    get interval() {
+        return this.every;
+    }
+    /** Average milliseconds one segmentation takes. */
+    get segmentMs() {
+        return this.average ?? 0;
+    }
+    shouldSegment(frameIndex) {
+        return frameIndex % this.every === 0;
+    }
+    /** The share of the frame time segmentation takes now, given how often it runs. */
+    load(frameIntervalMs) {
+        if (this.average === null || !(frameIntervalMs > 0))
+            return 0;
+        return this.average / (this.every * frameIntervalMs);
+    }
+    /** Whether segmenting even this rarely still takes too long. */
+    overloaded(frameIntervalMs) {
+        return this.every >= this.maxEvery && this.load(frameIntervalMs) > 0.3;
+    }
+    /** One segmentation took `ms`, at a frame interval of `frameIntervalMs`. */
+    report(ms, frameIntervalMs, now) {
+        if (!Number.isFinite(ms) || ms < 0)
+            return;
+        this.average = this.average === null ? ms : this.average * 0.8 + ms * 0.2;
+        const load = this.load(frameIntervalMs);
+        if (load > 0.3 && this.every < this.maxEvery && now - this.changedAt >= 1000) {
+            this.every += 1;
+            this.changedAt = now;
+        }
+        else if (load < 0.12 && this.every > 1 && now - this.changedAt >= 3000) {
+            this.every -= 1;
+            this.changedAt = now;
+        }
+    }
+}
+exports.SegmentationPacer = SegmentationPacer;
+/** Timestamps MediaPipe's video mode accepts: in milliseconds and always increasing. */
+class MonotonicClock {
+    last = 0;
+    next(now) {
+        this.last = Math.max(this.last + 1, Math.round(now));
+        return this.last;
+    }
+}
+exports.MonotonicClock = MonotonicClock;
+//# sourceMappingURL=effects-core.js.map

package/dist/cjs/effects-core.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"effects-core.js","sourceRoot":"","sources":["../../src/effects-core.ts"],"names":[],"mappings":";AAAA;;;;GAIG;;;AAoCH,wCASC;AAGD,8BAEC;AAMD,8BAEC;AAGD,wCAIC;AAMD,4BAWC;AAGD,4CAIC;AASD,gCAUC;AAjFY,QAAA,cAAc,GAA6B,MAAM,CAAC,MAAM,CAAC,EAAE,MAAM,EAAE,GAAG,EAAE,QAAQ,EAAE,GAAG,EAAE,OAAO,EAAE,GAAG,EAAE,OAAO,EAAE,GAAG,EAAE,CAAC,CAAC;AAElI,4DAA4D;AAC/C,QAAA,YAAY,GAAG,CAAC,CAAC;AAE9B,MAAM,IAAI,GAAG,CAAC,KAAc,EAAE,QAAgB,EAAU,EAAE,CACxD,OAAO,KAAK,KAAK,QAAQ,IAAI,MAAM,CAAC,QAAQ,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,QAAQ,CAAC;AAEnG,+FAA+F;AAC/F,SAAgB,cAAc,CAAC,OAAmD;IAChF,IAAI,CAAC,OAAO;QAAE,OAAO,IAAI,CAAC;IAC1B,MAAM,CAAC,GAAkB,OAAO,KAAK,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,OAAO,CAAC;IACzD,OAAO;QACL,MAAM,EAAE,IAAI,CAAC,CAAC,CAAC,MAAM,EAAE,sBAAc,CAAC,MAAM,CAAC;QAC7C,QAAQ,EAAE,IAAI,CAAC,CAAC,CAAC,QAAQ,EAAE,sBAAc,CAAC,QAAQ,CAAC;QACnD,OAAO,EAAE,IAAI,CAAC,CAAC,CAAC,OAAO,EAAE,sBAAc,CAAC,OAAO,CAAC;QAChD,OAAO,EAAE,IAAI,CAAC,CAAC,CAAC,OAAO,EAAE,sBAAc,CAAC,OAAO,CAAC;KACjD,CAAC;AACJ,CAAC;AAED,0FAA0F;AAC1F,SAAgB,SAAS,CAAC,KAAc;IACtC,OAAO,OAAO,KAAK,KAAK,QAAQ,IAAI,MAAM,CAAC,QAAQ,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,oBAAY,CAAC;AAC3H,CAAC;AAED;;;GAGG;AACH,SAAgB,SAAS,CAAC,KAAa,EAAE,UAAkB;IACzD,OAAO,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,CAAC,SAAS,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC,GAAG,CAAC,UAAU,GAAG,EAAE,CAAC,CAAC,CAAC;AAClE,CAAC;AAED,wDAAwD;AACxD,SAAgB,cAAc,CAAC,OAA6C;IAC1E,IAAI,OAAO,KAAK,MAAM;QAAE,OAAO,MAAM,CAAC;IACtC,IAAI,OAAO,IAAI,OAAO,OAAO,KAAK,QAAQ,IAAI,OAAO,IAAI,OAAO,IAAI,OAAO,CAAC,KAAK;QAAE,OAAO,OAAO,CAAC;IAClG,OAAO,MAAM,CAAC;AAChB,CAAC;AAED;;;GAGG;AACH,SAAgB,QAAQ,CAAC,UAAkB,EAAE,WAAmB,EAAE,UAAkB,EAAE,WAAmB;IACvG,IAAI,CAAC,CAAC,UAAU,GAAG,CAAC,IAAI,WAAW,GAAG,CAAC,IAAI,UAAU,GAAG,CAAC,IAAI,WAAW,GAAG,CAAC,CAAC;QAAE,OAAO,EAAE,EAAE,EAAE,CAAC,EAAE,EAAE,EAAE,CAAC,EAAE,EAAE,EAAE,CAAC,EAAE,EAAE,EAAE,CAAC,EAAE,CAAC;IACrH,MAAM,KAAK,GAAG,UAAU,GAAG,WAAW,CAAC;IACvC,MAAM,KAAK,GAAG,UAAU,GAAG,WAAW,CAAC;IACvC,IAAI,KAAK,GAAG,KAAK,EAAE,CAAC;QAClB,+CAA+C;QAC/C,MAAM,EAAE,GAAG,KAAK,GAAG,KAAK,CAAC;QACzB,OAAO,EAAE,EAAE,EAAE,EAAE,EAAE,CAAC,EAAE,EAAE,EAAE,CAAC,CAAC,GAAG,EAAE,CAAC,GAAG,CAAC,EAAE,EAAE,EAAE,CAAC,EAAE,CAAC;IAChD,CAAC;IACD,MAAM,EAAE,GAAG,KAAK,GAAG,KAAK,CAAC;IACzB,OAAO,EAAE,EAAE,EAAE,CAAC,EAAE,EAAE,EAAE,EAAE,EAAE,CAAC,EAAE,EAAE,EAAE,CAAC,CAAC,GAAG,EAAE,CAAC,GAAG,CAAC,EAAE,CAAC;AAChD,CAAC;AAED,8GAA8G;AAC9G,SAAgB,gBAAgB,CAAC,KAAa,EAAE,MAAc,EAAE,QAAQ,GAAG,GAAG;IAC5E,IAAI,CAAC,CAAC,KAAK,GAAG,CAAC,IAAI,MAAM,GAAG,CAAC,CAAC;QAAE,OAAO,EAAE,KAAK,EAAE,QAAQ,EAAE,MAAM,EAAE,QAAQ,EAAE,CAAC;IAC7E,MAAM,KAAK,GAAG,QAAQ,GAAG,IAAI,CAAC,GAAG,CAAC,KAAK,EAAE,MAAM,CAAC,CAAC;IACjD,OAAO,EAAE,KAAK,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,KAAK,CAAC,KAAK,GAAG,KAAK,CAAC,CAAC,EAAE,MAAM,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,KAAK,CAAC,MAAM,GAAG,KAAK,CAAC,CAAC,EAAE,CAAC;AAC5G,CAAC;AAED,6GAA6G;AAChG,QAAA,WAAW,GAAG,IAAI,CAAC;AAEhC;;;GAGG;AACH,SAAgB,UAAU,CAAC,UAAwB,EAAE,MAA2B,EAAE,GAAe,EAAE,MAAM,GAAG,mBAAW;IACrH,MAAM,KAAK,GAAG,MAAM,KAAK,IAAI,IAAI,MAAM,CAAC,MAAM,KAAK,UAAU,CAAC,MAAM,CAAC;IACrE,MAAM,OAAO,GAAG,KAAK,CAAC,CAAC,CAAC,IAAI,YAAY,CAAC,UAAU,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC;IAC9D,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,UAAU,CAAC,MAAM,EAAE,CAAC,IAAI,CAAC,EAAE,CAAC;QAC9C,MAAM,KAAK,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,UAAU,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;QACtD,MAAM,IAAI,GAAG,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,GAAG,CAAC,KAAK,GAAG,OAAO,CAAC,CAAC,CAAC,CAAC,GAAG,MAAM,CAAC;QACxE,OAAO,CAAC,CAAC,CAAC,GAAG,IAAI,CAAC;QAClB,GAAG,CAAC,CAAC,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,IAAI,GAAG,GAAG,CAAC,CAAC;IAClC,CAAC;IACD,OAAO,OAAO,CAAC;AACjB,CAAC;AAED,mFAAmF;AACtE,QAAA,iBAAiB,GAAG,CAAC,CAAC;AAEnC;;;;GAIG;AACH,MAAa,iBAAiB;IAKC;IAJrB,OAAO,GAAkB,IAAI,CAAC;IAC9B,SAAS,GAAG,MAAM,CAAC,iBAAiB,CAAC;IACrC,KAAK,GAAG,CAAC,CAAC;IAElB,YAA6B,WAAW,yBAAiB;QAA5B,aAAQ,GAAR,QAAQ,CAAoB;IAAG,CAAC;IAE7D,kDAAkD;IAClD,IAAI,QAAQ;QACV,OAAO,IAAI,CAAC,KAAK,CAAC;IACpB,CAAC;IAED,mDAAmD;IACnD,IAAI,SAAS;QACX,OAAO,IAAI,CAAC,OAAO,IAAI,CAAC,CAAC;IAC3B,CAAC;IAED,aAAa,CAAC,UAAkB;QAC9B,OAAO,UAAU,GAAG,IAAI,CAAC,KAAK,KAAK,CAAC,CAAC;IACvC,CAAC;IAED,mFAAmF;IACnF,IAAI,CAAC,eAAuB;QAC1B,IAAI,IAAI,CAAC,OAAO,KAAK,IAAI,IAAI,CAAC,CAAC,eAAe,GAAG,CAAC,CAAC;YAAE,OAAO,CAAC,CAAC;QAC9D,OAAO,IAAI,CAAC,OAAO,GAAG,CAAC,IAAI,CAAC,KAAK,GAAG,eAAe,CAAC,CAAC;IACvD,CAAC;IAED,gEAAgE;IAChE,UAAU,CAAC,eAAuB;QAChC,OAAO,IAAI,CAAC,KAAK,IAAI,IAAI,CAAC,QAAQ,IAAI,IAAI,CAAC,IAAI,CAAC,eAAe,CAAC,GAAG,GAAG,CAAC;IACzE,CAAC;IAED,4EAA4E;IAC5E,MAAM,CAAC,EAAU,EAAE,eAAuB,EAAE,GAAW;QACrD,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,EAAE,CAAC,IAAI,EAAE,GAAG,CAAC;YAAE,OAAO;QAC3C,IAAI,CAAC,OAAO,GAAG,IAAI,CAAC,OAAO,KAAK,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,OAAO,GAAG,GAAG,GAAG,EAAE,GAAG,GAAG,CAAC;QAC1E,MAAM,IAAI,GAAG,IAAI,CAAC,IAAI,CAAC,eAAe,CAAC,CAAC;QACxC,IAAI,IAAI,GAAG,GAAG,IAAI,IAAI,CAAC,KAAK,GAAG,IAAI,CAAC,QAAQ,IAAI,GAAG,GAAG,IAAI,CAAC,SAAS,IAAI,IAAI,EAAE,CAAC;YAC7E,IAAI,CAAC,KAAK,IAAI,CAAC,CAAC;YAChB,IAAI,CAAC,SAAS,GAAG,GAAG,CAAC;QACvB,CAAC;aAAM,IAAI,IAAI,GAAG,IAAI,IAAI,IAAI,CAAC,KAAK,GAAG,CAAC,IAAI,GAAG,GAAG,IAAI,CAAC,SAAS,IAAI,IAAI,EAAE,CAAC;YACzE,IAAI,CAAC,KAAK,IAAI,CAAC,CAAC;YAChB,IAAI,CAAC,SAAS,GAAG,GAAG,CAAC;QACvB,CAAC;IACH,CAAC;CACF;AA7CD,8CA6CC;AAED,wFAAwF;AACxF,MAAa,cAAc;IACjB,IAAI,GAAG,CAAC,CAAC;IACjB,IAAI,CAAC,GAAW;QACd,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,IAAI,GAAG,CAAC,EAAE,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC;QACrD,OAAO,IAAI,CAAC,IAAI,CAAC;IACnB,CAAC;CACF;AAND,wCAMC"}

package/dist/cjs/effects-gl.d.ts

@@ -0,0 +1,79 @@
+/**
+ * The WebGL2 side of video effects (docs/implementation-progress/RTC_PERF_PROGRESS.md P3-5): the camera frame and the
+ * person mask in, the finished picture out on one canvas. Done on the GPU as LiveKit, Stream, Twilio and TRTC do:
+ *
+ * - **Blur** at a quarter of the size (cheap at any resolution), weighted by the mask so the person's own colours never
+ *   bleed into the blur around them -- no halo (Stream's mask-weighted blur) -- in two separable Gaussian passes.
+ * - **Edges** follow the person's outline rather than the mask's coarse grid: the mask is sampled jointly with the
+ *   frame's colours (a joint bilateral filter, as Stream and Twilio refine theirs).
+ * - **Background picture** fitted like `object-fit: cover`.
+ * - **Beauty** without a face model: skin found by its colour (YCbCr), smoothed with an edge-preserving filter so eyes
+ *   and lips stay sharp, brightened in the midtones, a touch rosier, and detail off the skin sharpened.
+ *
+ * Every texture is kept in picture order (the first row is the top); only the last pass, to the canvas, flips.
+ */
+import type { BeautySettings } from './effects-core.js';
+export interface RenderSettings {
+    mode: 'none' | 'blur' | 'image';
+    /** Gaussian sigma in quarter-size texels. */
+    blurSigma: number;
+    beauty: BeautySettings | null;
+    /** False until the first mask: everything is treated as background, so nothing behind the person shows meanwhile. */
+    maskReady: boolean;
+    /** Where the image texture maps (coverFit). */
+    cover: {
+        sx: number;
+        sy: number;
+        ox: number;
+        oy: number;
+    };
+}
+export declare class EffectsRenderer {
+    readonly canvas: HTMLCanvasElement | OffscreenCanvas;
+    private readonly gl;
+    private readonly downsample;
+    private readonly blur;
+    private readonly composite;
+    private readonly frameTexture;
+    private readonly maskTexture;
+    private readonly pictureTexture;
+    private readonly blurTextures;
+    private readonly blurBuffers;
+    private readonly vertices;
+    private readonly floatTargets;
+    private width;
+    private height;
+    private blurWidth;
+    private blurHeight;
+    private maskWidth;
+    private maskHeight;
+    private pictureSize;
+    constructor(canvas: HTMLCanvasElement | OffscreenCanvas);
+    get lost(): boolean;
+    /** The picture's size; the canvas and the blur follow it. */
+    setSize(width: number, height: number): void;
+    get size(): {
+        width: number;
+        height: number;
+        blurHeight: number;
+    };
+    /** The camera frame (a VideoFrame or a playing video). */
+    uploadFrame(source: TexImageSource): void;
+    /** The person mask, one byte per pixel, picture order. */
+    uploadMask(data: Uint8Array, width: number, height: number): void;
+    /** The background picture, or null to let it go. */
+    setPicture(image: TexImageSource | null, width?: number, height?: number): void;
+    get picture(): {
+        width: number;
+        height: number;
+    } | null;
+    /** Draw the finished picture onto the canvas. */
+    render(settings: RenderSettings): void;
+    /** Let the GPU resources go. */
+    dispose(): void;
+    private pass;
+    private bindTexture;
+    private texture;
+    private program;
+}
+//# sourceMappingURL=effects-gl.d.ts.map

package/dist/cjs/effects-gl.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"effects-gl.d.ts","sourceRoot":"","sources":["../../src/effects-gl.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;GAaG;AACH,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,mBAAmB,CAAC;AAuIxD,MAAM,WAAW,cAAc;IAC7B,IAAI,EAAE,MAAM,GAAG,MAAM,GAAG,OAAO,CAAC;IAChC,6CAA6C;IAC7C,SAAS,EAAE,MAAM,CAAC;IAClB,MAAM,EAAE,cAAc,GAAG,IAAI,CAAC;IAC9B,qHAAqH;IACrH,SAAS,EAAE,OAAO,CAAC;IACnB,+CAA+C;IAC/C,KAAK,EAAE;QAAE,EAAE,EAAE,MAAM,CAAC;QAAC,EAAE,EAAE,MAAM,CAAC;QAAC,EAAE,EAAE,MAAM,CAAC;QAAC,EAAE,EAAE,MAAM,CAAA;KAAE,CAAC;CAC3D;AAOD,qBAAa,eAAe;IAC1B,QAAQ,CAAC,MAAM,EAAE,iBAAiB,GAAG,eAAe,CAAC;IACrD,OAAO,CAAC,QAAQ,CAAC,EAAE,CAAyB;IAC5C,OAAO,CAAC,QAAQ,CAAC,UAAU,CAAU;IACrC,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAU;IAC/B,OAAO,CAAC,QAAQ,CAAC,SAAS,CAAU;IACpC,OAAO,CAAC,QAAQ,CAAC,YAAY,CAAe;IAC5C,OAAO,CAAC,QAAQ,CAAC,WAAW,CAAe;IAC3C,OAAO,CAAC,QAAQ,CAAC,cAAc,CAAe;IAC9C,OAAO,CAAC,QAAQ,CAAC,YAAY,CAA+B;IAC5D,OAAO,CAAC,QAAQ,CAAC,WAAW,CAAuC;IACnE,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAc;IACvC,OAAO,CAAC,QAAQ,CAAC,YAAY,CAAU;IACvC,OAAO,CAAC,KAAK,CAAK;IAClB,OAAO,CAAC,MAAM,CAAK;IACnB,OAAO,CAAC,SAAS,CAAK;IACtB,OAAO,CAAC,UAAU,CAAK;IACvB,OAAO,CAAC,SAAS,CAAK;IACtB,OAAO,CAAC,UAAU,CAAK;IACvB,OAAO,CAAC,WAAW,CAAkD;gBAEzD,MAAM,EAAE,iBAAiB,GAAG,eAAe;IA0BvD,IAAI,IAAI,IAAI,OAAO,CAElB;IAED,6DAA6D;IAC7D,OAAO,CAAC,KAAK,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,GAAG,IAAI;IAmB5C,IAAI,IAAI,IAAI;QAAE,KAAK,EAAE,MAAM,CAAC;QAAC,MAAM,EAAE,MAAM,CAAC;QAAC,UAAU,EAAE,MAAM,CAAA;KAAE,CAEhE;IAED,0DAA0D;IAC1D,WAAW,CAAC,MAAM,EAAE,cAAc,GAAG,IAAI;IAOzC,0DAA0D;IAC1D,UAAU,CAAC,IAAI,EAAE,UAAU,EAAE,KAAK,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,GAAG,IAAI;IASjE,oDAAoD;IACpD,UAAU,CAAC,KAAK,EAAE,cAAc,GAAG,IAAI,EAAE,KAAK,SAAI,EAAE,MAAM,SAAI,GAAG,IAAI;IAarE,IAAI,OAAO,IAAI;QAAE,KAAK,EAAE,MAAM,CAAC;QAAC,MAAM,EAAE,MAAM,CAAA;KAAE,GAAG,IAAI,CAEtD;IAED,iDAAiD;IACjD,MAAM,CAAC,QAAQ,EAAE,cAAc,GAAG,IAAI;IA6CtC,gCAAgC;IAChC,OAAO,IAAI,IAAI;IASf,OAAO,CAAC,IAAI;IAaZ,OAAO,CAAC,WAAW;IAOnB,OAAO,CAAC,OAAO;IAWf,OAAO,CAAC,OAAO;CAoBhB"}