@pexip/media-processor 20.3.5 → 22.0.0

package/CHANGELOG.md

@@ -1,5 +1,114 @@
 # Changelog
 
+## 22.0.0
+
+### Major Changes
+
+- 498ff3b: A full architectural overhaul of the video processing pipeline:
+
+  BREAKING CHANGES:
+
+  - **`createCanvasTransform` removed**
+  - **`createVideoTrackProcessor` / `createVideoTrackProcessorWithFallback`
+    removed**
+  - **`createVideoProcessor` redesigned** — now accepts a `Segmenter` +
+    `InputProcessAdaptor` instead of an array of `Transform`s. Exposes
+    `update()`, `mute()`, `status`, `renderOptions`, and processing dimensions.
+  - **New `InputProcessAdaptor` system**:
+    - `createMediaStreamTrackInputProcessAdaptor` — uses native
+      `MediaStreamTrack` transfer (Safari v18+ compatible).
+    - `createReadWritableStreamInputProcessAdaptor` — uses
+      `MediaStreamTrackProcessor` + `MediaStreamTrackGenerator`.
+    - `createReadWritableStreamFallbackInputProcessAdaptor` — canvas/video
+      element fallback.
+    - `getDefaultInputProcessAdaptor` — picks the best path via feature
+      detection (`MediaStreamTrackTransferable`).
+  - **`createSegmenter` updated**
+    - Now processes `MediaStreamTrack` or `ReadWritableStream` directly (no more
+      `VideoFrameLike`).
+    - Added `mute()` method.
+    - Added `stats` callback for frame timing data.
+    - Worker is no longer re-created on every `open()`.
+  - **Worker rewritten**:
+    - Processing now uses `TransformStream` to pipe `VideoFrame`s through
+      segmentation.
+    - Supports both track-based
+      (`VideoTrackGenerator`/`MediaStreamTrackProcessor`) and ReadWritable
+      stream processing.
+    - Added frame timing stats (`FrameTransformStats`) batched and posted back
+      to main thread.
+    - Removes direct canvas-copy output (`directOutput`/`output`); renderer now
+      writes to `processCanvas` and creates a new `VideoFrame`.
+    - `close` and `suspend` separated; resource cleanup improved.
+    - Background image fetching is now abortable.
+  - **`Renderer.render()` signature simplified** — accepts `OffscreenCanvas` +
+    `VideoFrame` directly (no `VideoFrameLike` wrapper).
+  - **WebGL/WebGPU texture renderers** updated to accept
+    `VideoFrame | ImageBitmap | OffscreenCanvas` natively.
+  - **WebGL program release guard** added to prevent double-free.
+  - **`benchUtils`** extended to accept explicit timestamps in
+    `begin()`/`end()` + tests added.
+  - **`MediaStreamTrackTransferable`** feature-detection helper exported.
+  - **Default frame rate raised** from 20 → 30 fps.
+  - **Default backend changed** to `webgpu`.
+
+- fe0a0a4: Set a max width of 300px of in meeting selfview.
+
+  Rename openEffectsModal prop on InMeetingSelfview to openEffects
+
+  Make selfview muted indicator look more like an indicator
+
+  Show tooltips for background effects button and upload background effect
+
+### Patch Changes
+
+- f8bd1c5: Fix VideoProcessor and Segment open/update behavior
+- 925cce6: Fix f16 issue
+- fb41d4d: Changes related to Typescript V6 Upgrade
+- Updated dependencies [498ff3b]
+- Updated dependencies [fb41d4d]
+- Updated dependencies [b99c070]
+  - @pexip/utils@17.3.0
+
+## 21.0.0
+
+### Major Changes
+
+- 3963c2e: Post-processing the segmentation mask
+
+  - Add downsampling post-processing
+  - Refine mask boundaries for more edge snappy
+  - Reduce the gaps between the segmented areas
+  - Add bystander removal
+
+### Minor Changes
+
+- 4cc4701: WebGPU implementation improvements:
+
+  - Port WebGL2 post-processing steps to WebGPU, matching mask/edge quality and
+    performance
+  - Implement compute-based separable joint bilateral filter for mask refinement
+  - Add compute tent blur for edge feathering, using single half-float channel
+    for mask
+  - Introduce downsample/upsample and dual filter blur in compute for optimized
+    background effects
+  - Cache pipelines and textures to reduce per-frame allocation overhead
+  - Use `r16float` mask storage when available for efficiency
+  - Improve blend quality with smoothstep edge softening
+  - Integrate light wrap and H.264 noise stabilization into compositing
+
+### Patch Changes
+
+- 24bc717: Fix added noise YUV->RBG negative value issue
+- 872971f: Mark `excludeBystanders` related params as `@experimental`:
+  `personCenter`, `morphErodeRadiusPx`, `morphPass`, and `morphDilateRadiusPx`
+- 2e895ac: Add add an `@experimental` tag for `excludeBystanders`
+- 4eaf9d7: Use the correct region to apply noise
+- Updated dependencies [1fff975]
+- Updated dependencies [3963c2e]
+- Updated dependencies [762ebd2]
+  - @pexip/utils@17.2.0
+
 ## 20.3.5
 
 ### Patch Changes

package/README.md

@@ -106,10 +106,12 @@ await audioGraph.release();
 
 ```typescript
 import {
+  createMediaStreamTrackInputProcessAdaptor,
+  createReadWritableStreamFallbackInputProcessAdaptor,
+  createReadWritableStreamInputProcessAdaptor,
   createSegmenter,
-  createCanvasTransform,
-  createVideoTrackProcessor,
-  createVideoTrackProcessorWithFallback,
+  createVideoProcessor,
+  getDefaultInputProcessAdaptor,
 } from '@pexip/media-processor';
 
 // Grab the user's camera stream
@@ -129,17 +131,22 @@ const modelAsset = {
   modelName: 'selfie' as const,
 };
 
-const segmenter = createSegmenter(tasksVisionBasePath, {modelAssets});
-// Create a processing transformer and set the effects to `blur`
-const transformer = createCanvasTransform(segmenter, {effects: 'blur'});
-const processor = createVideoProcessor([transformer]);
+const segmenter = createSegmenter(tasksVisionBasePath, {modelAsset});
+// Choose the appropriate input process adaptor based on browser capabilities
+const inputProcessAdaptor = getDefaultInputProcessAdaptor(true);
+// Create a processor and set the effects to `blur`
+const processor = createVideoProcessor(segmenter, inputProcessAdaptor);
 
 // Start the processor
 await videoProcessor.open();
 
-// Passing the raw MediaStream to apply the effects
-// Then, use the output stream for whatever purpose
-const processedStream = await videoProcessor.process(stream);
+// Start the processor with blur effects
+await processor.open({renderOptions: {effects: 'blur'}});
+
+// Process the video track and create a new stream
+const videoTrack = stream.getVideoTracks()[0];
+const processedTrack = await processor.process(videoTrack);
+const processedStream = new MediaStream([processedTrack]);
 ```
 
 ## Profiling Web Audio

package/api-docs/README.mdx

@@ -106,10 +106,12 @@ await audioGraph.release();
 
 ```typescript
 import {
+  createMediaStreamTrackInputProcessAdaptor,
+  createReadWritableStreamFallbackInputProcessAdaptor,
+  createReadWritableStreamInputProcessAdaptor,
   createSegmenter,
-  createCanvasTransform,
-  createVideoTrackProcessor,
-  createVideoTrackProcessorWithFallback,
+  createVideoProcessor,
+  getDefaultInputProcessAdaptor,
 } from '@pexip/media-processor';
 
 // Grab the user's camera stream
@@ -129,17 +131,22 @@ const modelAsset = {
   modelName: 'selfie' as const,
 };
 
-const segmenter = createSegmenter(tasksVisionBasePath, {modelAssets});
-// Create a processing transformer and set the effects to `blur`
-const transformer = createCanvasTransform(segmenter, {effects: 'blur'});
-const processor = createVideoProcessor([transformer]);
+const segmenter = createSegmenter(tasksVisionBasePath, {modelAsset});
+// Choose the appropriate input process adaptor based on browser capabilities
+const inputProcessAdaptor = getDefaultInputProcessAdaptor(true);
+// Create a processor and set the effects to `blur`
+const processor = createVideoProcessor(segmenter, inputProcessAdaptor);
 
 // Start the processor
 await videoProcessor.open();
 
-// Passing the raw MediaStream to apply the effects
-// Then, use the output stream for whatever purpose
-const processedStream = await videoProcessor.process(stream);
+// Start the processor with blur effects
+await processor.open({renderOptions: {effects: 'blur'}});
+
+// Process the video track and create a new stream
+const videoTrack = stream.getVideoTracks()[0];
+const processedTrack = await processor.process(videoTrack);
+const processedStream = new MediaStream([processedTrack]);
 ```
 
 ## Profiling Web Audio
@@ -193,11 +200,13 @@ A library for media analysis using Web APIs.
 | Interface | Description |
 | ------ | ------ |
 | [Segmentation](interfaces/Segmentation.mdx) | - |
+| [ReadWritableStream](interfaces/ReadWritableStream.mdx) | - |
 | [VideoFrameLike](interfaces/VideoFrameLike.mdx) | - |
 | [ImageRecord](interfaces/ImageRecord.mdx) | - |
 | [Rect](interfaces/Rect.mdx) | - |
 | [ProcessEventOpen](interfaces/ProcessEventOpen.mdx) | Message event to initializing the Processor |
 | [ProcessEventProcess](interfaces/ProcessEventProcess.mdx) | Message event to process the video frame |
+| [ProcessEventMute](interfaces/ProcessEventMute.mdx) | - |
 | [ProcessEventUpdate](interfaces/ProcessEventUpdate.mdx) | Message event to update processor options |
 | [ProcessEventClose](interfaces/ProcessEventClose.mdx) | Message event to stop the processing |
 | [ProcessEventDestroy](interfaces/ProcessEventDestroy.mdx) | Message event to release all the resources |
@@ -207,11 +216,13 @@ A library for media analysis using Web APIs.
 | [ProcessWorkerEventUpdated](interfaces/ProcessWorkerEventUpdated.mdx) | - |
 | [ProcessWorkerEvent](interfaces/ProcessWorkerEvent.mdx) | - |
 | [ProcessWorkerEventDebug](interfaces/ProcessWorkerEventDebug.mdx) | - |
+| [FrameTransformStats](interfaces/FrameTransformStats.mdx) | Frame transform statistic, i.e. the timing data for processing each frame |
+| [ProcessWorkerEventStats](interfaces/ProcessWorkerEventStats.mdx) | - |
 | [ProcessorOptions](interfaces/ProcessorOptions.mdx) | - |
 | [ProcessorProcessOptions](interfaces/ProcessorProcessOptions.mdx) | - |
 | [ProcessWorkerDebugMessage](interfaces/ProcessWorkerDebugMessage.mdx) | - |
 | [ProcessorEvent](interfaces/ProcessorEvent.mdx) | - |
-| [RendererOptions](interfaces/RendererOptions.mdx) | - |
+| [RendererOptions](interfaces/RendererOptions.mdx) | Options controlling the rendering and effects for person/background segmentation. All values are stable per-frame and can be safely updated in real-time. |
 | [Renderer](interfaces/Renderer.mdx) | - |
 | [RenderEventHandlers](interfaces/RenderEventHandlers.mdx) | - |
 | [SegmentationModelAsset](interfaces/SegmentationModelAsset.mdx) | Segmentation model asset that can be used to load a segmentation model. |
@@ -219,6 +230,9 @@ A library for media analysis using Web APIs.
 | [SegmenterOptions](interfaces/SegmenterOptions.mdx) | - |
 | [SegmentationSmoothingConfig](interfaces/SegmentationSmoothingConfig.mdx) | - |
 | [MPMask](interfaces/MPMask.mdx) | Wrapper for a mask produced by a Segmentation Task. |
+| [Stats](interfaces/Stats.mdx) | - |
+| [Weights](interfaces/Weights.mdx) | - |
+| [SelectionOptions](interfaces/SelectionOptions.mdx) | - |
 | [Benchmark](interfaces/Benchmark.mdx) | - |
 | [Point](interfaces/Point.mdx) | Interface for Point consist of coordinates x and y |
 | [Size](interfaces/Size.mdx) | - |
@@ -239,13 +253,12 @@ A library for media analysis using Web APIs.
 | [Clock](interfaces/Clock.mdx) | Clock interface to get the current time with now method, |
 | [ThrottleOptions](interfaces/ThrottleOptions.mdx) | Limit the rate of flow in terms of millisecond, and provided Clock |
 | [Runner](interfaces/Runner.mdx) | - |
-| [Transform](interfaces/Transform.mdx) | - |
 | [AsyncAssets](interfaces/AsyncAssets.mdx) | - |
 | [Process](interfaces/Process.mdx) | - |
 | [BackgroundImage](interfaces/BackgroundImage.mdx) | - |
 | [Segmenter](interfaces/Segmenter.mdx) | - |
-| [Detector](interfaces/Detector.mdx) | - |
-| [SegmentationParams](interfaces/SegmentationParams.mdx) | - |
+| [InputProcessAdaptor](interfaces/InputProcessAdaptor.mdx) | - |
+| [ProcessorUpdateOptions](interfaces/ProcessorUpdateOptions.mdx) | - |
 | [VideoProcessor](interfaces/VideoProcessor.mdx) | - |
 
 ## Type Aliases
@@ -263,11 +276,13 @@ A library for media analysis using Web APIs.
 | [ProcessWorkerEvents](type-aliases/ProcessWorkerEvents.mdx) | Message events returned from the processor |
 | [ProcessorEvents](type-aliases/ProcessorEvents.mdx) | - |
 | [ProcessorWorkerEvents](type-aliases/ProcessorWorkerEvents.mdx) | - |
+| [Struct](type-aliases/Struct.mdx) | - |
 | [SegmentationModel](type-aliases/SegmentationModel.mdx) | - |
 | [Delegate](type-aliases/Delegate.mdx) | - |
 | [ExtractMessageEventType](type-aliases/ExtractMessageEventType.mdx) | - |
 | [IncludeMessageEventDataType](type-aliases/IncludeMessageEventDataType.mdx) | - |
 | [TupleOf](type-aliases/TupleOf.mdx) | From https://github.com/Microsoft/TypeScript/issues/26223#issuecomment-674500430 |
+| [OptionalKeys](type-aliases/OptionalKeys.mdx) | - |
 | [Canvas](type-aliases/Canvas.mdx) | - |
 | [Unsubscribe](type-aliases/Unsubscribe.mdx) | Unsubscribe the subscription |
 | [AudioBufferFloats](type-aliases/AudioBufferFloats.mdx) | Same as [AudioBuffer](https://developer.mozilla.org/en-US/docs/Web/API/AudioBuffer) Or the return from [AnalyserNode.getFloatFrequencyData()](https://developer.mozilla.org/en-US/docs/Web/API/AnalyserNode/getFloatFrequencyData) |
@@ -305,35 +320,55 @@ A library for media analysis using Web APIs.
 | [WasmPaths](type-aliases/WasmPaths.mdx) | - |
 | [Color](type-aliases/Color.mdx) | - |
 | [InputFrame](type-aliases/InputFrame.mdx) | - |
-| [SegmentationTransform](type-aliases/SegmentationTransform.mdx) | - |
-| [ProcessVideoTrack](type-aliases/ProcessVideoTrack.mdx) | - |
 
 ## Variables
 
 | Variable | Description |
 | ------ | ------ |
-| [PROCESSING\_WIDTH](variables/PROCESSING_WIDTH.mdx) | - |
-| [PROCESSING\_HEIGHT](variables/PROCESSING_HEIGHT.mdx) | - |
-| [FOREGROUND\_THRESHOLD](variables/FOREGROUND_THRESHOLD.mdx) | - |
+| [getCanvasRenderingContext2D](variables/getCanvasRenderingContext2D.mdx) | - |
+| [getImageSize](variables/getImageSize.mdx) | - |
 | [BACKGROUND\_BLUR\_AMOUNT](variables/BACKGROUND_BLUR_AMOUNT.mdx) | - |
+| [BACKGROUND\_THRESHOLD](variables/BACKGROUND_THRESHOLD.mdx) | - |
+| [DOWN\_SAMPLE\_FACTOR](variables/DOWN_SAMPLE_FACTOR.mdx) | - |
 | [EDGE\_BLUR\_AMOUNT](variables/EDGE_BLUR_AMOUNT.mdx) | - |
+| [EXCLUDE\_BYSTANDERS](variables/EXCLUDE_BYSTANDERS.mdx) | - |
+| [FOREGROUND\_THRESHOLD](variables/FOREGROUND_THRESHOLD.mdx) | - |
+| [LIGHT\_WRAP\_BLUR\_AMOUNT](variables/LIGHT_WRAP_BLUR_AMOUNT.mdx) | - |
+| [LIGHT\_WRAP\_EDGE\_BAND](variables/LIGHT_WRAP_EDGE_BAND.mdx) | - |
+| [LIGHT\_WRAP\_INTENSITY](variables/LIGHT_WRAP_INTENSITY.mdx) | - |
+| [LIGHT\_WRAP\_TIGHTNESS](variables/LIGHT_WRAP_TIGHTNESS.mdx) | - |
 | [MASK\_COMBINE\_RATIO](variables/MASK_COMBINE_RATIO.mdx) | - |
+| [MORPH\_DILATE\_RADIUS\_PX](variables/MORPH_DILATE_RADIUS_PX.mdx) | - |
+| [MORPH\_ERODE\_RADIUS\_PX](variables/MORPH_ERODE_RADIUS_PX.mdx) | - |
+| [MORPH\_PASS](variables/MORPH_PASS.mdx) | - |
+| [PERSON\_CENTER](variables/PERSON_CENTER.mdx) | - |
+| [PROCESSING\_HEIGHT](variables/PROCESSING_HEIGHT.mdx) | - |
+| [PROCESSING\_WIDTH](variables/PROCESSING_WIDTH.mdx) | - |
+| [SIGMA\_RANGE](variables/SIGMA_RANGE.mdx) | - |
+| [SIGMA\_SPACE](variables/SIGMA_SPACE.mdx) | - |
+| [STRONG\_EDGE\_BLUR\_AMOUNT](variables/STRONG_EDGE_BLUR_AMOUNT.mdx) | - |
 | [PROCESS\_STATUS](variables/PROCESS_STATUS.mdx) | Process status |
 | [RENDER\_EFFECT](variables/RENDER_EFFECT.mdx) | - |
 | [RENDER\_BACKEND](variables/RENDER_BACKEND.mdx) | - |
 | [RENDERING\_EVENTS](variables/RENDERING_EVENTS.mdx) | - |
+| [isValidGpuAPI](variables/isValidGpuAPI.mdx) | - |
+| [isRenderOptionsKey](variables/isRenderOptionsKey.mdx) | - |
+| [RENDERER\_OPTIONS\_STRUCT](variables/RENDERER_OPTIONS_STRUCT.mdx) | - |
 | [SEGMENTATION\_MODELS](variables/SEGMENTATION_MODELS.mdx) | - |
 | [isSegmentationModel](variables/isSegmentationModel.mdx) | - |
 | [isRenderingEvents](variables/isRenderingEvents.mdx) | - |
 | [toRenderingEvents](variables/toRenderingEvents.mdx) | - |
 | [getErrorMessage](variables/getErrorMessage.mdx) | - |
 | [createObjectUpdater](variables/createObjectUpdater.mdx) | - |
+| [ImageFetchError](variables/ImageFetchError.mdx) | - |
 | [createRemoteImageBitmap](variables/createRemoteImageBitmap.mdx) | - |
 | [clamping](variables/clamping.mdx) | - |
 | [calculateMaxBlurPass](variables/calculateMaxBlurPass.mdx) | - |
 | [resize](variables/resize.mdx) | Calculate the coordinates and size of the source and destination to fit the target aspect ratio. |
 | [createLazyProps](variables/createLazyProps.mdx) | - |
 | [handleWebGLContextLoss](variables/handleWebGLContextLoss.mdx) | - |
+| [cloneImageRecord](variables/cloneImageRecord.mdx) | - |
+| [compareImageRecords](variables/compareImageRecords.mdx) | Compare provided ImageRecord |
 | [urls](variables/urls.mdx) | - |
 | [SILENT\_THRESHOLD](variables/SILENT_THRESHOLD.mdx) | Default silent threshold At least one LSB 16-bit data (compare is on absolute value). |
 | [MONO\_THRESHOLD](variables/MONO_THRESHOLD.mdx) | Default mono detection threshold Data must be identical within one LSB 16-bit to be identified as mono. |
@@ -341,6 +376,7 @@ A library for media analysis using Web APIs.
 | [CLIP\_THRESHOLD](variables/CLIP_THRESHOLD.mdx) | Default clipping detection threshold |
 | [VOICE\_PROBABILITY\_THRESHOLD](variables/VOICE_PROBABILITY_THRESHOLD.mdx) | Default Voice probability threshold |
 | [CLIP\_COUNT\_THRESHOLD](variables/CLIP_COUNT_THRESHOLD.mdx) | Default clipping count threshold Number of consecutive clipThreshold level samples that indicate clipping. |
+| [MediaStreamTrackTransferable](variables/MediaStreamTrackTransferable.mdx) | Result of the feature detection if `MediaStreamTrack` is `Transferable` |
 | [FRAME\_RATE](variables/FRAME_RATE.mdx) | - |
 | [PLAY\_VIDEO\_TIMEOUT](variables/PLAY_VIDEO_TIMEOUT.mdx) | - |
 
@@ -348,6 +384,10 @@ A library for media analysis using Web APIs.
 
 | Function | Description |
 | ------ | ------ |
+| [nearestPowerOfTwo](functions/nearestPowerOfTwo.mdx) | Coerce any positive number to the nearest power of two within [minPow, maxPow] Example: nearestPowerOfTwo(13) => 16; nearestPowerOfTwo(3.2) => 4 |
+| [iou](functions/iou.mdx) | Intersection over Union (IoU), how much two masks overlap as a fraction of either's total. 1.0 = perfect match, 0 = no overlap. Used for stable, robust tracking of the main person between frames, especially with bystander removal and temporal instance switching. |
+| [labelComponents](functions/labelComponents.mdx) | - |
+| [scoreComponent](functions/scoreComponent.mdx) | - |
 | [createAudioContext](functions/createAudioContext.mdx) | A function to create `AudioContext` using constructor or factory function depends on the browser supports |
 | [resumeAudioOnInterruption](functions/resumeAudioOnInterruption.mdx) | Resume the stream whenever interrupted |
 | [resumeAudioOnUnmute](functions/resumeAudioOnUnmute.mdx) | Resume the AudioContext whenever the source track is unmuted |
@@ -366,7 +406,8 @@ A library for media analysis using Web APIs.
 | [createChannelMergerGraphNode](functions/createChannelMergerGraphNode.mdx) | Create a ChannelMergerNode |
 | [createAudioGraph](functions/createAudioGraph.mdx) | Accepts AudioNodeInitConnections to build the audio graph within a signal audio context |
 | [createAudioGraphProxy](functions/createAudioGraphProxy.mdx) | - |
-| [createBenchmark](functions/createBenchmark.mdx) | - |
+| [createBenchmark](functions/createBenchmark.mdx) | Creates a benchmarking utility for measuring frame durations and calculating frames per second (FPS). |
+| [createWindowedStats](functions/createWindowedStats.mdx) | - |
 | [sum](functions/sum.mdx) | Sum an array of numbers |
 | [avg](functions/avg.mdx) | Average an array of numbers |
 | [pow](functions/pow.mdx) | pow function from Math in functional form `number -> number -> number` |
@@ -394,16 +435,21 @@ A library for media analysis using Web APIs.
 | [isAudioNodeInit](functions/isAudioNodeInit.mdx) | - |
 | [isAnalyzerNodeInit](functions/isAnalyzerNodeInit.mdx) | - |
 | [createAsyncCallbackLoop](functions/createAsyncCallbackLoop.mdx) | Create an async callback loop to be called recursively with delay based on the `frameRate` |
-| [createCanvasTransform](functions/createCanvasTransform.mdx) | - |
+| [createReadWritableStreamInputProcessAdaptor](functions/createReadWritableStreamInputProcessAdaptor.mdx) | An implementation of `InputProcessAdaptor` which uses `MediaStreamTrackGenerator` and `MediaStreamTrackProcessor` APIs to create `ReadableStream` & `WritableStream` |
+| [createReadWritableStreamFallbackInputProcessAdaptor](functions/createReadWritableStreamFallbackInputProcessAdaptor.mdx) | An implementation of `InputProcessAdaptor` which uses `HTMLCanvasElement` or `OffscreenCanvas` API to create `ReadableStream` & `WritableStream` |
+| [createMediaStreamTrackInputProcessAdaptor](functions/createMediaStreamTrackInputProcessAdaptor.mdx) | An implementation of `InputProcessAdaptor` which adapts `MediaStreamTrack` directly as specified from the spec https://w3c.github.io/mediacapture-transform |
+| [getDefaultInputProcessAdaptor](functions/getDefaultInputProcessAdaptor.mdx) | Get the supported InputProcessAdaptor based on feature detection and preference provided |
 | [loadScript](functions/loadScript.mdx) | - |
 | [loadWasms](functions/loadWasms.mdx) | - |
 | [createSegmenter](functions/createSegmenter.mdx) | - |
 | [isRenderEffects](functions/isRenderEffects.mdx) | - |
+| [isMediaStreamVideoTrack](functions/isMediaStreamVideoTrack.mdx) | - |
+| [assertMediaStreamVideoTrack](functions/assertMediaStreamVideoTrack.mdx) | - |
 | [createCanvas](functions/createCanvas.mdx) | Create a Canvas element with provided width and height |
+| [setVideoElementSrc](functions/setVideoElementSrc.mdx) | - |
+| [playVideo](functions/playVideo.mdx) | - |
 | [createFrameCallbackRequest](functions/createFrameCallbackRequest.mdx) | Create a callback loop for video frame processing using `requestVideoFrameCallback` under-the-hood when available otherwise our fallback implementation based on `setTimeout`. |
 | [createVideoProcessor](functions/createVideoProcessor.mdx) | - |
-| [createVideoTrackProcessor](functions/createVideoTrackProcessor.mdx) | Video track processor using MediaStreamTrackProcessor API |
-| [createVideoTrackProcessorWithFallback](functions/createVideoTrackProcessorWithFallback.mdx) | Video track processor using Canvas[captureStream] API |
 | [calculateDistance](functions/calculateDistance.mdx) | Calculate the distance between two Points |
 | [getBezierCurveControlPoints](functions/getBezierCurveControlPoints.mdx) | Spline Interpolation for Bezier Curve |
 | [line](functions/line.mdx) | Create a straight line path command |

package/api-docs/functions/assertMediaStreamVideoTrack.mdx

@@ -0,0 +1,13 @@
+```ts
+function assertMediaStreamVideoTrack(t): asserts t is MediaStreamVideoTrack;
+```
+
+## Parameters
+
+| Parameter | Type |
+| ------ | ------ |
+| `t` | `MediaStreamTrack` |
+
+## Returns
+
+`asserts t is MediaStreamVideoTrack`

package/api-docs/functions/createBenchmark.mdx

@@ -1,14 +1,26 @@
 ```ts
-function createBenchmark(clock, __namedParameters): Benchmark;
+function createBenchmark(clock, options): Benchmark;
 ```
 
+Creates a benchmarking utility for measuring frame durations and calculating frames per second (FPS).
+
 ## Parameters
 
-| Parameter | Type | Default value |
-| ------ | ------ | ------ |
-| `clock` | [`Clock`](../interfaces/Clock.mdx) | `performance` |
-| `__namedParameters` | `BenchmarkOptions` | `{}` |
+| Parameter | Type | Default value | Description |
+| ------ | ------ | ------ | ------ |
+| `clock` | [`Clock`](../interfaces/Clock.mdx) | `performance` | An object with a `now()` method returning the current timestamp (in milliseconds). Defaults to `window.performance`. |
+| `options` | `BenchmarkOptions` | `{}` | Optional configuration options for the benchmark utility. |
 
 ## Returns
 
 [`Benchmark`](../interfaces/Benchmark.mdx)
+
+An object with methods to start/end frame measurements, reset the benchmark, and calculate current FPS.
+
+## Remarks
+
+This function is designed for performance measurement in animation or game loops.
+It records frame start/end times, ignores an initial warmup period, utilizes a circular buffer
+to average recent frame durations, and computes a smoothed FPS value on demand.
+The calculation may only occur if sufficient samples have been gathered and the minimum interval between
+calculations has elapsed.

package/api-docs/functions/createFrameCallbackRequest.mdx

@@ -1,5 +1,5 @@
 ```ts
-function createFrameCallbackRequest(
+function createFrameCallbackRequest<T>(
    callback, 
    frameRate, 
    __namedParameters): object;
@@ -9,11 +9,17 @@ Create a callback loop for video frame processing using
 `requestVideoFrameCallback` under-the-hood when available otherwise our
 fallback implementation based on `setTimeout`.
 
+## Type Parameters
+
+| Type Parameter |
+| ------ |
+| `T` |
+
 ## Parameters
 
 | Parameter | Type | Description |
 | ------ | ------ | ------ |
-| `callback` | [`Callback`](../type-aliases/Callback.mdx)\<`Promise`\<`void`\>, \[[`ProcessInputType`](../type-aliases/ProcessInputType.mdx)\]\> | To be called by the loop |
+| `callback` | [`Callback`](../type-aliases/Callback.mdx)\<`Promise`\<`void`\>, \[`T`\]\> | To be called by the loop |
 | `frameRate` | `number` | A fallback frame rate when we are not able to get the rate from API |
 | `__namedParameters` | `FrameCallbackRequestOptions` | - |
 

package/api-docs/functions/createMediaStreamTrackInputProcessAdaptor.mdx

@@ -0,0 +1,10 @@
+```ts
+function createMediaStreamTrackInputProcessAdaptor(): InputProcessAdaptor;
+```
+
+An implementation of `InputProcessAdaptor` which adapts `MediaStreamTrack`
+directly as specified from the spec https://w3c.github.io/mediacapture-transform
+
+## Returns
+
+[`InputProcessAdaptor`](../interfaces/InputProcessAdaptor.mdx)

package/api-docs/functions/createReadWritableStreamFallbackInputProcessAdaptor.mdx

@@ -0,0 +1,17 @@
+```ts
+function createReadWritableStreamFallbackInputProcessAdaptor(__namedParameters): InputProcessAdaptor;
+```
+
+An implementation of `InputProcessAdaptor` which uses `HTMLCanvasElement` or
+`OffscreenCanvas` API to create `ReadableStream` & `WritableStream`
+
+## Parameters
+
+| Parameter | Type | Description |
+| ------ | ------ | ------ |
+| `__namedParameters` | \{ `frameRate`: `number`; \} | - |
+| `__namedParameters.frameRate` | `number` | Fallback frame rate |
+
+## Returns
+
+[`InputProcessAdaptor`](../interfaces/InputProcessAdaptor.mdx)

package/api-docs/functions/createReadWritableStreamInputProcessAdaptor.mdx

@@ -0,0 +1,10 @@
+```ts
+function createReadWritableStreamInputProcessAdaptor(): InputProcessAdaptor;
+```
+
+An implementation of `InputProcessAdaptor` which uses `MediaStreamTrackGenerator` and
+`MediaStreamTrackProcessor` APIs to create `ReadableStream` & `WritableStream`
+
+## Returns
+
+[`InputProcessAdaptor`](../interfaces/InputProcessAdaptor.mdx)

package/api-docs/functions/createVideoProcessor.mdx

@@ -1,13 +1,13 @@
 ```ts
-function createVideoProcessor(transformers, processTrack): VideoProcessor;
+function createVideoProcessor(segmenter, inputProcessAdaptor): VideoProcessor;
 ```
 
 ## Parameters
 
 | Parameter | Type |
 | ------ | ------ |
-| `transformers` | [`Transform`](../interfaces/Transform.mdx)\<[`InputFrame`](../type-aliases/InputFrame.mdx), [`InputFrame`](../type-aliases/InputFrame.mdx)\>[] |
-| `processTrack` | [`ProcessVideoTrack`](../type-aliases/ProcessVideoTrack.mdx) |
+| `segmenter` | [`Segmenter`](../interfaces/Segmenter.mdx) |
+| `inputProcessAdaptor` | [`InputProcessAdaptor`](../interfaces/InputProcessAdaptor.mdx) |
 
 ## Returns
 

package/api-docs/functions/createWindowedStats.mdx

@@ -0,0 +1,23 @@
+```ts
+function createWindowedStats(windowSize): object;
+```
+
+## Parameters
+
+| Parameter | Type |
+| ------ | ------ |
+| `windowSize` | `number` |
+
+## Returns
+
+`object`
+
+| Name | Type |
+| ------ | ------ |
+| `add()` | (`value`) => `void` |
+| `get mean` | `number` |
+| `get variance` | `number` |
+| `get stddev` | `number` |
+| `get stdError` | `number` |
+| `get count` | `number` |
+| `clear()` | () => `void` |

package/api-docs/functions/getDefaultInputProcessAdaptor.mdx

@@ -0,0 +1,17 @@
+```ts
+function getDefaultInputProcessAdaptor(tryNativeAPI, targetFrameRate): InputProcessAdaptor;
+```
+
+Get the supported InputProcessAdaptor based on feature detection
+and preference provided
+
+## Parameters
+
+| Parameter | Type | Default value | Description |
+| ------ | ------ | ------ | ------ |
+| `tryNativeAPI` | `boolean` | `true` | Whether or not to try to use the Native MediaStreamTrackProcessor APIs |
+| `targetFrameRate` | `number` | `FRAME_RATE` | The target frame rate to use when fallback implementation of video frame extraction rate |
+
+## Returns
+
+[`InputProcessAdaptor`](../interfaces/InputProcessAdaptor.mdx)

package/api-docs/functions/iou.mdx

@@ -0,0 +1,19 @@
+```ts
+function iou(a, b): number;
+```
+
+Intersection over Union (IoU), how much two masks overlap as a fraction of either's total.
+1.0 = perfect match, 0 = no overlap.
+Used for stable, robust tracking of the main person between frames,
+especially with bystander removal and temporal instance switching.
+
+## Parameters
+
+| Parameter | Type |
+| ------ | ------ |
+| `a` | `Uint8Array` |
+| `b` | `Uint8Array` |
+
+## Returns
+
+`number`

package/api-docs/functions/isMediaStreamVideoTrack.mdx

@@ -0,0 +1,13 @@
+```ts
+function isMediaStreamVideoTrack(t): t is MediaStreamVideoTrack;
+```
+
+## Parameters
+
+| Parameter | Type |
+| ------ | ------ |
+| `t` | `MediaStreamTrack` |
+
+## Returns
+
+`t is MediaStreamVideoTrack`

package/api-docs/functions/labelComponents.mdx

@@ -0,0 +1,20 @@
+```ts
+function labelComponents(
+   input, 
+   labels, 
+   width, 
+height): Map<number, Stats>;
+```
+
+## Parameters
+
+| Parameter | Type |
+| ------ | ------ |
+| `input` | `Uint8Array` |
+| `labels` | `Int32Array` |
+| `width` | `number` |
+| `height` | `number` |
+
+## Returns
+
+`Map`\<`number`, [`Stats`](../interfaces/Stats.mdx)\>

package/api-docs/functions/nearestPowerOfTwo.mdx

@@ -0,0 +1,21 @@
+```ts
+function nearestPowerOfTwo(
+   n, 
+   minPow?, 
+   maxPow?): number;
+```
+
+Coerce any positive number to the nearest power of two within [minPow, maxPow]
+Example: nearestPowerOfTwo(13) => 16; nearestPowerOfTwo(3.2) => 4
+
+## Parameters
+
+| Parameter | Type |
+| ------ | ------ |
+| `n` | `number` |
+| `minPow?` | `number` |
+| `maxPow?` | `number` |
+
+## Returns
+
+`number`