@pexip/media-processor 21.0.0 → 22.0.0

package/CHANGELOG.md

@@ -1,5 +1,75 @@
 # Changelog
 
+## 22.0.0
+
+### Major Changes
+
+- 498ff3b: A full architectural overhaul of the video processing pipeline:
+
+  BREAKING CHANGES:
+
+  - **`createCanvasTransform` removed**
+  - **`createVideoTrackProcessor` / `createVideoTrackProcessorWithFallback`
+    removed**
+  - **`createVideoProcessor` redesigned** — now accepts a `Segmenter` +
+    `InputProcessAdaptor` instead of an array of `Transform`s. Exposes
+    `update()`, `mute()`, `status`, `renderOptions`, and processing dimensions.
+  - **New `InputProcessAdaptor` system**:
+    - `createMediaStreamTrackInputProcessAdaptor` — uses native
+      `MediaStreamTrack` transfer (Safari v18+ compatible).
+    - `createReadWritableStreamInputProcessAdaptor` — uses
+      `MediaStreamTrackProcessor` + `MediaStreamTrackGenerator`.
+    - `createReadWritableStreamFallbackInputProcessAdaptor` — canvas/video
+      element fallback.
+    - `getDefaultInputProcessAdaptor` — picks the best path via feature
+      detection (`MediaStreamTrackTransferable`).
+  - **`createSegmenter` updated**
+    - Now processes `MediaStreamTrack` or `ReadWritableStream` directly (no more
+      `VideoFrameLike`).
+    - Added `mute()` method.
+    - Added `stats` callback for frame timing data.
+    - Worker is no longer re-created on every `open()`.
+  - **Worker rewritten**:
+    - Processing now uses `TransformStream` to pipe `VideoFrame`s through
+      segmentation.
+    - Supports both track-based
+      (`VideoTrackGenerator`/`MediaStreamTrackProcessor`) and ReadWritable
+      stream processing.
+    - Added frame timing stats (`FrameTransformStats`) batched and posted back
+      to main thread.
+    - Removes direct canvas-copy output (`directOutput`/`output`); renderer now
+      writes to `processCanvas` and creates a new `VideoFrame`.
+    - `close` and `suspend` separated; resource cleanup improved.
+    - Background image fetching is now abortable.
+  - **`Renderer.render()` signature simplified** — accepts `OffscreenCanvas` +
+    `VideoFrame` directly (no `VideoFrameLike` wrapper).
+  - **WebGL/WebGPU texture renderers** updated to accept
+    `VideoFrame | ImageBitmap | OffscreenCanvas` natively.
+  - **WebGL program release guard** added to prevent double-free.
+  - **`benchUtils`** extended to accept explicit timestamps in
+    `begin()`/`end()` + tests added.
+  - **`MediaStreamTrackTransferable`** feature-detection helper exported.
+  - **Default frame rate raised** from 20 → 30 fps.
+  - **Default backend changed** to `webgpu`.
+
+- fe0a0a4: Set a max width of 300px of in meeting selfview.
+
+  Rename openEffectsModal prop on InMeetingSelfview to openEffects
+
+  Make selfview muted indicator look more like an indicator
+
+  Show tooltips for background effects button and upload background effect
+
+### Patch Changes
+
+- f8bd1c5: Fix VideoProcessor and Segment open/update behavior
+- 925cce6: Fix f16 issue
+- fb41d4d: Changes related to Typescript V6 Upgrade
+- Updated dependencies [498ff3b]
+- Updated dependencies [fb41d4d]
+- Updated dependencies [b99c070]
+  - @pexip/utils@17.3.0
+
 ## 21.0.0
 
 ### Major Changes

package/README.md

@@ -106,10 +106,12 @@ await audioGraph.release();
 
 ```typescript
 import {
+  createMediaStreamTrackInputProcessAdaptor,
+  createReadWritableStreamFallbackInputProcessAdaptor,
+  createReadWritableStreamInputProcessAdaptor,
   createSegmenter,
-  createCanvasTransform,
-  createVideoTrackProcessor,
-  createVideoTrackProcessorWithFallback,
+  createVideoProcessor,
+  getDefaultInputProcessAdaptor,
 } from '@pexip/media-processor';
 
 // Grab the user's camera stream
@@ -129,17 +131,22 @@ const modelAsset = {
   modelName: 'selfie' as const,
 };
 
-const segmenter = createSegmenter(tasksVisionBasePath, {modelAssets});
-// Create a processing transformer and set the effects to `blur`
-const transformer = createCanvasTransform(segmenter, {effects: 'blur'});
-const processor = createVideoProcessor([transformer]);
+const segmenter = createSegmenter(tasksVisionBasePath, {modelAsset});
+// Choose the appropriate input process adaptor based on browser capabilities
+const inputProcessAdaptor = getDefaultInputProcessAdaptor(true);
+// Create a processor and set the effects to `blur`
+const processor = createVideoProcessor(segmenter, inputProcessAdaptor);
 
 // Start the processor
 await videoProcessor.open();
 
-// Passing the raw MediaStream to apply the effects
-// Then, use the output stream for whatever purpose
-const processedStream = await videoProcessor.process(stream);
+// Start the processor with blur effects
+await processor.open({renderOptions: {effects: 'blur'}});
+
+// Process the video track and create a new stream
+const videoTrack = stream.getVideoTracks()[0];
+const processedTrack = await processor.process(videoTrack);
+const processedStream = new MediaStream([processedTrack]);
 ```
 
 ## Profiling Web Audio

package/api-docs/README.mdx

@@ -106,10 +106,12 @@ await audioGraph.release();
 
 ```typescript
 import {
+  createMediaStreamTrackInputProcessAdaptor,
+  createReadWritableStreamFallbackInputProcessAdaptor,
+  createReadWritableStreamInputProcessAdaptor,
   createSegmenter,
-  createCanvasTransform,
-  createVideoTrackProcessor,
-  createVideoTrackProcessorWithFallback,
+  createVideoProcessor,
+  getDefaultInputProcessAdaptor,
 } from '@pexip/media-processor';
 
 // Grab the user's camera stream
@@ -129,17 +131,22 @@ const modelAsset = {
   modelName: 'selfie' as const,
 };
 
-const segmenter = createSegmenter(tasksVisionBasePath, {modelAssets});
-// Create a processing transformer and set the effects to `blur`
-const transformer = createCanvasTransform(segmenter, {effects: 'blur'});
-const processor = createVideoProcessor([transformer]);
+const segmenter = createSegmenter(tasksVisionBasePath, {modelAsset});
+// Choose the appropriate input process adaptor based on browser capabilities
+const inputProcessAdaptor = getDefaultInputProcessAdaptor(true);
+// Create a processor and set the effects to `blur`
+const processor = createVideoProcessor(segmenter, inputProcessAdaptor);
 
 // Start the processor
 await videoProcessor.open();
 
-// Passing the raw MediaStream to apply the effects
-// Then, use the output stream for whatever purpose
-const processedStream = await videoProcessor.process(stream);
+// Start the processor with blur effects
+await processor.open({renderOptions: {effects: 'blur'}});
+
+// Process the video track and create a new stream
+const videoTrack = stream.getVideoTracks()[0];
+const processedTrack = await processor.process(videoTrack);
+const processedStream = new MediaStream([processedTrack]);
 ```
 
 ## Profiling Web Audio
@@ -193,11 +200,13 @@ A library for media analysis using Web APIs.
 | Interface | Description |
 | ------ | ------ |
 | [Segmentation](interfaces/Segmentation.mdx) | - |
+| [ReadWritableStream](interfaces/ReadWritableStream.mdx) | - |
 | [VideoFrameLike](interfaces/VideoFrameLike.mdx) | - |
 | [ImageRecord](interfaces/ImageRecord.mdx) | - |
 | [Rect](interfaces/Rect.mdx) | - |
 | [ProcessEventOpen](interfaces/ProcessEventOpen.mdx) | Message event to initializing the Processor |
 | [ProcessEventProcess](interfaces/ProcessEventProcess.mdx) | Message event to process the video frame |
+| [ProcessEventMute](interfaces/ProcessEventMute.mdx) | - |
 | [ProcessEventUpdate](interfaces/ProcessEventUpdate.mdx) | Message event to update processor options |
 | [ProcessEventClose](interfaces/ProcessEventClose.mdx) | Message event to stop the processing |
 | [ProcessEventDestroy](interfaces/ProcessEventDestroy.mdx) | Message event to release all the resources |
@@ -207,6 +216,8 @@ A library for media analysis using Web APIs.
 | [ProcessWorkerEventUpdated](interfaces/ProcessWorkerEventUpdated.mdx) | - |
 | [ProcessWorkerEvent](interfaces/ProcessWorkerEvent.mdx) | - |
 | [ProcessWorkerEventDebug](interfaces/ProcessWorkerEventDebug.mdx) | - |
+| [FrameTransformStats](interfaces/FrameTransformStats.mdx) | Frame transform statistic, i.e. the timing data for processing each frame |
+| [ProcessWorkerEventStats](interfaces/ProcessWorkerEventStats.mdx) | - |
 | [ProcessorOptions](interfaces/ProcessorOptions.mdx) | - |
 | [ProcessorProcessOptions](interfaces/ProcessorProcessOptions.mdx) | - |
 | [ProcessWorkerDebugMessage](interfaces/ProcessWorkerDebugMessage.mdx) | - |
@@ -242,13 +253,12 @@ A library for media analysis using Web APIs.
 | [Clock](interfaces/Clock.mdx) | Clock interface to get the current time with now method, |
 | [ThrottleOptions](interfaces/ThrottleOptions.mdx) | Limit the rate of flow in terms of millisecond, and provided Clock |
 | [Runner](interfaces/Runner.mdx) | - |
-| [Transform](interfaces/Transform.mdx) | - |
 | [AsyncAssets](interfaces/AsyncAssets.mdx) | - |
 | [Process](interfaces/Process.mdx) | - |
 | [BackgroundImage](interfaces/BackgroundImage.mdx) | - |
 | [Segmenter](interfaces/Segmenter.mdx) | - |
-| [Detector](interfaces/Detector.mdx) | - |
-| [SegmentationParams](interfaces/SegmentationParams.mdx) | Options controlling the rendering and effects for person/background segmentation. All values are stable per-frame and can be safely updated in real-time. |
+| [InputProcessAdaptor](interfaces/InputProcessAdaptor.mdx) | - |
+| [ProcessorUpdateOptions](interfaces/ProcessorUpdateOptions.mdx) | - |
 | [VideoProcessor](interfaces/VideoProcessor.mdx) | - |
 
 ## Type Aliases
@@ -266,6 +276,7 @@ A library for media analysis using Web APIs.
 | [ProcessWorkerEvents](type-aliases/ProcessWorkerEvents.mdx) | Message events returned from the processor |
 | [ProcessorEvents](type-aliases/ProcessorEvents.mdx) | - |
 | [ProcessorWorkerEvents](type-aliases/ProcessorWorkerEvents.mdx) | - |
+| [Struct](type-aliases/Struct.mdx) | - |
 | [SegmentationModel](type-aliases/SegmentationModel.mdx) | - |
 | [Delegate](type-aliases/Delegate.mdx) | - |
 | [ExtractMessageEventType](type-aliases/ExtractMessageEventType.mdx) | - |
@@ -309,8 +320,6 @@ A library for media analysis using Web APIs.
 | [WasmPaths](type-aliases/WasmPaths.mdx) | - |
 | [Color](type-aliases/Color.mdx) | - |
 | [InputFrame](type-aliases/InputFrame.mdx) | - |
-| [SegmentationTransform](type-aliases/SegmentationTransform.mdx) | - |
-| [ProcessVideoTrack](type-aliases/ProcessVideoTrack.mdx) | - |
 
 ## Variables
 
@@ -342,18 +351,24 @@ A library for media analysis using Web APIs.
 | [RENDER\_EFFECT](variables/RENDER_EFFECT.mdx) | - |
 | [RENDER\_BACKEND](variables/RENDER_BACKEND.mdx) | - |
 | [RENDERING\_EVENTS](variables/RENDERING_EVENTS.mdx) | - |
+| [isValidGpuAPI](variables/isValidGpuAPI.mdx) | - |
+| [isRenderOptionsKey](variables/isRenderOptionsKey.mdx) | - |
+| [RENDERER\_OPTIONS\_STRUCT](variables/RENDERER_OPTIONS_STRUCT.mdx) | - |
 | [SEGMENTATION\_MODELS](variables/SEGMENTATION_MODELS.mdx) | - |
 | [isSegmentationModel](variables/isSegmentationModel.mdx) | - |
 | [isRenderingEvents](variables/isRenderingEvents.mdx) | - |
 | [toRenderingEvents](variables/toRenderingEvents.mdx) | - |
 | [getErrorMessage](variables/getErrorMessage.mdx) | - |
 | [createObjectUpdater](variables/createObjectUpdater.mdx) | - |
+| [ImageFetchError](variables/ImageFetchError.mdx) | - |
 | [createRemoteImageBitmap](variables/createRemoteImageBitmap.mdx) | - |
 | [clamping](variables/clamping.mdx) | - |
 | [calculateMaxBlurPass](variables/calculateMaxBlurPass.mdx) | - |
 | [resize](variables/resize.mdx) | Calculate the coordinates and size of the source and destination to fit the target aspect ratio. |
 | [createLazyProps](variables/createLazyProps.mdx) | - |
 | [handleWebGLContextLoss](variables/handleWebGLContextLoss.mdx) | - |
+| [cloneImageRecord](variables/cloneImageRecord.mdx) | - |
+| [compareImageRecords](variables/compareImageRecords.mdx) | Compare provided ImageRecord |
 | [urls](variables/urls.mdx) | - |
 | [SILENT\_THRESHOLD](variables/SILENT_THRESHOLD.mdx) | Default silent threshold At least one LSB 16-bit data (compare is on absolute value). |
 | [MONO\_THRESHOLD](variables/MONO_THRESHOLD.mdx) | Default mono detection threshold Data must be identical within one LSB 16-bit to be identified as mono. |
@@ -361,6 +376,7 @@ A library for media analysis using Web APIs.
 | [CLIP\_THRESHOLD](variables/CLIP_THRESHOLD.mdx) | Default clipping detection threshold |
 | [VOICE\_PROBABILITY\_THRESHOLD](variables/VOICE_PROBABILITY_THRESHOLD.mdx) | Default Voice probability threshold |
 | [CLIP\_COUNT\_THRESHOLD](variables/CLIP_COUNT_THRESHOLD.mdx) | Default clipping count threshold Number of consecutive clipThreshold level samples that indicate clipping. |
+| [MediaStreamTrackTransferable](variables/MediaStreamTrackTransferable.mdx) | Result of the feature detection if `MediaStreamTrack` is `Transferable` |
 | [FRAME\_RATE](variables/FRAME_RATE.mdx) | - |
 | [PLAY\_VIDEO\_TIMEOUT](variables/PLAY_VIDEO_TIMEOUT.mdx) | - |
 
@@ -419,18 +435,21 @@ A library for media analysis using Web APIs.
 | [isAudioNodeInit](functions/isAudioNodeInit.mdx) | - |
 | [isAnalyzerNodeInit](functions/isAnalyzerNodeInit.mdx) | - |
 | [createAsyncCallbackLoop](functions/createAsyncCallbackLoop.mdx) | Create an async callback loop to be called recursively with delay based on the `frameRate` |
-| [createCanvasTransform](functions/createCanvasTransform.mdx) | - |
+| [createReadWritableStreamInputProcessAdaptor](functions/createReadWritableStreamInputProcessAdaptor.mdx) | An implementation of `InputProcessAdaptor` which uses `MediaStreamTrackGenerator` and `MediaStreamTrackProcessor` APIs to create `ReadableStream` & `WritableStream` |
+| [createReadWritableStreamFallbackInputProcessAdaptor](functions/createReadWritableStreamFallbackInputProcessAdaptor.mdx) | An implementation of `InputProcessAdaptor` which uses `HTMLCanvasElement` or `OffscreenCanvas` API to create `ReadableStream` & `WritableStream` |
+| [createMediaStreamTrackInputProcessAdaptor](functions/createMediaStreamTrackInputProcessAdaptor.mdx) | An implementation of `InputProcessAdaptor` which adapts `MediaStreamTrack` directly as specified from the spec https://w3c.github.io/mediacapture-transform |
+| [getDefaultInputProcessAdaptor](functions/getDefaultInputProcessAdaptor.mdx) | Get the supported InputProcessAdaptor based on feature detection and preference provided |
 | [loadScript](functions/loadScript.mdx) | - |
 | [loadWasms](functions/loadWasms.mdx) | - |
 | [createSegmenter](functions/createSegmenter.mdx) | - |
 | [isRenderEffects](functions/isRenderEffects.mdx) | - |
+| [isMediaStreamVideoTrack](functions/isMediaStreamVideoTrack.mdx) | - |
+| [assertMediaStreamVideoTrack](functions/assertMediaStreamVideoTrack.mdx) | - |
 | [createCanvas](functions/createCanvas.mdx) | Create a Canvas element with provided width and height |
 | [setVideoElementSrc](functions/setVideoElementSrc.mdx) | - |
 | [playVideo](functions/playVideo.mdx) | - |
 | [createFrameCallbackRequest](functions/createFrameCallbackRequest.mdx) | Create a callback loop for video frame processing using `requestVideoFrameCallback` under-the-hood when available otherwise our fallback implementation based on `setTimeout`. |
 | [createVideoProcessor](functions/createVideoProcessor.mdx) | - |
-| [createVideoTrackProcessor](functions/createVideoTrackProcessor.mdx) | Video track processor using MediaStreamTrackProcessor API |
-| [createVideoTrackProcessorWithFallback](functions/createVideoTrackProcessorWithFallback.mdx) | Video track processor using Canvas[captureStream] API |
 | [calculateDistance](functions/calculateDistance.mdx) | Calculate the distance between two Points |
 | [getBezierCurveControlPoints](functions/getBezierCurveControlPoints.mdx) | Spline Interpolation for Bezier Curve |
 | [line](functions/line.mdx) | Create a straight line path command |

package/api-docs/functions/assertMediaStreamVideoTrack.mdx

@@ -0,0 +1,13 @@
+```ts
+function assertMediaStreamVideoTrack(t): asserts t is MediaStreamVideoTrack;
+```
+
+## Parameters
+
+| Parameter | Type |
+| ------ | ------ |
+| `t` | `MediaStreamTrack` |
+
+## Returns
+
+`asserts t is MediaStreamVideoTrack`

package/api-docs/functions/createFrameCallbackRequest.mdx

@@ -1,5 +1,5 @@
 ```ts
-function createFrameCallbackRequest(
+function createFrameCallbackRequest<T>(
    callback, 
    frameRate, 
    __namedParameters): object;
@@ -9,11 +9,17 @@ Create a callback loop for video frame processing using
 `requestVideoFrameCallback` under-the-hood when available otherwise our
 fallback implementation based on `setTimeout`.
 
+## Type Parameters
+
+| Type Parameter |
+| ------ |
+| `T` |
+
 ## Parameters
 
 | Parameter | Type | Description |
 | ------ | ------ | ------ |
-| `callback` | [`Callback`](../type-aliases/Callback.mdx)\<`Promise`\<`void`\>, \[[`ProcessInputType`](../type-aliases/ProcessInputType.mdx)\]\> | To be called by the loop |
+| `callback` | [`Callback`](../type-aliases/Callback.mdx)\<`Promise`\<`void`\>, \[`T`\]\> | To be called by the loop |
 | `frameRate` | `number` | A fallback frame rate when we are not able to get the rate from API |
 | `__namedParameters` | `FrameCallbackRequestOptions` | - |
 

package/api-docs/functions/createMediaStreamTrackInputProcessAdaptor.mdx

@@ -0,0 +1,10 @@
+```ts
+function createMediaStreamTrackInputProcessAdaptor(): InputProcessAdaptor;
+```
+
+An implementation of `InputProcessAdaptor` which adapts `MediaStreamTrack`
+directly as specified from the spec https://w3c.github.io/mediacapture-transform
+
+## Returns
+
+[`InputProcessAdaptor`](../interfaces/InputProcessAdaptor.mdx)

package/api-docs/functions/createReadWritableStreamFallbackInputProcessAdaptor.mdx

@@ -0,0 +1,17 @@
+```ts
+function createReadWritableStreamFallbackInputProcessAdaptor(__namedParameters): InputProcessAdaptor;
+```
+
+An implementation of `InputProcessAdaptor` which uses `HTMLCanvasElement` or
+`OffscreenCanvas` API to create `ReadableStream` & `WritableStream`
+
+## Parameters
+
+| Parameter | Type | Description |
+| ------ | ------ | ------ |
+| `__namedParameters` | \{ `frameRate`: `number`; \} | - |
+| `__namedParameters.frameRate` | `number` | Fallback frame rate |
+
+## Returns
+
+[`InputProcessAdaptor`](../interfaces/InputProcessAdaptor.mdx)

package/api-docs/functions/createReadWritableStreamInputProcessAdaptor.mdx

@@ -0,0 +1,10 @@
+```ts
+function createReadWritableStreamInputProcessAdaptor(): InputProcessAdaptor;
+```
+
+An implementation of `InputProcessAdaptor` which uses `MediaStreamTrackGenerator` and
+`MediaStreamTrackProcessor` APIs to create `ReadableStream` & `WritableStream`
+
+## Returns
+
+[`InputProcessAdaptor`](../interfaces/InputProcessAdaptor.mdx)

package/api-docs/functions/createVideoProcessor.mdx

@@ -1,13 +1,13 @@
 ```ts
-function createVideoProcessor(transformers, processTrack): VideoProcessor;
+function createVideoProcessor(segmenter, inputProcessAdaptor): VideoProcessor;
 ```
 
 ## Parameters
 
 | Parameter | Type |
 | ------ | ------ |
-| `transformers` | [`Transform`](../interfaces/Transform.mdx)\<[`InputFrame`](../type-aliases/InputFrame.mdx), [`InputFrame`](../type-aliases/InputFrame.mdx)\>[] |
-| `processTrack` | [`ProcessVideoTrack`](../type-aliases/ProcessVideoTrack.mdx) |
+| `segmenter` | [`Segmenter`](../interfaces/Segmenter.mdx) |
+| `inputProcessAdaptor` | [`InputProcessAdaptor`](../interfaces/InputProcessAdaptor.mdx) |
 
 ## Returns
 

package/api-docs/functions/getDefaultInputProcessAdaptor.mdx

@@ -0,0 +1,17 @@
+```ts
+function getDefaultInputProcessAdaptor(tryNativeAPI, targetFrameRate): InputProcessAdaptor;
+```
+
+Get the supported InputProcessAdaptor based on feature detection
+and preference provided
+
+## Parameters
+
+| Parameter | Type | Default value | Description |
+| ------ | ------ | ------ | ------ |
+| `tryNativeAPI` | `boolean` | `true` | Whether or not to try to use the Native MediaStreamTrackProcessor APIs |
+| `targetFrameRate` | `number` | `FRAME_RATE` | The target frame rate to use when fallback implementation of video frame extraction rate |
+
+## Returns
+
+[`InputProcessAdaptor`](../interfaces/InputProcessAdaptor.mdx)

package/api-docs/functions/isMediaStreamVideoTrack.mdx

@@ -0,0 +1,13 @@
+```ts
+function isMediaStreamVideoTrack(t): t is MediaStreamVideoTrack;
+```
+
+## Parameters
+
+| Parameter | Type |
+| ------ | ------ |
+| `t` | `MediaStreamTrack` |
+
+## Returns
+
+`t is MediaStreamVideoTrack`

package/api-docs/functions/playVideo.mdx

@@ -1,5 +1,5 @@
 ```ts
-function playVideo(video, __namedParameters): Promise<void>;
+function playVideo(video, __namedParameters): Promise<[void, void]> | undefined;
 ```
 
 ## Parameters
@@ -13,4 +13,4 @@ function playVideo(video, __namedParameters): Promise<void>;
 
 ## Returns
 
-`Promise`\<`void`\>
+`Promise`\<\[`void`, `void`\]\> \| `undefined`

package/api-docs/interfaces/Analyzer.mdx

@@ -163,6 +163,8 @@ connect(
    input?): AudioNode;
 ```
 
+The **`connect()`** method of the AudioNode interface lets you connect one of the node's outputs to a target, which may be either another AudioNode (thereby directing the sound data to the specified node) or an AudioParam, so that the node's output data is automatically used to change the value of that parameter over time.
+
 [MDN Reference](https://developer.mozilla.org/docs/Web/API/AudioNode/connect)
 
 ##### Parameters
@@ -216,6 +218,8 @@ BaseAudioNode.connect
 disconnect(): void;
 ```
 
+The **`disconnect()`** method of the AudioNode interface lets you disconnect one or more nodes from the node on which the method is called.
+
 [MDN Reference](https://developer.mozilla.org/docs/Web/API/AudioNode/disconnect)
 
 ##### Returns

package/api-docs/interfaces/Benchmark.mdx

@@ -3,9 +3,15 @@
 ### begin()
 
 ```ts
-begin(): void;
+begin(timestamp?): void;
 ```
 
+#### Parameters
+
+| Parameter | Type |
+| ------ | ------ |
+| `timestamp?` | `number` |
+
 #### Returns
 
 `void`
@@ -15,9 +21,15 @@ begin(): void;
 ### end()
 
 ```ts
-end(): void;
+end(timestamp?): void;
 ```
 
+#### Parameters
+
+| Parameter | Type |
+| ------ | ------ |
+| `timestamp?` | `number` |
+
 #### Returns
 
 `void`

package/api-docs/interfaces/FrameTransformStats.mdx

@@ -0,0 +1,9 @@
+Frame transform statistic, i.e. the timing data for processing each frame
+
+## Properties
+
+| Property | Type | Description |
+| ------ | ------ | ------ |
+| <a id="start"></a> `start` | `number` | Start time of the transform |
+| <a id="end"></a> `end` | `number` | End time of the transform |
+| <a id="segmentation"></a> `segmentation?` | `number` | Segmentation inference end time |

package/api-docs/interfaces/Gain.mdx

@@ -22,6 +22,8 @@ connect(
    input?): AudioNode;
 ```
 
+The **`connect()`** method of the AudioNode interface lets you connect one of the node's outputs to a target, which may be either another AudioNode (thereby directing the sound data to the specified node) or an AudioParam, so that the node's output data is automatically used to change the value of that parameter over time.
+
 [MDN Reference](https://developer.mozilla.org/docs/Web/API/AudioNode/connect)
 
 ##### Parameters
@@ -75,6 +77,8 @@ BaseAudioNode.connect
 disconnect(): void;
 ```
 
+The **`disconnect()`** method of the AudioNode interface lets you disconnect one or more nodes from the node on which the method is called.
+
 [MDN Reference](https://developer.mozilla.org/docs/Web/API/AudioNode/disconnect)
 
 ##### Returns

package/api-docs/interfaces/ImageSegmenterOptions.mdx

@@ -2,7 +2,5 @@
 
 | Property | Type | Description |
 | ------ | ------ | ------ |
-| <a id="canvas"></a> `canvas?` | `OffscreenCanvas` | The canvas element to bind textures to. This has to be set for GPU processing. The task will initialize a WebGL or WebGPU context and throw an error if this fails (e.g. if you have already initialized a different type of context). |
-| <a id="segmentercanvas"></a> `segmenterCanvas?` | `OffscreenCanvas` | The Google TaskVision segmentation model only works with WebGl so we need a separate canvas for this. |
 | <a id="modelasset"></a> `modelAsset?` | [`SegmentationModelAsset`](SegmentationModelAsset.mdx) | - |
 | <a id="delegate"></a> `delegate?` | [`Delegate`](../type-aliases/Delegate.mdx) | Overrides the default backend to use for the provided model. |

package/api-docs/interfaces/InputProcessAdaptor.mdx

@@ -0,0 +1,28 @@
+## Properties
+
+| Property | Type | Description |
+| ------ | ------ | ------ |
+| <a id="outputtrack"></a> `outputTrack?` | `MediaStreamTrack` | The generated track from the `adapt` function, and only available after calling the `adapt` method otherwise `undefined` |
+
+## Methods
+
+### adapt()
+
+```ts
+adapt(track): 
+  | MediaStreamTrack
+| ReadWritableStream<VideoFrame>;
+```
+
+Adapt the provided input track based on browser capabilities.
+
+#### Parameters
+
+| Parameter | Type |
+| ------ | ------ |
+| `track` | `MediaStreamTrack` |
+
+#### Returns
+
+  \| `MediaStreamTrack`
+  \| [`ReadWritableStream`](ReadWritableStream.mdx)\<`VideoFrame`\>

package/api-docs/interfaces/Process.mdx

@@ -1,7 +1,12 @@
 ## Extended by
 
 - [`Segmenter`](Segmenter.mdx)
-- [`Detector`](Detector.mdx)
+
+## Type Parameters
+
+| Type Parameter |
+| ------ |
+| `T` |
 
 ## Properties
 
@@ -14,12 +19,12 @@
 ### open()
 
 ```ts
-open(): Promise<void>;
+open(): Promise<T>;
 ```
 
 #### Returns
 
-`Promise`\<`void`\>
+`Promise`\<`T`\>
 
 ***
 

package/api-docs/interfaces/ProcessEventMute.mdx

@@ -0,0 +1,6 @@
+## Properties
+
+| Property | Type |
+| ------ | ------ |
+| <a id="type"></a> `type` | `"mute"` |
+| <a id="data"></a> `data` | `boolean` |

package/api-docs/interfaces/ProcessWorkerEventOpened.mdx

@@ -1,5 +1,12 @@
+## Type Parameters
+
+| Type Parameter |
+| ------ |
+| `T` |
+
 ## Properties
 
 | Property | Type |
 | ------ | ------ |
 | <a id="type"></a> `type` | `"opened"` |
+| <a id="data"></a> `data` | `T` |

package/api-docs/interfaces/ProcessWorkerEventStats.mdx

@@ -0,0 +1,6 @@
+## Properties
+
+| Property | Type |
+| ------ | ------ |
+| <a id="type"></a> `type` | `"stats"` |
+| <a id="data"></a> `data` | [`FrameTransformStats`](FrameTransformStats.mdx)[] |

package/api-docs/interfaces/ProcessWorkerEventUpdated.mdx

@@ -1,5 +1,12 @@
+## Type Parameters
+
+| Type Parameter |
+| ------ |
+| `T` |
+
 ## Properties
 
 | Property | Type |
 | ------ | ------ |
 | <a id="type"></a> `type` | `"updated"` |
+| <a id="data"></a> `data` | `T` |

package/api-docs/interfaces/ProcessorEvent.mdx

@@ -4,4 +4,4 @@
 | ------ | ------ |
 | <a id="type"></a> `type` | \| `"Error"` \| `"ContextCreationError"` \| `"ContextLost"` \| `"ContextRestored"` \| `"ProcessorRestarted"` \| `"SamplingError"` |
 | <a id="message"></a> `message?` | `string` |
-| <a id="frame"></a> `frame?` | [`VideoFrameLike`](VideoFrameLike.mdx) |
+| <a id="track"></a> `track?` | `MediaStreamTrack` |

package/api-docs/interfaces/ProcessorOptions.mdx

@@ -10,7 +10,6 @@
 | <a id="backgroundimage"></a> `backgroundImage?` | [`ImageRecord`](ImageRecord.mdx) | - |
 | <a id="restart"></a> `restart?` | `boolean` | - |
 | <a id="basepath"></a> `basePath?` | `string` | [`SegmenterOptions`](SegmenterOptions.mdx).[`basePath`](SegmenterOptions.mdx#basepath) |
-| <a id="output"></a> `output?` | `OffscreenCanvas` \| `null` | [`SegmenterOptions`](SegmenterOptions.mdx).[`output`](SegmenterOptions.mdx#output) |
 | <a id="processingwidth"></a> `processingWidth?` | `number` | [`SegmenterOptions`](SegmenterOptions.mdx).[`processingWidth`](SegmenterOptions.mdx#processingwidth) |
 | <a id="processingheight"></a> `processingHeight?` | `number` | [`SegmenterOptions`](SegmenterOptions.mdx).[`processingHeight`](SegmenterOptions.mdx#processingheight) |
 | <a id="imagesegmenteroptions"></a> `imageSegmenterOptions?` | [`ImageSegmenterOptions`](ImageSegmenterOptions.mdx) | [`SegmenterOptions`](SegmenterOptions.mdx).[`imageSegmenterOptions`](SegmenterOptions.mdx#imagesegmenteroptions) |

package/api-docs/interfaces/ProcessorProcessOptions.mdx

@@ -2,5 +2,7 @@
 
 | Property | Type |
 | ------ | ------ |
-| <a id="videoframe"></a> `videoFrame` | [`VideoFrameLike`](VideoFrameLike.mdx) |
+| <a id="track"></a> `track?` | `MediaStreamTrack` |
+| <a id="readable"></a> `readable?` | `ReadableStream`\<`VideoFrame`\> |
+| <a id="writable"></a> `writable?` | `WritableStream`\<`VideoFrame`\> |
 | <a id="renderoptions"></a> `renderOptions?` | [`RendererOptions`](RendererOptions.mdx) |