@pexip/media-processor 16.7.0

package/LICENSE

@@ -0,0 +1,188 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+1. Definitions.
+
+    "License" shall mean the terms and conditions for use, reproduction, and
+    distribution as defined by Sections 1 through 9 of this document.
+
+    "Licensor" shall mean the copyright owner or entity authorized by the
+    copyright owner that is granting the License.
+
+    "Legal Entity" shall mean the union of the acting entity and all other
+    entities that control, are controlled by, or are under common control with
+    that entity. For the purposes of this definition, "control" means (i) the
+    power, direct or indirect, to cause the direction or management of such
+    entity, whether by contract or otherwise, or (ii) ownership of fifty percent
+    (50%) or more of the outstanding shares, or (iii) beneficial ownership of
+    such entity.
+
+    "You" (or "Your") shall mean an individual or Legal Entity exercising
+    permissions granted by this License.
+
+    "Source" form shall mean the preferred form for making modifications,
+    including but not limited to software source code, documentation source, and
+    configuration files.
+
+    "Object" form shall mean any form resulting from mechanical transformation
+    or translation of a Source form, including but not limited to compiled
+    object code, generated documentation, and conversions to other media types.
+
+    "Work" shall mean the work of authorship, whether in Source or Object form,
+    made available under the License, as indicated by a copyright notice that is
+    included in or attached to the work (an example is provided in the Appendix
+    below).
+
+    "Derivative Works" shall mean any work, whether in Source or Object form,
+    that is based on (or derived from) the Work and for which the editorial
+    revisions, annotations, elaborations, or other modifications represent, as a
+    whole, an original work of authorship. For the purposes of this License,
+    Derivative Works shall not include works that remain separable from, or
+    merely link (or bind by name) to the interfaces of, the Work and Derivative
+    Works thereof.
+
+    "Contribution" shall mean any work of authorship, including the original
+    version of the Work and any modifications or additions to that Work or
+    Derivative Works thereof, that is intentionally submitted to Licensor for
+    inclusion in the Work by the copyright owner or by an individual or Legal
+    Entity authorized to submit on behalf of the copyright owner. For the
+    purposes of this definition, "submitted" means any form of electronic,
+    verbal, or written communication sent to the Licensor or its
+    representatives, including but not limited to communication on electronic
+    mailing lists, source code control systems, and issue tracking systems that
+    are managed by, or on behalf of, the Licensor for the purpose of discussing
+    and improving the Work, but excluding communication that is conspicuously
+    marked or otherwise designated in writing by the copyright owner as "Not a
+    Contribution."
+
+    "Contributor" shall mean Licensor and any individual or Legal Entity on
+    behalf of whom a Contribution has been received by Licensor and subsequently
+    incorporated within the Work.
+
+2. Grant of Copyright License. Subject to the terms and conditions of this
+   License, each Contributor hereby grants to You a perpetual, worldwide,
+   non-exclusive, no-charge, royalty-free, irrevocable copyright license to
+   reproduce, prepare Derivative Works of, publicly display, publicly perform,
+   sublicense, and distribute the Work and such Derivative Works in Source or
+   Object form.
+
+3. Grant of Patent License. Subject to the terms and conditions of this License,
+   each Contributor hereby grants to You a perpetual, worldwide, non-exclusive,
+   no-charge, royalty-free, irrevocable (except as stated in this section)
+   patent license to make, have made, use, offer to sell, sell, import, and
+   otherwise transfer the Work, where such license applies only to those patent
+   claims licensable by such Contributor that are necessarily infringed by their
+   Contribution(s) alone or by combination of their Contribution(s) with the
+   Work to which such Contribution(s) was submitted. If You institute patent
+   litigation against any entity (including a cross-claim or counterclaim in a
+   lawsuit) alleging that the Work or a Contribution incorporated within the
+   Work constitutes direct or contributory patent infringement, then any patent
+   licenses granted to You under this License for that Work shall terminate as
+   of the date such litigation is filed.
+
+4. Redistribution. You may reproduce and distribute copies of the Work or
+   Derivative Works thereof in any medium, with or without modifications, and in
+   Source or Object form, provided that You meet the following conditions:
+
+    (a) You must give any other recipients of the Work or Derivative Works a
+    copy of this License; and
+
+    (b) You must cause any modified files to carry prominent notices stating
+    that You changed the files; and
+
+    (c) You must retain, in the Source form of any Derivative Works that You
+    distribute, all copyright, patent, trademark, and attribution notices from
+    the Source form of the Work, excluding those notices that do not pertain to
+    any part of the Derivative Works; and
+
+    (d) If the Work includes a "NOTICE" text file as part of its distribution,
+    then any Derivative Works that You distribute must include a readable copy
+    of the attribution notices contained within such NOTICE file, excluding
+    those notices that do not pertain to any part of the Derivative Works, in at
+    least one of the following places: within a NOTICE text file distributed as
+    part of the Derivative Works; within the Source form or documentation, if
+    provided along with the Derivative Works; or, within a display generated by
+    the Derivative Works, if and wherever such third-party notices normally
+    appear. The contents of the NOTICE file are for informational purposes only
+    and do not modify the License. You may add Your own attribution notices
+    within Derivative Works that You distribute, alongside or as an addendum to
+    the NOTICE text from the Work, provided that such additional attribution
+    notices cannot be construed as modifying the License.
+
+    You may add Your own copyright statement to Your modifications and may
+    provide additional or different license terms and conditions for use,
+    reproduction, or distribution of Your modifications, or for any such
+    Derivative Works as a whole, provided Your use, reproduction, and
+    distribution of the Work otherwise complies with the conditions stated in
+    this License.
+
+5. Submission of Contributions. Unless You explicitly state otherwise, any
+   Contribution intentionally submitted for inclusion in the Work by You to the
+   Licensor shall be under the terms and conditions of this License, without any
+   additional terms or conditions. Notwithstanding the above, nothing herein
+   shall supersede or modify the terms of any separate license agreement you may
+   have executed with Licensor regarding such Contributions.
+
+6. Trademarks. This License does not grant permission to use the trade names,
+   trademarks, service marks, or product names of the Licensor, except as
+   required for reasonable and customary use in describing the origin of the
+   Work and reproducing the content of the NOTICE file.
+
+7. Disclaimer of Warranty. Unless required by applicable law or agreed to in
+   writing, Licensor provides the Work (and each Contributor provides its
+   Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+   KIND, either express or implied, including, without limitation, any
+   warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or
+   FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining
+   the appropriateness of using or redistributing the Work and assume any risks
+   associated with Your exercise of permissions under this License.
+
+8. Limitation of Liability. In no event and under no legal theory, whether in
+   tort (including negligence), contract, or otherwise, unless required by
+   applicable law (such as deliberate and grossly negligent acts) or agreed to
+   in writing, shall any Contributor be liable to You for damages, including any
+   direct, indirect, special, incidental, or consequential damages of any
+   character arising as a result of this License or out of the use or inability
+   to use the Work (including but not limited to damages for loss of goodwill,
+   work stoppage, computer failure or malfunction, or any and all other
+   commercial damages or losses), even if such Contributor has been advised of
+   the possibility of such damages.
+
+9. Accepting Warranty or Additional Liability. While redistributing the Work or
+   Derivative Works thereof, You may choose to offer, and charge a fee for,
+   acceptance of support, warranty, indemnity, or other liability obligations
+   and/or rights consistent with this License. However, in accepting such
+   obligations, You may act only on Your own behalf and on Your sole
+   responsibility, not on behalf of any other Contributor, and only if You agree
+   to indemnify, defend, and hold each Contributor harmless for any liability
+   incurred by, or claims asserted against, such Contributor by reason of your
+   accepting any such warranty or additional liability.
+
+END OF TERMS AND CONDITIONS
+
+APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+Copyright 2023 Pexip AS
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use
+this file except in compliance with the License. You may obtain a copy of the
+License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed
+under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
+CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.

package/README.md

@@ -0,0 +1,193 @@
+# Media Processor
+
+An package for media data processing using Web API.
+
+## Documentation
+
+- [Changelog](./CHANGELOG.md)
+
+## Examples
+
+### Use Analyzer to get the MediaStream data
+
+```typescript
+const stream = await navigator.mediaDevices.getUserMedia({audio:true});
+
+const fftSize = 64;
+// Setup an Audio Graph with `source` -> `analyzer`
+const source = createStreamSourceGraphNode(stream);
+const analyzer = createAnalyzerGraphNode({fftSize});
+const audioGraph = createAudioGraph([[source, analyzer]]);
+
+// Grab the current time domain data in floating point representation
+const buffer = new Float32Array(fftSize);
+analyzer.node?.getFloatTimeDomainData(buffer);
+// Do some work with the buffer
+buffer.forEach(...);
+
+// Get the current volume, [0, 1]
+const volume = analyzer.getAverageVolume(buffer);
+
+// Release the resources when you have done with the analyzer
+await audioGraph.release();
+```
+
+### Use AudioGraph to control the audio gain to mute/unmute
+
+```typescript
+const stream = await navigator.mediaDevices.getUserMedia({
+  audio: true,
+  video: true,
+});
+
+const mute = !stream.getAudioTracks()[0]?.enabled;
+
+// Setup an Audio Graph with `source` -> `gain`
+const source = createStreamSourceGraphNode(stream);
+const gain = createGainGraphNode(mute);
+const destination = createStreamDestinationGraphNode();
+
+const audioGraph = createAudioGraph([[source, gain, destination]]);
+
+// Use the output MediaStream to for the altered AudioTrack
+const alteredStream = new MediaStream([
+  ...stream.getVideoTracks(),
+  ...destination.stream.getAudioTracks(),
+]);
+
+// Mute the audio
+if (gain.node) {
+  gain.node.mute = true;
+}
+
+// Check if the audio is muted
+gain.node.mute; // returns `true`, since we have just set the gain to 0
+
+// Release the resources when you have done with the analyzer
+await audioGraph.release();
+```
+
+### Use noise suppression
+
+```typescript
+const stream = await navigator.mediaDevices.getUserMedia({
+  audio: true,
+  video: true,
+});
+
+// Fetch denoise WebAssembly
+const response = await fetch(
+  new URL('@pexip/denoise/denoise_bg.wasm', import.meta.url).href,
+);
+const wasmBuffer = await response.arrayBuffer();
+
+// Setup an Audio Graph with `source` -> `gain`
+const source = createStreamSourceGraphNode(stream);
+const destination = createStreamDestinationGraphNode();
+
+const audioGraph = createAudioGraph([[source, destination]]);
+// Add worklet module
+await audioGraph.addWorklet(
+new URL(
+        '@pexip/media-processor/dist/worklets/denoise.worklet',
+        import.meta.url,
+      ).href
+);
+const denoise = createDenoiseWorkletGraphNode(wasmBuffer);
+// Route the source through the denoise node
+const audioGraph.connect([source, denoise, destination]);
+const audioGraph.disconnect([source, destination]);
+
+// Release the resources when you have done with the analyzer
+await audioGraph.release();
+```
+
+### Use background blur
+
+```typescript
+import {urls} from '@pexip/bg-blur';
+
+const PROCESSING_WIDTH = 768;
+const PROCESSING_HEIGHT = 432;
+
+const stream = await navigator.mediaDevices.getUserMedia({video: true});
+
+// Create symbolic link to link bg-blur/selfie_segmentation to a public folder
+// Thus we can create a URL object to get the link to be used for the `gluePath`
+const selfieJs = new URL(
+  `./selfie_segmentation/selfie_segmentation.js`,
+  document.baseURI,
+);
+const segmenter = createMediapipeSegmenter('selfie_segmentation', {
+  modelType: selfieModel,
+  processingWidth: PROCESSING_WIDTH,
+  processingHeight: PROCESSING_HEIGHT,
+  gluePath: selfieJs.href,
+});
+
+const transformer = createCanvasTransform(segmenter, {
+  effects: 'blur',
+  width: mediaState.width,
+  height: mediaState.height,
+});
+
+const getTrackProcessor = () => {
+  // Feature detection if the browser has the `MediaStreamProcessor` API
+  if ('MediaStreamTrackProcessor' in window) {
+    return processAPI === 'Streams API'
+      ? createVideoTrackProcessor() // Using the latest Streams API
+      : createVideoTrackProcessorWithFallback(); // Using the fallback implementation
+  }
+  return createVideoTrackProcessorWithFallback(); // Using the fallback implementation
+};
+
+const videoProcessor = createVideoProcessor([transformer], getTrackProcessor());
+
+await videoProcessor.open();
+
+// Passing the raw MediaStream to apply the effects
+// Then, use the output stream for whatever purpose
+const processedStream = await videoProcessor.process(stream);
+```
+
+### More examples
+
+Please check the [Media Processor Demo](../../examples/media-processor-demo)
+
+## Profiling Web Audio
+
+You can do it with chrome, [See here][profiling web audio].
+
+## How AudioWorkletNode and the AudioWorkletProcessor work together
+
+```
+┌─────────────────────────┐             ┌──────────────────────────┐
+│                         │             │                          │
+│    Main Global Scope    │             │  AudioWorkletGlobalScope │
+│                         │             │                          │
+│  ┌───────────────────┐  │             │  ┌────────────────────┐  │
+│  │                   │  │ MessagePort │  │                    │  │
+│  │   AudioWorklet    │◄─┼─────────────┼─►│    AudioWorklet    │  │
+│  │       Node        │  │             │  │      Processor     │  │
+│  │                   │  │             │  │                    │  │
+│  └───────────────────┘  │             │  └────────────────────┘  │
+│                         │             │                          │
+└─────────────────────────┘             └──────────────────────────┘
+       Main Thread                          WebAudio Render Thread
+```
+
+### Constraints when using the AudioWorklet
+
+- Each `BaseAudioContext` possesses exactly one `AudioWorklet`
+- 128 samples-frames
+- No `fetch` API in the `AudioWorkletGlobalScope`
+- No `TextEncoder/Decoder` APIs in the `AudioWorkletGlobalScope`
+
+## References
+
+- [MDN - Web Audio API][web audio api]
+- [Web Audio Spec][web audio spec]
+
+[web audio api]: https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API
+[web audio spec]: https://webaudio.github.io/web-audio-api
+[profiling web audio]: https://web.dev/profiling-web-audio-apps-in-chrome