omnivad 0.2.8 → 0.2.10

package/README.md

@@ -0,0 +1,194 @@
+# omnivad
+
+[![npm](https://img.shields.io/npm/v/omnivad)](https://www.npmjs.com/package/omnivad)
+[![npm bundle size](https://img.shields.io/bundlephobia/min/omnivad)](https://bundlephobia.com/package/omnivad)
+[![license](https://img.shields.io/npm/l/omnivad)](https://github.com/lifeiteng/OmniVAD-Kit/blob/main/LICENSE)
+
+Cross-platform Voice Activity Detection and Audio Event Detection via WebAssembly.
+Runs in **browsers, Web Workers, and Node.js** with a single API. Zero runtime
+dependencies. Built on [FireRedVAD](https://github.com/FireRedTeam/FireRedVAD)
+from Xiaohongshu (DFSMN architecture, ~2.2 MB per model).
+
+## What's in the box
+
+| Class | Use case | Output |
+|-------|----------|--------|
+| **`OmniVAD`** | Whole-audio voice activity detection | `[start, end]` timestamps |
+| **`OmniStreamVAD`** | Real-time, frame-by-frame VAD with segment-boundary events | per-frame probability + start/end events |
+| **`OmniAED`** | Audio event detection (3-class) | `speech` / `singing` / `music` timestamps |
+| **`mergeChunks`** | Pack VAD output into Whisper-style 30 s chunks | `{ start, end, segStartIdx, segCount }[]` |
+
+All four share one WASM module (~2.2 MB SIMD-enabled), one C implementation,
+and a single bundle (~24 KB JS, ESM + CJS + types).
+
+## Install
+
+```bash
+pnpm add omnivad     # or: npm install omnivad / yarn add omnivad
+```
+
+Models are served from jsDelivr by default (zero config). For air-gapped or
+custom deployments, pass `modelUrl` or pre-loaded `modelData`.
+
+## Quickstart — whole-audio VAD
+
+```ts
+import { OmniVAD } from "omnivad";
+
+const vad = await OmniVAD.create();
+
+// Float32Array in [-1, 1] (Web Audio, decodeAudioData) or Int16Array (raw PCM)
+const result = vad.detect(audioFloat32);
+// { duration: 12.4, timestamps: [[0.35, 4.8], [5.1, 12.4]] }
+```
+
+## Streaming VAD — real-time, frame-by-frame
+
+`OmniStreamVAD` processes 10 ms frames (160 samples @ 16 kHz) and emits
+segment-boundary events on the same call that confirms the boundary —
+bit-identical to upstream FireRedVAD's `FireRedStreamVad`.
+
+`processFrame()` accepts `Float32Array` in `[-1, 1]` (Web Audio,
+`AudioWorkletProcessor`, decoded WebRTC tracks) or `Int16Array` PCM
+(WAV / microphone). Dispatch is by dtype — no scaling in JS.
+
+```ts
+import { OmniStreamVAD } from "omnivad";
+
+const vad = await OmniStreamVAD.create();
+
+// Float32Array [-1, 1] from Web Audio:
+for (let i = 0; i + 160 <= floatPcm.length; i += 160) {
+  const r = vad.processFrame(floatPcm.subarray(i, i + 160));
+  if (!r) continue;
+  if (r.isSpeechStart) console.log(`START @ ${(r.speechStartFrame * 0.01).toFixed(2)}s`);
+  if (r.isSpeechEnd)   console.log(`END   @ ${(r.speechEndFrame   * 0.01).toFixed(2)}s`);
+}
+
+// Or Int16Array PCM from a WAV file — same call, same result:
+for (let i = 0; i + 160 <= int16Pcm.length; i += 160) {
+  vad.processFrame(int16Pcm.subarray(i, i + 160));
+}
+```
+
+`processFrame()` returns `{ confidence, smoothedProb, isSpeech, isSpeechStart,
+isSpeechEnd, frameIdx, speechStartFrame, speechEndFrame }` — every field comes
+straight from the C state machine.
+
+## Audio Event Detection — speech / singing / music
+
+```ts
+import { OmniAED } from "omnivad";
+
+const aed = await OmniAED.create();
+const events = aed.detect(audioFloat32);
+// { duration: 22.0,
+//   events: { speech: [[...]], singing: [[...]], music: [[...]] },
+//   ratios: { speech: 0.41, singing: 0.0, music: 0.59 } }
+```
+
+## Whisper / WhisperX-style chunking
+
+`OmniVAD` + `mergeChunks(mode: "greedy")` is the 1:1 equivalent of WhisperX's
+`Binarize(max_duration=chunk_size)` + greedy packing. Use this recipe when
+feeding chunks into Whisper-family ASR models that expect a fixed 30 s window:
+
+```ts
+import { OmniVAD, mergeChunks } from "omnivad";
+
+const vad = await OmniVAD.create();                 // threshold=0.4 default — safer for Whisper
+const result = vad.detect(audioFloat32);
+
+const chunks = await mergeChunks(result.timestamps, {
+  maxChunkSecs:    30.0,                            // Whisper input window
+  mode:            "greedy",                        // WhisperX behavior
+  padOnsetSecs:    0.04,
+  padOffsetSecs:   0.04,
+  minSilenceSecs:  0.20,
+});
+// Slice the audio at [chunk.start, chunk.end] and feed each slice to Whisper.
+```
+
+A second mode `"longest_gap"` exists for variable-length-input models
+(forced alignment, TTS) — see the GitHub README for the comparison table.
+
+## Multi-stream concurrency
+
+`OmniStreamVAD` instances have mutable per-stream state and **must not** be
+shared across concurrent streams. Use `clone()` to spin up a fresh instance
+that shares the underlying model weights but has its own state — instant,
+near-zero memory overhead per stream.
+
+```ts
+const base = await OmniStreamVAD.create();
+const streamA = base.clone();
+const streamB = base.clone();
+// Process two independent audio sessions in parallel.
+```
+
+## Models and CDN
+
+By default, models are fetched from jsDelivr:
+
+```
+https://cdn.jsdelivr.net/npm/omnivad@<version>/models/{vad,stream-vad,aed}.omnivad
+```
+
+Override per call when you need to host them yourself or pre-bundle:
+
+```ts
+const vad = await OmniVAD.create({
+  modelUrl: "https://your-cdn/vad.omnivad",   // or
+  modelData: arrayBufferYouAlreadyHave,
+});
+```
+
+In Node.js, models are read from the installed package (`omnivad/models/`) — no
+network access required at runtime.
+
+## Performance
+
+Real-Time Factor (lower = faster) on Apple M-series:
+
+| Model | RTF | Speed |
+|-------|-----|-------|
+| VAD | ~0.003 | ~330× real-time |
+| Streaming VAD | ~0.002 | ~500× real-time |
+| AED | ~0.002 | ~500× real-time |
+
+WASM is built with SIMD enabled and ncnn fp16 weights.
+
+## Accuracy
+
+Verified bit-identical to upstream PyTorch reference on 5 audio files × 3
+models — see the [accuracy table](https://github.com/lifeiteng/OmniVAD-Kit#testing)
+in the main repo.
+
+## Browser, Worker, Node — same API
+
+The package detects its runtime and loads the right glue:
+
+- **Browsers (main thread)** — classic-script injection of the Emscripten glue
+  (works around `MODULARIZE=1` IIFE issues with `import()`).
+- **Web Workers / ServiceWorkers** — same path via `importScripts`.
+- **Node.js (≥ 18)** — `createRequire` + local CJS resolution. No bundler
+  config needed.
+
+## See also
+
+- Full documentation, accuracy tables, C/C++ API, Python package, native build:
+  [GitHub repository](https://github.com/lifeiteng/OmniVAD-Kit)
+- [中文 README](https://github.com/lifeiteng/OmniVAD-Kit/blob/main/README.zh.md)
+- [Local development guide](https://github.com/lifeiteng/OmniVAD-Kit#local-development)
+
+## Credits
+
+- [**FireRedVAD**](https://github.com/FireRedTeam/FireRedVAD) — Kaituo Xu,
+  Wenpeng Li, Kai Huang, Kun Liu (Xiaohongshu). Source models, DFSMN
+  architecture, training pipeline.
+- [ncnn](https://github.com/Tencent/ncnn) — Tencent. Inference backend.
+- [Emscripten](https://emscripten.org/) — WebAssembly toolchain.
+
+## License
+
+Apache-2.0 — same as upstream FireRedVAD.

package/dist/index.cjs

@@ -38,7 +38,7 @@ var SIZEOF_AED_SEGMENT = 16;
 var SIZEOF_CHUNK_CONFIG = 28;
 var SIZEOF_CHUNK = 16;
 var OMNI_ERR_NO_FRAMES = -7;
-var VERSION = "0.2.8";
+var VERSION = "0.2.10";
 var DEFAULT_CDN_BASE = `https://cdn.jsdelivr.net/npm/omnivad@${VERSION}/models`;
 var MODEL_FILES = {
   vad: "vad.omnivad",
@@ -163,6 +163,23 @@ function copyAudioToHeap(M, audio) {
   heap.set(audio);
   return ptr;
 }
+function copyInt16ToHeap(M, audio) {
+  const ptr = M._malloc(audio.length * 2);
+  const heap = new Int16Array(M.HEAPU8.buffer, ptr, audio.length);
+  heap.set(audio);
+  return ptr;
+}
+function dispatchAudio(M, audio) {
+  if (audio instanceof Float32Array) {
+    return { ptr: copyAudioToHeap(M, audio), length: audio.length, format: "f32" };
+  }
+  if (audio instanceof Int16Array) {
+    return { ptr: copyInt16ToHeap(M, audio), length: audio.length, format: "int16" };
+  }
+  throw new TypeError(
+    `unsupported audio dtype; expected Float32Array in [-1, 1] or Int16Array`
+  );
+}
 function writePostConfig(M, ptr, cfg) {
   M.setValue(ptr + 0, cfg.threshold, "float");
   M.setValue(ptr + 4, cfg.smoothWindowSize, "i32");
@@ -402,14 +419,15 @@ function streamVadCreate(M, modelBuffer, config = {}) {
   }
 }
 var SIZEOF_STREAM_VAD_RESULT = 24;
-function streamVadProcess(M, handle, pcm16Ptr, numSamples) {
+function streamVadProcess(M, handle, audioPtr, numSamples, format = "f32") {
   const resultPtr = M._malloc(SIZEOF_STREAM_VAD_RESULT);
+  const fn = format === "int16" ? "omni_stream_vad_process_int16" : "omni_stream_vad_process";
   try {
     const ret = M.ccall(
-      "omni_stream_vad_process",
+      fn,
       "number",
       ["number", "number", "number", "number"],
-      [handle, pcm16Ptr, numSamples, resultPtr]
+      [handle, audioPtr, numSamples, resultPtr]
     );
     if (ret === OMNI_ERR_NO_FRAMES) return null;
     if (ret !== 0) throw new Error(`StreamVAD process failed: ${ret}`);
@@ -427,6 +445,28 @@ function streamVadProcess(M, handle, pcm16Ptr, numSamples) {
     M._free(resultPtr);
   }
 }
+function streamVadDetectFull(M, handle, audioPtr, numSamples, format = "f32") {
+  const probsPtrPtr = M._malloc(4);
+  const framesPtr = M._malloc(4);
+  const fn = format === "int16" ? "omni_stream_vad_detect_full_int16" : "omni_stream_vad_detect_full";
+  try {
+    const ret = M.ccall(
+      fn,
+      "number",
+      ["number", "number", "number", "number", "number"],
+      [handle, audioPtr, numSamples, probsPtrPtr, framesPtr]
+    );
+    if (ret !== 0) throw new Error(`StreamVAD detectFull failed: ${ret}`);
+    const numFrames = M.getValue(framesPtr, "i32");
+    const probsPtr = M.getValue(probsPtrPtr, "i32");
+    const probabilities = probsPtr ? new Float32Array(new Float32Array(M.HEAPU8.buffer, probsPtr, numFrames)) : new Float32Array(0);
+    if (probsPtr) M._free(probsPtr);
+    return { probabilities, numFrames };
+  } finally {
+    M._free(probsPtrPtr);
+    M._free(framesPtr);
+  }
+}
 function streamVadClone(M, handle) {
   const errPtr = M._malloc(4);
   try {
@@ -486,7 +526,7 @@ var OmniVAD = class _OmniVAD {
    */
   detect(audio) {
     const M = getModule();
-    const { ptr, length, format } = prepareAudio(M, audio);
+    const { ptr, length, format } = dispatchAudio(M, audio);
     try {
       const timestamps = vadDetect(M, this.handle, ptr, length, this.config, format);
       return {
@@ -505,16 +545,6 @@ var OmniVAD = class _OmniVAD {
     }
   }
 };
-function prepareAudio(M, audio) {
-  const f32 = audio instanceof Int16Array ? int16ToNormalizedFloat32(audio) : audio;
-  const ptr = copyAudioToHeap(M, f32);
-  return { ptr, length: f32.length, format: "f32" };
-}
-function int16ToNormalizedFloat32(i16) {
-  const f32 = new Float32Array(i16.length);
-  for (let i = 0; i < i16.length; i++) f32[i] = i16[i] / 32768;
-  return f32;
-}
 
 // src/stream-vad.ts
 var SAMPLE_RATE2 = 16e3;
@@ -553,20 +583,23 @@ var OmniStreamVAD = class _OmniStreamVAD {
     return new _OmniStreamVAD(newHandle);
   }
   /**
-   * Process one frame of audio (160 int16 samples = 10ms @ 16kHz).
+   * Process one frame of audio (160 samples = 10ms @ 16kHz).
+   *
+   * Accepts Float32Array in [-1, 1] (Web Audio, soundfile, torch) or
+   * Int16Array PCM (WAV, microphone). Dispatches by dtype to the matching
+   * C entry — no scaling in JS.
+   *
    * Returns null until enough audio is accumulated.
    *
    * Segment-boundary events (isSpeechStart / isSpeechEnd and the matching
    * speech_*_frame indices) come straight from the C-layer state machine
    * (bit-identical to upstream FireRedVAD) — the wrapper is just a marshaller.
    */
-  processFrame(pcm160) {
+  processFrame(audio) {
     const M = getModule();
-    const ptr = M._malloc(pcm160.length * 2);
-    const heap16 = new Int16Array(M.HEAPU8.buffer, ptr, pcm160.length);
-    heap16.set(pcm160);
+    const { ptr, length, format } = dispatchAudio(M, audio);
     try {
-      const result = streamVadProcess(M, this.handle, ptr, pcm160.length);
+      const result = streamVadProcess(M, this.handle, ptr, length, format);
       if (!result) return null;
       return {
         confidence: result.confidence,
@@ -588,31 +621,22 @@ var OmniStreamVAD = class _OmniStreamVAD {
    */
   detectFull(audio) {
     const M = getModule();
-    const f32 = prepareDetectFullAudio(audio);
-    const audioPtr = copyAudioToHeap(M, f32);
-    const probsPtrPtr = M._malloc(4);
-    const framesPtr = M._malloc(4);
+    const { ptr, length, format } = dispatchAudio(M, audio);
     try {
-      const ret = M.ccall(
-        "omni_stream_vad_detect_full",
-        "number",
-        ["number", "number", "number", "number", "number"],
-        [this.handle, audioPtr, f32.length, probsPtrPtr, framesPtr]
+      const { probabilities, numFrames } = streamVadDetectFull(
+        M,
+        this.handle,
+        ptr,
+        length,
+        format
       );
-      if (ret !== 0) throw new Error(`StreamVAD detectFull failed: ${ret}`);
-      const numFrames = M.getValue(framesPtr, "i32");
-      const probsPtr = M.getValue(probsPtrPtr, "i32");
-      const probabilities = probsPtr ? new Float32Array(new Float32Array(M.HEAPU8.buffer, probsPtr, numFrames)) : new Float32Array(0);
-      if (probsPtr) M._free(probsPtr);
       return {
         probabilities,
         numFrames,
-        duration: Math.round(f32.length / SAMPLE_RATE2 * 1e3) / 1e3
+        duration: Math.round(length / SAMPLE_RATE2 * 1e3) / 1e3
       };
     } finally {
-      M._free(audioPtr);
-      M._free(probsPtrPtr);
-      M._free(framesPtr);
+      M._free(ptr);
     }
   }
   /** Reset all internal state (model cache, audio buffer, postprocessor). */
@@ -627,31 +651,6 @@ var OmniStreamVAD = class _OmniStreamVAD {
     }
   }
 };
-function int16ToFloat32(i16) {
-  const f32 = new Float32Array(i16.length);
-  for (let i = 0; i < i16.length; i++) f32[i] = i16[i];
-  return f32;
-}
-function prepareDetectFullAudio(audio) {
-  if (audio instanceof Int16Array) {
-    return int16ToFloat32(audio);
-  }
-  if (isNormalizedFloat(audio)) {
-    const scaled = new Float32Array(audio.length);
-    for (let i = 0; i < audio.length; i++) scaled[i] = audio[i] * 32768;
-    return scaled;
-  }
-  return audio;
-}
-function isNormalizedFloat(audio) {
-  const step = Math.max(1, Math.floor(audio.length / 1e3));
-  let maxAbs = 0;
-  for (let i = 0; i < audio.length; i += step) {
-    const v = Math.abs(audio[i]);
-    if (v > maxAbs) maxAbs = v;
-  }
-  return maxAbs <= 1;
-}
 
 // src/aed.ts
 var SAMPLE_RATE3 = 16e3;
@@ -691,7 +690,7 @@ var OmniAED = class _OmniAED {
    */
   detect(audio) {
     const M = getModule();
-    const { ptr, length, format } = prepareAudio2(M, audio);
+    const { ptr, length, format } = dispatchAudio(M, audio);
     const duration = Math.round(length / SAMPLE_RATE3 * 1e3) / 1e3;
     try {
       const events = aedDetect(M, this.handle, ptr, length, this.config, format);
@@ -712,18 +711,6 @@ var OmniAED = class _OmniAED {
     }
   }
 };
-function prepareAudio2(M, audio) {
-  const f32 = audio instanceof Int16Array ? int16ToNormalizedFloat322(audio) : audio;
-  const ptr = M._malloc(f32.length * 4);
-  const heap = new Float32Array(M.HEAPU8.buffer, ptr, f32.length);
-  heap.set(f32);
-  return { ptr, length: f32.length, format: "f32" };
-}
-function int16ToNormalizedFloat322(i16) {
-  const f32 = new Float32Array(i16.length);
-  for (let i = 0; i < i16.length; i++) f32[i] = i16[i] / 32768;
-  return f32;
-}
 function computeCoverageRatios(events, duration) {
   const ratios = {
     speech: 0,