@tensamin/audio 0.1.2 → 0.1.4

package/README.md

@@ -1,65 +1,56 @@
 # @tensamin/audio
 
-A audio processing library for the web, featuring RNNoise-based noise suppression and robust Voice Activity Detection (VAD). Designed for seamless integration with LiveKit.
+Audio processing library for the web with RNNoise-based noise suppression and Voice Activity Detection (VAD). Designed for voice communication applications with LiveKit integration support.
 
 ## Features
 
-- **Noise Suppression**: Uses `@sapphi-red/web-noise-suppressor` (RNNoise) for high-quality noise reduction.
-- **Robust VAD**: Energy-based VAD with hysteresis, hangover, and pre-roll buffering to prevent cutting off speech onset.
-- **Intelligent Muting**: Automatically gates audio or mutes LiveKit tracks when silent.
-- **LiveKit Integration**: First-class support for `LocalAudioTrack`.
-- **Extensible**: Plugin system for custom WASM/Worklet processors.
+- Configurable Voice Activity Detection with energy-based algorithm
+- RNNoise noise suppression via `@sapphi-red/web-noise-suppressor`
+- Automatic audio gating based on voice detection
+- Runtime configuration updates
+- LiveKit `LocalAudioTrack` integration
+- Plugin system for custom audio processors
+- Optional dynamic range compression
 
 ## Installation
 
 ```bash
 npm install @tensamin/audio livekit-client
-bun add @tensamin/audio livekit-client
-pnpm install @tensamin/audio livekit-client
 ```
 
-## Setup Assets
+## Requirements
 
-This library uses WASM and AudioWorklets for processing. You must manually download these assets and serve them from your application (e.g., in your `public/` folder).
-
-1.  Download the following files from `https://unpkg.com/@sapphi-red/web-noise-suppressor@0.3.5/dist/`:
+For noise suppression, the following files must be provided:
 
 - `rnnoise.wasm`
 - `rnnoise_simd.wasm`
-- `noise-suppressor-worklet.min.js`
+- `worklet.js`
+
+Available at: `https://unpkg.com/@sapphi-red/web-noise-suppressor@0.3.5/dist/`
 
-2.  Place them in your project's public directory (e.g., `public/audio-processor/`).
+Place these files in a publicly accessible directory (e.g., `public/audio-processor/`).
 
 ## Usage
 
-### Basic Usage (Raw MediaStream)
+### Basic Example
 
 ```ts
 import { createAudioPipeline } from "@tensamin/audio";
 
-// Get a stream
 const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
 const track = stream.getAudioTracks()[0];
 
-// Create pipeline
 const pipeline = await createAudioPipeline(track, {
   noiseSuppression: {
     enabled: true,
     wasmUrl: "/audio-processor/rnnoise.wasm",
     simdUrl: "/audio-processor/rnnoise_simd.wasm",
-    workletUrl: "/audio-processor/noise-suppressor-worklet.min.js",
+    workletUrl: "/audio-processor/worklet.js",
   },
   vad: { enabled: true },
 });
 
-// Use the processed track
 const processedStream = new MediaStream([pipeline.processedTrack]);
-// audioElement.srcObject = processedStream;
-
-// Listen to VAD events
-pipeline.events.on("vadChange", (state) => {
-  console.log("Is Speaking:", state.isSpeaking);
-});
 ```
 
 ### LiveKit Integration
@@ -68,21 +59,218 @@ pipeline.events.on("vadChange", (state) => {
 import { attachProcessingToTrack } from "@tensamin/audio";
 import { LocalAudioTrack } from "livekit-client";
 
-// Assume you have a LocalAudioTrack
 const localTrack = await LocalAudioTrack.create();
 
-// Attach processing (replaces the underlying track)
 const pipeline = await attachProcessingToTrack(localTrack, {
   noiseSuppression: {
     enabled: true,
     wasmUrl: "/audio-processor/rnnoise.wasm",
     simdUrl: "/audio-processor/rnnoise_simd.wasm",
-    workletUrl: "/audio-processor/noise-suppressor-worklet.min.js",
+    workletUrl: "/audio-processor/worklet.js",
   },
   vad: { enabled: true },
-  livekit: { manageTrackMute: true }, // Optional: mute the track object itself
+  livekit: { manageTrackMute: true },
 });
 
-// Publish the track
 await room.localParticipant.publishTrack(localTrack);
 ```
+
+### Monitoring VAD State
+
+```ts
+pipeline.events.on("vadChange", (state) => {
+  console.log("Speaking:", state.isSpeaking);
+  console.log("Probability:", state.probability);
+  console.log("State:", state.state);
+});
+```
+
+## Configuration
+
+### Voice Activity Detection
+
+```ts
+vad: {
+  enabled: boolean;
+  startThreshold: number;              // Default: 0.6 (range: 0-1)
+  stopThreshold: number;               // Default: 0.45 (range: 0-1)
+  hangoverMs: number;                  // Default: 400
+  preRollMs: number;                   // Default: 250
+  minSpeechDurationMs: number;         // Default: 100
+  minSilenceDurationMs: number;        // Default: 150
+  energyVad?: {
+    smoothing: number;                 // Default: 0.95
+    initialNoiseFloor: number;         // Default: 0.001
+    noiseFloorAdaptRateQuiet: number;  // Default: 0.01
+    noiseFloorAdaptRateLoud: number;   // Default: 0.001
+    minSNR: number;                    // Default: 2.0
+    snrRange: number;                  // Default: 8.0
+  };
+}
+```
+
+**Threshold Parameters:**
+
+- `startThreshold`: Probability threshold to unmute audio
+- `stopThreshold`: Probability threshold to mute audio (after hangover)
+- `hangoverMs`: Delay before muting after speech stops
+- `preRollMs`: Audio buffer duration before speech onset
+- `minSpeechDurationMs`: Minimum duration to consider as valid speech
+- `minSilenceDurationMs`: Minimum silence duration between speech segments
+
+**Energy VAD Parameters:**
+
+- `smoothing`: Energy calculation smoothing factor (0-1)
+- `minSNR`: Minimum signal-to-noise ratio for speech detection
+- `snrRange`: Range for probability scaling from minSNR
+
+### Output Control
+
+```ts
+output: {
+  speechGain: number;                  // Default: 1.0
+  silenceGain: number;                 // Default: 0.0
+  gainRampTime: number;                // Default: 0.015 (seconds)
+  smoothTransitions: boolean;          // Default: true
+  maxGainDb: number;                   // Default: 6.0
+  enableCompression: boolean;          // Default: false
+  compression?: {
+    threshold: number;                 // Default: -24.0 (dB)
+    ratio: number;                     // Default: 3.0
+    attack: number;                    // Default: 0.003 (seconds)
+    release: number;                   // Default: 0.05 (seconds)
+  };
+}
+```
+
+**Gain Parameters:**
+
+- `speechGain`: Gain multiplier when speaking (1.0 = unity)
+- `silenceGain`: Gain multiplier when silent (0.0 = mute)
+- `gainRampTime`: Transition duration for gain changes
+- `maxGainDb`: Maximum gain limit to prevent clipping
+
+**Compression Parameters:**
+
+- `threshold`: Level above which compression is applied
+- `ratio`: Compression ratio (e.g., 3.0 = 3:1)
+- `attack`: Time to reach full compression
+- `release`: Time to release compression
+
+### Runtime Configuration Updates
+
+```ts
+pipeline.setConfig({
+  vad: {
+    startThreshold: 0.7,
+    stopThreshold: 0.55,
+  },
+  output: {
+    speechGain: 1.3,
+  },
+});
+```
+
+## Configuration Examples
+
+### Noisy Environment
+
+```ts
+{
+  vad: {
+    startThreshold: 0.7,
+    stopThreshold: 0.55,
+    minSpeechDurationMs: 150,
+    energyVad: { minSNR: 3.0 }
+  }
+}
+```
+
+### Quiet Speaker
+
+```ts
+{
+  vad: {
+    startThreshold: 0.4,
+    stopThreshold: 0.25,
+    energyVad: { minSNR: 1.5 }
+  },
+  output: {
+    speechGain: 1.5
+  }
+}
+```
+
+### Natural Conversation
+
+```ts
+{
+  vad: {
+    startThreshold: 0.5,
+    stopThreshold: 0.3,
+    hangoverMs: 600,
+  },
+  output: {
+    silenceGain: 0.2
+  }
+}
+```
+
+## API Reference
+
+### `createAudioPipeline(track, config)`
+
+Creates an audio processing pipeline from a MediaStreamTrack.
+
+**Parameters:**
+
+- `track`: MediaStreamTrack - Source audio track
+- `config`: AudioProcessingConfig - Configuration object
+
+**Returns:** `Promise<AudioPipelineHandle>`
+
+### AudioPipelineHandle
+
+```ts
+interface AudioPipelineHandle {
+  processedTrack: MediaStreamTrack;
+  events: Emitter<AudioPipelineEvents>;
+  state: VADState;
+  setConfig(config: Partial<AudioProcessingConfig>): void;
+  dispose(): void;
+}
+```
+
+### AudioPipelineEvents
+
+```ts
+type AudioPipelineEvents = {
+  vadChange: VADState;
+  error: Error;
+};
+```
+
+### VADState
+
+```ts
+interface VADState {
+  isSpeaking: boolean;
+  probability: number;
+  state: "silent" | "speech_starting" | "speaking" | "speech_ending";
+}
+```
+
+## Default Values
+
+| Parameter              | Default | Description                      |
+| ---------------------- | ------- | -------------------------------- |
+| `startThreshold`       | 0.6     | Unmute at 60% confidence         |
+| `stopThreshold`        | 0.45    | Mute below 45% confidence        |
+| `hangoverMs`           | 400     | Wait 400ms before muting         |
+| `preRollMs`            | 250     | Buffer 250ms before speech       |
+| `minSpeechDurationMs`  | 100     | Minimum valid speech duration    |
+| `minSilenceDurationMs` | 150     | Minimum silence between speech   |
+| `silenceGain`          | 0.0     | Complete mute when silent        |
+| `speechGain`           | 1.0     | Unity gain when speaking         |
+| `minSNR`               | 2.0     | Voice must be 2x noise floor     |
+| `snrRange`             | 8.0     | Probability scales over SNR 2-10 |

package/dist/chunk-AHBRT4RD.mjs

@@ -0,0 +1,307 @@
+import {
+  VADStateMachine
+} from "./chunk-N553RHTI.mjs";
+import {
+  getAudioContext,
+  registerPipeline,
+  unregisterPipeline
+} from "./chunk-OZ7KMC4S.mjs";
+import {
+  getNoiseSuppressionPlugin,
+  getVADPlugin
+} from "./chunk-YOSTLLCS.mjs";
+
+// src/pipeline/audio-pipeline.ts
+import mitt from "mitt";
+async function createAudioPipeline(sourceTrack, config = {}) {
+  const context = getAudioContext();
+  registerPipeline();
+  const nsEnabled = config.noiseSuppression?.enabled !== false && Boolean(
+    config.noiseSuppression?.wasmUrl && config.noiseSuppression?.simdUrl && config.noiseSuppression?.workletUrl
+  );
+  const vadEnabled = config.vad?.enabled !== false;
+  const fullConfig = {
+    noiseSuppression: {
+      enabled: nsEnabled,
+      ...config.noiseSuppression
+    },
+    vad: {
+      enabled: vadEnabled,
+      // Voice-optimized defaults (will be overridden by config)
+      startThreshold: 0.6,
+      stopThreshold: 0.45,
+      hangoverMs: 400,
+      preRollMs: 250,
+      minSpeechDurationMs: 100,
+      minSilenceDurationMs: 150,
+      energyVad: {
+        smoothing: 0.95,
+        initialNoiseFloor: 1e-3,
+        noiseFloorAdaptRateQuiet: 0.01,
+        noiseFloorAdaptRateLoud: 1e-3,
+        minSNR: 2,
+        snrRange: 8
+      },
+      ...config.vad
+    },
+    output: {
+      speechGain: 1,
+      silenceGain: 0,
+      // Full mute for voice-only
+      gainRampTime: 0.015,
+      // Fast but smooth transitions
+      smoothTransitions: true,
+      maxGainDb: 6,
+      enableCompression: false,
+      compression: {
+        threshold: -24,
+        ratio: 3,
+        attack: 3e-3,
+        release: 0.05
+      },
+      ...config.output
+    },
+    livekit: { manageTrackMute: false, ...config.livekit }
+  };
+  console.log("Audio pipeline config:", {
+    noiseSuppression: fullConfig.noiseSuppression?.enabled,
+    vad: fullConfig.vad?.enabled,
+    output: fullConfig.output
+  });
+  if (!sourceTrack || sourceTrack.kind !== "audio") {
+    throw new Error(
+      "createAudioPipeline requires a valid audio MediaStreamTrack"
+    );
+  }
+  if (sourceTrack.readyState === "ended") {
+    throw new Error("Cannot create pipeline from an ended MediaStreamTrack");
+  }
+  const sourceStream = new MediaStream([sourceTrack]);
+  const sourceNode = context.createMediaStreamSource(sourceStream);
+  let nsNode;
+  let vadNode;
+  const emitter = mitt();
+  try {
+    const nsPlugin = getNoiseSuppressionPlugin(
+      fullConfig.noiseSuppression?.pluginName
+    );
+    nsNode = await nsPlugin.createNode(context, fullConfig.noiseSuppression);
+  } catch (error) {
+    const err = error instanceof Error ? error : new Error(String(error));
+    console.error("Failed to create noise suppression node:", err);
+    emitter.emit("error", err);
+    throw err;
+  }
+  const vadStateMachine = new VADStateMachine(fullConfig.vad);
+  try {
+    const vadPlugin = getVADPlugin(fullConfig.vad?.pluginName);
+    vadNode = await vadPlugin.createNode(context, fullConfig.vad, (prob) => {
+      try {
+        const timestamp = context.currentTime * 1e3;
+        const newState = vadStateMachine.processFrame(prob, timestamp);
+        if (newState.state !== lastVadState.state || Math.abs(newState.probability - lastVadState.probability) > 0.1) {
+          emitter.emit("vadChange", newState);
+          lastVadState = newState;
+          updateGain(newState);
+        }
+      } catch (vadError) {
+        const err = vadError instanceof Error ? vadError : new Error(String(vadError));
+        console.error("Error in VAD callback:", err);
+        emitter.emit("error", err);
+      }
+    });
+  } catch (error) {
+    const err = error instanceof Error ? error : new Error(String(error));
+    console.error("Failed to create VAD node:", err);
+    emitter.emit("error", err);
+    throw err;
+  }
+  let lastVadState = {
+    isSpeaking: false,
+    probability: 0,
+    state: "silent"
+  };
+  const splitter = context.createGain();
+  sourceNode.connect(nsNode);
+  nsNode.connect(splitter);
+  splitter.connect(vadNode);
+  const delayNode = context.createDelay(1);
+  const preRollSeconds = (fullConfig.vad?.preRollMs ?? 250) / 1e3;
+  delayNode.delayTime.value = preRollSeconds;
+  const gainNode = context.createGain();
+  gainNode.gain.value = fullConfig.output?.silenceGain ?? 0;
+  let compressor = null;
+  if (fullConfig.output?.enableCompression) {
+    compressor = context.createDynamicsCompressor();
+    const comp = fullConfig.output.compression;
+    compressor.threshold.value = comp.threshold ?? -24;
+    compressor.ratio.value = comp.ratio ?? 3;
+    compressor.attack.value = comp.attack ?? 3e-3;
+    compressor.release.value = comp.release ?? 0.05;
+    compressor.knee.value = 10;
+  }
+  const destination = context.createMediaStreamDestination();
+  try {
+    splitter.connect(delayNode);
+    delayNode.connect(gainNode);
+    if (compressor) {
+      gainNode.connect(compressor);
+      compressor.connect(destination);
+      console.log("Compression enabled:", fullConfig.output?.compression);
+    } else {
+      gainNode.connect(destination);
+    }
+  } catch (error) {
+    const err = error instanceof Error ? error : new Error(String(error));
+    console.error("Failed to wire audio pipeline:", err);
+    emitter.emit("error", err);
+    throw err;
+  }
+  function updateGain(state) {
+    try {
+      const {
+        speechGain = 1,
+        silenceGain = 0,
+        gainRampTime = 0.015,
+        smoothTransitions = true,
+        maxGainDb = 6
+      } = fullConfig.output;
+      const maxGainLinear = Math.pow(10, maxGainDb / 20);
+      const limitedSpeechGain = Math.min(speechGain, maxGainLinear);
+      const targetGain = state.isSpeaking ? limitedSpeechGain : silenceGain;
+      const now = context.currentTime;
+      if (smoothTransitions) {
+        gainNode.gain.cancelScheduledValues(now);
+        gainNode.gain.setValueAtTime(gainNode.gain.value, now);
+        gainNode.gain.setTargetAtTime(targetGain, now, gainRampTime / 3);
+      } else {
+        gainNode.gain.setValueAtTime(targetGain, now);
+      }
+    } catch (error) {
+      const err = error instanceof Error ? error : new Error(String(error));
+      console.error("Failed to update gain:", err);
+      emitter.emit("error", err);
+    }
+  }
+  const audioTracks = destination.stream.getAudioTracks();
+  console.log("Destination stream tracks:", {
+    count: audioTracks.length,
+    tracks: audioTracks.map((t) => ({
+      id: t.id,
+      label: t.label,
+      enabled: t.enabled,
+      readyState: t.readyState
+    }))
+  });
+  if (audioTracks.length === 0) {
+    const err = new Error(
+      "Failed to create processed audio track: destination stream has no audio tracks. This may indicate an issue with the audio graph connection."
+    );
+    console.error(err);
+    emitter.emit("error", err);
+    throw err;
+  }
+  const processedTrack = audioTracks[0];
+  if (!processedTrack || processedTrack.readyState === "ended") {
+    const err = new Error("Processed audio track is invalid or ended");
+    console.error(err);
+    emitter.emit("error", err);
+    throw err;
+  }
+  console.log("Audio pipeline created successfully:", {
+    sourceTrack: {
+      id: sourceTrack.id,
+      label: sourceTrack.label,
+      readyState: sourceTrack.readyState
+    },
+    processedTrack: {
+      id: processedTrack.id,
+      label: processedTrack.label,
+      readyState: processedTrack.readyState
+    },
+    config: {
+      noiseSuppression: fullConfig.noiseSuppression?.enabled,
+      vad: fullConfig.vad?.enabled
+    }
+  });
+  function dispose() {
+    try {
+      sourceNode.disconnect();
+      nsNode.disconnect();
+      splitter.disconnect();
+      vadNode.disconnect();
+      delayNode.disconnect();
+      gainNode.disconnect();
+      if (compressor) {
+        compressor.disconnect();
+      }
+      destination.stream.getTracks().forEach((t) => t.stop());
+      unregisterPipeline();
+    } catch (error) {
+      console.error("Error during pipeline disposal:", error);
+    }
+  }
+  return {
+    processedTrack,
+    events: emitter,
+    get state() {
+      return lastVadState;
+    },
+    setConfig: (newConfig) => {
+      try {
+        if (newConfig.vad) {
+          vadStateMachine.updateConfig(newConfig.vad);
+          Object.assign(fullConfig.vad, newConfig.vad);
+          if (newConfig.vad.preRollMs !== void 0) {
+            const preRollSeconds2 = newConfig.vad.preRollMs / 1e3;
+            delayNode.delayTime.setValueAtTime(
+              preRollSeconds2,
+              context.currentTime
+            );
+          }
+        }
+        if (newConfig.output) {
+          Object.assign(fullConfig.output, newConfig.output);
+          updateGain(lastVadState);
+          if (compressor && newConfig.output.compression) {
+            const comp = newConfig.output.compression;
+            if (comp.threshold !== void 0) {
+              compressor.threshold.setValueAtTime(
+                comp.threshold,
+                context.currentTime
+              );
+            }
+            if (comp.ratio !== void 0) {
+              compressor.ratio.setValueAtTime(comp.ratio, context.currentTime);
+            }
+            if (comp.attack !== void 0) {
+              compressor.attack.setValueAtTime(
+                comp.attack,
+                context.currentTime
+              );
+            }
+            if (comp.release !== void 0) {
+              compressor.release.setValueAtTime(
+                comp.release,
+                context.currentTime
+              );
+            }
+          }
+        }
+        if (newConfig.livekit) {
+          Object.assign(fullConfig.livekit, newConfig.livekit);
+        }
+        console.log("Pipeline config updated:", newConfig);
+      } catch (error) {
+        const err = error instanceof Error ? error : new Error(String(error));
+        console.error("Failed to update config:", err);
+        emitter.emit("error", err);
+      }
+    },
+    dispose
+  };
+}
+
+export {
+  createAudioPipeline
+};

package/dist/chunk-ERJVV5JR.mjs

@@ -0,0 +1,91 @@
+import {
+  createAudioPipeline
+} from "./chunk-AHBRT4RD.mjs";
+
+// src/livekit/integration.ts
+async function attachProcessingToTrack(track, config = {}) {
+  if (!track) {
+    throw new Error("attachProcessingToTrack requires a valid LocalAudioTrack");
+  }
+  const originalTrack = track.mediaStreamTrack;
+  if (!originalTrack) {
+    throw new Error("LocalAudioTrack has no underlying MediaStreamTrack");
+  }
+  if (originalTrack.readyState === "ended") {
+    throw new Error("Cannot attach processing to an ended MediaStreamTrack");
+  }
+  let pipeline;
+  try {
+    console.log("Creating audio processing pipeline...");
+    pipeline = await createAudioPipeline(originalTrack, config);
+    console.log("Audio processing pipeline created successfully");
+  } catch (error) {
+    const err = new Error(
+      `Failed to create audio pipeline: ${error instanceof Error ? error.message : String(error)}`
+    );
+    console.error(err);
+    throw err;
+  }
+  if (!pipeline.processedTrack) {
+    throw new Error("Pipeline did not return a processed track");
+  }
+  try {
+    console.log("Replacing LiveKit track with processed track...");
+    await track.replaceTrack(pipeline.processedTrack);
+    console.log("LiveKit track replaced successfully");
+  } catch (error) {
+    pipeline.dispose();
+    const err = new Error(
+      `Failed to replace LiveKit track: ${error instanceof Error ? error.message : String(error)}`
+    );
+    console.error(err);
+    throw err;
+  }
+  if (config.livekit?.manageTrackMute) {
+    let isVadMuted = false;
+    pipeline.events.on("vadChange", async (state) => {
+      try {
+        if (state.isSpeaking) {
+          if (isVadMuted) {
+            await track.unmute();
+            isVadMuted = false;
+          }
+        } else {
+          if (!track.isMuted) {
+            await track.mute();
+            isVadMuted = true;
+          }
+        }
+      } catch (error) {
+        console.error("Error handling VAD-based track muting:", error);
+      }
+    });
+  }
+  pipeline.events.on("error", (error) => {
+    console.error("Audio pipeline error:", error);
+  });
+  const originalDispose = pipeline.dispose;
+  pipeline.dispose = () => {
+    try {
+      if (originalTrack.readyState === "live") {
+        console.log("Restoring original track...");
+        track.replaceTrack(originalTrack).catch((error) => {
+          console.error("Failed to restore original track:", error);
+        });
+      }
+      originalDispose();
+    } catch (error) {
+      console.error("Error during pipeline disposal:", error);
+      try {
+        originalDispose();
+      } catch (disposeError) {
+        console.error("Error calling original dispose:", disposeError);
+      }
+    }
+  };
+  return pipeline;
+}
+
+export {
+  attachProcessingToTrack
+};