livekit-plugins-dtln 0.1.0__tar.gz

livekit_plugins_dtln-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Aloware, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

livekit_plugins_dtln-0.1.0/PKG-INFO

@@ -0,0 +1,228 @@
+Metadata-Version: 2.4
+Name: livekit-plugins-dtln
+Version: 0.1.0
+Summary: DTLN noise suppression plugin for LiveKit Agents — self-hosted, in-process, no cloud API
+Author-email: "Aloware, Inc." <dev@aloware.com>
+License: MIT License
+        
+        Copyright (c) 2024 Aloware, Inc.
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Project-URL: Homepage, https://github.com/aloware/livekit-plugins-dtln
+Project-URL: Repository, https://github.com/aloware/livekit-plugins-dtln
+Project-URL: Bug Tracker, https://github.com/aloware/livekit-plugins-dtln/issues
+Project-URL: Demo, https://aloware.github.io/livekit-plugins-dtln/
+Keywords: livekit,noise-suppression,noise-cancellation,dtln,onnx,speech,audio,voip
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Multimedia :: Sound/Audio :: Speech
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: livekit>=1.1.0
+Requires-Dist: livekit-agents>=1.4.4
+Requires-Dist: onnxruntime>=1.17.0
+Requires-Dist: numpy>=1.26.0
+Dynamic: license-file
+
+# livekit-plugins-dtln
+
+Python [LiveKit](https://livekit.io) plugin for **DTLN** (Dual-Signal Transformation LSTM Network) noise suppression — a fully self-hosted, open-source alternative to cloud-based noise cancellation services like Krisp or AI-coustics.
+
+Runs entirely **in-process** using [ONNX Runtime](https://onnxruntime.ai). No cloud API, no per-minute fees, no proprietary binaries. Works with **self-hosted LiveKit** servers.
+
+> Based on [Westhausen & Meyer, "Noise Reduction with DTLN", Interspeech 2020](https://www.isca-archive.org/interspeech_2020/westhausen20_interspeech.html)
+> Original implementation: [github.com/breizhn/DTLN](https://github.com/breizhn/DTLN)
+
+**[Live audio comparison demo →](https://aloware.github.io/livekit-plugins-dtln/)**
+
+---
+
+## Why DTLN?
+
+| | DTLN (this plugin) | Krisp / AI-coustics |
+|---|---|---|
+| **Hosting** | Self-hosted, in-process | Cloud API required |
+| **Cost** | Free (open weights) | Per-minute billing |
+| **LiveKit** | Works with self-hosted | Requires LiveKit Cloud |
+| **Latency** | ~8 ms (one block shift) | Network round-trip |
+| **Privacy** | Audio never leaves your server | Audio sent to third party |
+| **Real-time factor** | ~0.05× (20× faster than real-time) | Varies |
+
+---
+
+## Installation
+
+**pip:**
+
+```bash
+pip install livekit-plugins-dtln
+```
+
+**requirements.txt:**
+
+```
+livekit-plugins-dtln
+```
+
+**From source:**
+
+```bash
+git clone https://github.com/aloware/livekit-plugins-dtln.git
+pip install -e ./livekit-plugins-dtln
+```
+
+> The pretrained ONNX model weights (~4 MB) are bundled in the PyPI wheel — no separate download step needed.
+
+---
+
+## Usage
+
+### Session pipeline (recommended)
+
+```python
+from livekit.agents import room_io
+from livekit.plugins import dtln
+
+await session.start(
+    # ...,
+    room_options=room_io.RoomOptions(
+        audio_input=room_io.AudioInputOptions(
+            noise_cancellation=dtln.noise_suppression(),
+        ),
+    ),
+)
+```
+
+### Custom AudioStream
+
+```python
+from livekit import rtc
+from livekit.plugins import dtln
+
+stream = rtc.AudioStream.from_track(
+    track=track,
+    noise_cancellation=dtln.noise_suppression(),
+)
+```
+
+> **Note:** Create one `dtln.noise_suppression()` instance **per session**. Each instance holds stateful LSTM hidden states that must be scoped to a single call.
+
+> **Note:** DTLN is trained on raw microphone audio. Do not chain it with another noise cancellation model — applying two models in series produces unexpected results.
+
+### Custom model paths
+
+```python
+dtln.noise_suppression(
+    model_1_path="/path/to/model_1.onnx",
+    model_2_path="/path/to/model_2.onnx",
+)
+```
+
+---
+
+## Requirements
+
+- Python >= 3.10
+- livekit >= 1.1.0
+- livekit-agents >= 1.4.4
+- onnxruntime >= 1.17.0
+- numpy >= 1.26.0
+
+---
+
+## How It Works
+
+DTLN uses two sequential LSTM-based models:
+
+1. **Model 1 — Spectral masking**: Computes the magnitude spectrum of a 32 ms window, runs it through an LSTM to produce a spectral mask, applies the mask in the frequency domain (preserving phase), and reconstructs the time-domain signal via IFFT.
+
+2. **Model 2 — Time-domain refinement**: Refines the output of Model 1 with a second LSTM that operates directly on the waveform, capturing residual artifacts that spectral processing misses.
+
+The two models are chained: Model 1's output feeds Model 2. Both LSTMs are stateful — their hidden states persist across audio frames, giving the network temporal context across the full duration of a call.
+
+**Signal flow:**
+
+```
+Input frame (any sample rate, any channels)
+  → downsample to 16 kHz mono
+  → overlap-add loop (512-sample window, 128-sample shift)
+      → FFT → magnitude → Model 1 (spectral mask) → masked IFFT
+      → Model 2 (time-domain refinement)
+  → upsample back to original sample rate
+  → restore original channel count
+→ Denoised output frame
+```
+
+The overlap-add synthesis uses 75% overlap (512-sample window, 128-sample shift), identical to the original DTLN paper. This gives ~8 ms of algorithmic latency at 16 kHz.
+
+---
+
+## Performance
+
+Benchmarked on Apple M3 Pro, processing 16 kHz mono audio:
+
+| Metric | Value |
+|---|---|
+| Steady-state latency per block | ~0.7 ms |
+| Real-time factor | ~0.05× |
+| Headroom vs real-time | ~20× |
+| Cold-start (first inference) | ~500 ms (amortized by warmup in `__init__`) |
+
+The `__init__` method runs a dummy forward pass to trigger ONNX Runtime's JIT compilation before the first real audio frame arrives, eliminating the cold-start stall.
+
+---
+
+## Models
+
+Pretrained weights are the official DTLN models published by the original authors:
+
+| File | Source |
+|---|---|
+| `model_1.onnx` | [breizhn/DTLN · pretrained_model/](https://github.com/breizhn/DTLN/tree/master/pretrained_model) |
+| `model_2.onnx` | [breizhn/DTLN · pretrained_model/](https://github.com/breizhn/DTLN/tree/master/pretrained_model) |
+
+The models are not bundled in this repository (to keep it lightweight). They are downloaded automatically by `python agent.py download-files` or by calling `download_models()` directly.
+
+---
+
+## References
+
+- **Original DTLN paper**: [Westhausen & Meyer, "Noise Reduction with DTLN", Interspeech 2020](https://www.isca-archive.org/interspeech_2020/westhausen20_interspeech.html)
+- **Original DTLN implementation & pretrained models**: [github.com/breizhn/DTLN](https://github.com/breizhn/DTLN)
+- **DataDog engineering article** — the inspiration for this plugin: [Building a Real-Time Noise Suppression Library](https://www.datadoghq.com/blog/engineering/noise-suppression-library/)
+- **LiveKit noise cancellation overview**: [docs.livekit.io — Noise Cancellation](https://docs.livekit.io/transport/media/noise-cancellation/)
+- **LiveKit Agents SDK**: [github.com/livekit/agents](https://github.com/livekit/agents)
+- **ONNX Runtime**: [onnxruntime.ai](https://onnxruntime.ai)
+
+---
+
+## License
+
+The plugin code in this repository is released under the **MIT License**.
+
+The pretrained DTLN model weights are published by the original authors under the **MIT License** — see [breizhn/DTLN](https://github.com/breizhn/DTLN/blob/master/LICENSE).

livekit_plugins_dtln-0.1.0/README.md

@@ -0,0 +1,177 @@
+# livekit-plugins-dtln
+
+Python [LiveKit](https://livekit.io) plugin for **DTLN** (Dual-Signal Transformation LSTM Network) noise suppression — a fully self-hosted, open-source alternative to cloud-based noise cancellation services like Krisp or AI-coustics.
+
+Runs entirely **in-process** using [ONNX Runtime](https://onnxruntime.ai). No cloud API, no per-minute fees, no proprietary binaries. Works with **self-hosted LiveKit** servers.
+
+> Based on [Westhausen & Meyer, "Noise Reduction with DTLN", Interspeech 2020](https://www.isca-archive.org/interspeech_2020/westhausen20_interspeech.html)
+> Original implementation: [github.com/breizhn/DTLN](https://github.com/breizhn/DTLN)
+
+**[Live audio comparison demo →](https://aloware.github.io/livekit-plugins-dtln/)**
+
+---
+
+## Why DTLN?
+
+| | DTLN (this plugin) | Krisp / AI-coustics |
+|---|---|---|
+| **Hosting** | Self-hosted, in-process | Cloud API required |
+| **Cost** | Free (open weights) | Per-minute billing |
+| **LiveKit** | Works with self-hosted | Requires LiveKit Cloud |
+| **Latency** | ~8 ms (one block shift) | Network round-trip |
+| **Privacy** | Audio never leaves your server | Audio sent to third party |
+| **Real-time factor** | ~0.05× (20× faster than real-time) | Varies |
+
+---
+
+## Installation
+
+**pip:**
+
+```bash
+pip install livekit-plugins-dtln
+```
+
+**requirements.txt:**
+
+```
+livekit-plugins-dtln
+```
+
+**From source:**
+
+```bash
+git clone https://github.com/aloware/livekit-plugins-dtln.git
+pip install -e ./livekit-plugins-dtln
+```
+
+> The pretrained ONNX model weights (~4 MB) are bundled in the PyPI wheel — no separate download step needed.
+
+---
+
+## Usage
+
+### Session pipeline (recommended)
+
+```python
+from livekit.agents import room_io
+from livekit.plugins import dtln
+
+await session.start(
+    # ...,
+    room_options=room_io.RoomOptions(
+        audio_input=room_io.AudioInputOptions(
+            noise_cancellation=dtln.noise_suppression(),
+        ),
+    ),
+)
+```
+
+### Custom AudioStream
+
+```python
+from livekit import rtc
+from livekit.plugins import dtln
+
+stream = rtc.AudioStream.from_track(
+    track=track,
+    noise_cancellation=dtln.noise_suppression(),
+)
+```
+
+> **Note:** Create one `dtln.noise_suppression()` instance **per session**. Each instance holds stateful LSTM hidden states that must be scoped to a single call.
+
+> **Note:** DTLN is trained on raw microphone audio. Do not chain it with another noise cancellation model — applying two models in series produces unexpected results.
+
+### Custom model paths
+
+```python
+dtln.noise_suppression(
+    model_1_path="/path/to/model_1.onnx",
+    model_2_path="/path/to/model_2.onnx",
+)
+```
+
+---
+
+## Requirements
+
+- Python >= 3.10
+- livekit >= 1.1.0
+- livekit-agents >= 1.4.4
+- onnxruntime >= 1.17.0
+- numpy >= 1.26.0
+
+---
+
+## How It Works
+
+DTLN uses two sequential LSTM-based models:
+
+1. **Model 1 — Spectral masking**: Computes the magnitude spectrum of a 32 ms window, runs it through an LSTM to produce a spectral mask, applies the mask in the frequency domain (preserving phase), and reconstructs the time-domain signal via IFFT.
+
+2. **Model 2 — Time-domain refinement**: Refines the output of Model 1 with a second LSTM that operates directly on the waveform, capturing residual artifacts that spectral processing misses.
+
+The two models are chained: Model 1's output feeds Model 2. Both LSTMs are stateful — their hidden states persist across audio frames, giving the network temporal context across the full duration of a call.
+
+**Signal flow:**
+
+```
+Input frame (any sample rate, any channels)
+  → downsample to 16 kHz mono
+  → overlap-add loop (512-sample window, 128-sample shift)
+      → FFT → magnitude → Model 1 (spectral mask) → masked IFFT
+      → Model 2 (time-domain refinement)
+  → upsample back to original sample rate
+  → restore original channel count
+→ Denoised output frame
+```
+
+The overlap-add synthesis uses 75% overlap (512-sample window, 128-sample shift), identical to the original DTLN paper. This gives ~8 ms of algorithmic latency at 16 kHz.
+
+---
+
+## Performance
+
+Benchmarked on Apple M3 Pro, processing 16 kHz mono audio:
+
+| Metric | Value |
+|---|---|
+| Steady-state latency per block | ~0.7 ms |
+| Real-time factor | ~0.05× |
+| Headroom vs real-time | ~20× |
+| Cold-start (first inference) | ~500 ms (amortized by warmup in `__init__`) |
+
+The `__init__` method runs a dummy forward pass to trigger ONNX Runtime's JIT compilation before the first real audio frame arrives, eliminating the cold-start stall.
+
+---
+
+## Models
+
+Pretrained weights are the official DTLN models published by the original authors:
+
+| File | Source |
+|---|---|
+| `model_1.onnx` | [breizhn/DTLN · pretrained_model/](https://github.com/breizhn/DTLN/tree/master/pretrained_model) |
+| `model_2.onnx` | [breizhn/DTLN · pretrained_model/](https://github.com/breizhn/DTLN/tree/master/pretrained_model) |
+
+The models are not bundled in this repository (to keep it lightweight). They are downloaded automatically by `python agent.py download-files` or by calling `download_models()` directly.
+
+---
+
+## References
+
+- **Original DTLN paper**: [Westhausen & Meyer, "Noise Reduction with DTLN", Interspeech 2020](https://www.isca-archive.org/interspeech_2020/westhausen20_interspeech.html)
+- **Original DTLN implementation & pretrained models**: [github.com/breizhn/DTLN](https://github.com/breizhn/DTLN)
+- **DataDog engineering article** — the inspiration for this plugin: [Building a Real-Time Noise Suppression Library](https://www.datadoghq.com/blog/engineering/noise-suppression-library/)
+- **LiveKit noise cancellation overview**: [docs.livekit.io — Noise Cancellation](https://docs.livekit.io/transport/media/noise-cancellation/)
+- **LiveKit Agents SDK**: [github.com/livekit/agents](https://github.com/livekit/agents)
+- **ONNX Runtime**: [onnxruntime.ai](https://onnxruntime.ai)
+
+---
+
+## License
+
+The plugin code in this repository is released under the **MIT License**.
+
+The pretrained DTLN model weights are published by the original authors under the **MIT License** — see [breizhn/DTLN](https://github.com/breizhn/DTLN/blob/master/LICENSE).

livekit_plugins_dtln-0.1.0/pyproject.toml

@@ -0,0 +1,43 @@
+[build-system]
+requires = ["setuptools>=70", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "livekit-plugins-dtln"
+version = "0.1.0"
+description = "DTLN noise suppression plugin for LiveKit Agents — self-hosted, in-process, no cloud API"
+readme = "README.md"
+license = { file = "LICENSE" }
+authors = [{ name = "Aloware, Inc.", email = "dev@aloware.com" }]
+requires-python = ">=3.10"
+keywords = ["livekit", "noise-suppression", "noise-cancellation", "dtln", "onnx", "speech", "audio", "voip"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Multimedia :: Sound/Audio :: Speech",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = [
+    "livekit>=1.1.0",
+    "livekit-agents>=1.4.4",
+    "onnxruntime>=1.17.0",
+    "numpy>=1.26.0",
+]
+
+[project.urls]
+Homepage = "https://github.com/aloware/livekit-plugins-dtln"
+Repository = "https://github.com/aloware/livekit-plugins-dtln"
+"Bug Tracker" = "https://github.com/aloware/livekit-plugins-dtln/issues"
+Demo = "https://aloware.github.io/livekit-plugins-dtln/"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+"livekit.plugins.dtln" = ["models/*.onnx"]

livekit_plugins_dtln-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

livekit_plugins_dtln-0.1.0/src/livekit/plugins/dtln/__init__.py

@@ -0,0 +1,33 @@
+from livekit.agents import Plugin
+import logging
+
+from .noise_suppressor import DTLNNoiseSuppressor
+
+logger = logging.getLogger(__name__)
+
+
+class DTLNPlugin(Plugin):
+    def __init__(self):
+        super().__init__(
+            title="DTLN",
+            version="0.1.0",
+            package="livekit-plugins-dtln",
+            logger=logger,
+        )
+
+    def download_files(self):
+        from .noise_suppressor import download_models
+        download_models()
+
+
+def noise_suppression(**kwargs) -> DTLNNoiseSuppressor:
+    """Create a DTLNNoiseSuppressor instance.
+
+    Pass to AudioInputOptions(noise_cancellation=dtln.noise_suppression()).
+    """
+    return DTLNNoiseSuppressor(**kwargs)
+
+
+Plugin.register_plugin(DTLNPlugin())
+
+__all__ = ["DTLNNoiseSuppressor", "noise_suppression"]

livekit_plugins_dtln-0.1.0/src/livekit/plugins/dtln/noise_suppressor.py

@@ -0,0 +1,287 @@
+"""DTLN noise suppression as a LiveKit FrameProcessor.
+
+Implements the Dual-Signal Transformation LSTM Network (DTLN) from:
+  Noise Reduction with DTLN (Westhausen & Meyer, Interspeech 2020)
+  https://github.com/breizhn/DTLN
+
+Uses ONNX Runtime for in-process inference — no cloud API required.
+Drop-in replacement for livekit-plugins-noise-cancellation / ai-coustics.
+"""
+
+import logging
+import os
+import urllib.request
+
+import numpy as np
+import onnxruntime as ort
+from livekit import rtc
+
+logger = logging.getLogger(__name__)
+
+# DTLN requires 16 kHz mono audio (fixed by the pretrained model)
+_SAMPLE_RATE = 16_000
+# 32 ms window, 8 ms shift (75% overlap)
+_BLOCK_LEN = 512
+_BLOCK_SHIFT = 128
+# rfft of 512-sample block gives 257 unique frequency bins
+_N_BINS = _BLOCK_LEN // 2 + 1
+
+_DEFAULT_MODEL_DIR = os.path.join(os.path.dirname(__file__), "models")
+
+_DTLN_BASE_URL = "https://github.com/breizhn/DTLN/raw/master/pretrained_model"
+_MODEL_FILES = ["model_1.onnx", "model_2.onnx"]
+
+
+def download_models(models_dir: str = _DEFAULT_MODEL_DIR) -> None:
+    """Download pretrained DTLN ONNX models (skips files already present)."""
+    os.makedirs(models_dir, exist_ok=True)
+    for filename in _MODEL_FILES:
+        dest = os.path.join(models_dir, filename)
+        if os.path.exists(dest):
+            logger.info("Model already present: %s", dest)
+            continue
+        url = f"{_DTLN_BASE_URL}/{filename}"
+        logger.info("Downloading %s ...", url)
+        urllib.request.urlretrieve(url, dest)
+        logger.info("Saved to %s", dest)
+
+
+class DTLNNoiseSuppressor(rtc.FrameProcessor[rtc.AudioFrame]):
+    """In-process DTLN noise suppressor that works with self-hosted LiveKit.
+
+    Pass to AudioInputOptions(noise_cancellation=DTLNNoiseSuppressor(...)).
+    Each instance is stateful (LSTM state persists across frames) — create
+    one instance per call session.
+
+    Args:
+        model_1_path: Path to model_1.onnx (spectral masking stage).
+        model_2_path: Path to model_2.onnx (time-domain refinement stage).
+    """
+
+    def __init__(
+        self,
+        model_1_path: str = os.path.join(_DEFAULT_MODEL_DIR, "model_1.onnx"),
+        model_2_path: str = os.path.join(_DEFAULT_MODEL_DIR, "model_2.onnx"),
+    ) -> None:
+        self._sess1 = ort.InferenceSession(model_1_path)
+        self._sess2 = ort.InferenceSession(model_2_path)
+
+        # Input/output tensor names (read from model to avoid hardcoding)
+        self._m1_in_mag = self._sess1.get_inputs()[0].name
+        self._m1_in_state = self._sess1.get_inputs()[1].name
+        self._m2_in_time = self._sess2.get_inputs()[0].name
+        self._m2_in_state = self._sess2.get_inputs()[1].name
+
+        # LSTM states — shape read from model (e.g. [1, 2, 128, 2])
+        state1_shape = self._sess1.get_inputs()[1].shape
+        state2_shape = self._sess2.get_inputs()[1].shape
+        self._state1 = np.zeros(state1_shape, dtype=np.float32)
+        self._state2 = np.zeros(state2_shape, dtype=np.float32)
+
+        # Rolling 512-sample buffers at 16 kHz (float32)
+        self._in_buf = np.zeros(_BLOCK_LEN, dtype=np.float32)
+        self._out_buf = np.zeros(_BLOCK_LEN, dtype=np.float32)
+
+        # Queues for handling arbitrary incoming frame sizes
+        self._input_queue = np.zeros(0, dtype=np.float32)
+        self._output_queue = np.zeros(0, dtype=np.float32)
+
+        # Resamplers — created lazily on the first frame
+        self._downsampler: rtc.AudioResampler | None = None
+        self._upsampler: rtc.AudioResampler | None = None
+        self._native_rate: int = 0
+
+        self._enabled = True
+
+        # Pre-warm ONNX Runtime's JIT compiler so the first real frame
+        # doesn't stall the audio pipeline (~500ms cold-start otherwise).
+        self._warmup()
+
+    def _warmup(self) -> None:
+        dummy_mag = np.zeros((1, 1, _N_BINS), dtype=np.float32)
+        dummy_time = np.zeros((1, 1, _BLOCK_LEN), dtype=np.float32)
+        self._sess1.run(None, {self._m1_in_mag: dummy_mag, self._m1_in_state: self._state1})
+        self._sess2.run(None, {self._m2_in_time: dummy_time, self._m2_in_state: self._state2})
+        # Reset states — warmup outputs shouldn't carry over into real audio
+        state1_shape = self._sess1.get_inputs()[1].shape
+        state2_shape = self._sess2.get_inputs()[1].shape
+        self._state1 = np.zeros(state1_shape, dtype=np.float32)
+        self._state2 = np.zeros(state2_shape, dtype=np.float32)
+
+    # ------------------------------------------------------------------ #
+    # FrameProcessor interface                                             #
+    # ------------------------------------------------------------------ #
+
+    @property
+    def enabled(self) -> bool:
+        return self._enabled
+
+    @enabled.setter
+    def enabled(self, value: bool) -> None:
+        self._enabled = value
+
+    def _process(self, frame: rtc.AudioFrame) -> rtc.AudioFrame:
+        if not self._enabled:
+            return frame
+
+        # Lazily create resamplers when we learn the incoming sample rate.
+        if frame.sample_rate != self._native_rate:
+            self._native_rate = frame.sample_rate
+            if frame.sample_rate != _SAMPLE_RATE:
+                self._downsampler = rtc.AudioResampler(
+                    input_rate=frame.sample_rate,
+                    output_rate=_SAMPLE_RATE,
+                    num_channels=1,
+                    quality=rtc.AudioResamplerQuality.MEDIUM,
+                )
+                self._upsampler = rtc.AudioResampler(
+                    input_rate=_SAMPLE_RATE,
+                    output_rate=frame.sample_rate,
+                    num_channels=1,
+                    quality=rtc.AudioResamplerQuality.MEDIUM,
+                )
+            else:
+                self._downsampler = None
+                self._upsampler = None
+
+        # Convert int16 → float32 mono
+        samples = np.frombuffer(frame.data, dtype=np.int16).astype(np.float32) / 32768.0
+        if frame.num_channels > 1:
+            samples = samples.reshape(-1, frame.num_channels).mean(axis=1)
+
+        # Build a mono AudioFrame at the native rate for the resampler
+        mono_int16 = (np.clip(samples, -1.0, 1.0) * 32767.0).astype(np.int16)
+        mono_frame = rtc.AudioFrame(
+            data=mono_int16.tobytes(),
+            sample_rate=frame.sample_rate,
+            num_channels=1,
+            samples_per_channel=len(mono_int16),
+        )
+
+        # Downsample to 16 kHz
+        if self._downsampler is not None:
+            frames_16k = self._downsampler.push(mono_frame)
+        else:
+            frames_16k = [mono_frame]
+
+        if not frames_16k:
+            return frame  # resampler buffering startup, pass through
+
+        samples_16k = np.concatenate([
+            np.frombuffer(f.data, dtype=np.int16).astype(np.float32) / 32768.0
+            for f in frames_16k
+        ])
+
+        self._input_queue = np.concatenate([self._input_queue, samples_16k])
+
+        # Process in BLOCK_SHIFT (128-sample) steps; count steps taken.
+        n_steps = 0
+        while len(self._input_queue) >= _BLOCK_SHIFT:
+            new = self._input_queue[:_BLOCK_SHIFT]
+            self._input_queue = self._input_queue[_BLOCK_SHIFT:]
+
+            self._in_buf[:-_BLOCK_SHIFT] = self._in_buf[_BLOCK_SHIFT:]
+            self._in_buf[-_BLOCK_SHIFT:] = new
+
+            denoised = self._infer_block(self._in_buf)
+
+            self._out_buf[:-_BLOCK_SHIFT] = self._out_buf[_BLOCK_SHIFT:]
+            self._out_buf[-_BLOCK_SHIFT:] = 0.0
+            self._out_buf += denoised
+
+            self._output_queue = np.concatenate([
+                self._output_queue, self._out_buf[:_BLOCK_SHIFT]
+            ])
+            n_steps += 1
+
+        # Drain exactly what we produced this step — no more, no less.
+        # This eliminates the periodic silence that occurs when draining by
+        # n_16k (downsampler output) instead of n_steps * _BLOCK_SHIFT.
+        # During startup the output_queue fills with pipeline latency (~24ms);
+        # in steady state it stays constant.
+        n_produced = n_steps * _BLOCK_SHIFT
+        if n_produced == 0:
+            return frame
+
+        out_16k = self._output_queue[:n_produced]
+        self._output_queue = self._output_queue[n_produced:]
+
+        # Build 16 kHz AudioFrame and upsample back to native rate
+        out_int16_16k = (np.clip(out_16k, -1.0, 1.0) * 32767.0).astype(np.int16)
+        out_frame_16k = rtc.AudioFrame(
+            data=out_int16_16k.tobytes(),
+            sample_rate=_SAMPLE_RATE,
+            num_channels=1,
+            samples_per_channel=len(out_int16_16k),
+        )
+
+        if self._upsampler is not None:
+            out_frames = self._upsampler.push(out_frame_16k)
+        else:
+            out_frames = [out_frame_16k]
+
+        if not out_frames:
+            return frame
+
+        out_samples = np.concatenate([
+            np.frombuffer(f.data, dtype=np.int16).astype(np.float32) / 32768.0
+            for f in out_frames
+        ])
+
+        # Trim or pad to exactly match the input frame length
+        target = frame.samples_per_channel
+        if len(out_samples) > target:
+            out_samples = out_samples[:target]
+        elif len(out_samples) < target:
+            out_samples = np.pad(out_samples, (0, target - len(out_samples)))
+
+        # Restore original channel count (duplicate mono → stereo if needed)
+        if frame.num_channels > 1:
+            out_samples = np.repeat(out_samples, frame.num_channels)
+
+        out_int16 = (np.clip(out_samples, -1.0, 1.0) * 32767.0).astype(np.int16)
+        return rtc.AudioFrame(
+            data=out_int16.tobytes(),
+            sample_rate=frame.sample_rate,
+            num_channels=frame.num_channels,
+            samples_per_channel=frame.samples_per_channel,
+        )
+
+    def _close(self) -> None:
+        self._sess1 = None  # type: ignore[assignment]
+        self._sess2 = None  # type: ignore[assignment]
+
+    # ------------------------------------------------------------------ #
+    # Internal inference                                                   #
+    # ------------------------------------------------------------------ #
+
+    def _infer_block(self, block: np.ndarray) -> np.ndarray:
+        """Run one 512-sample block through both DTLN models.
+
+        Returns a 512-sample denoised block (float32, range ~[-1, 1]).
+        """
+        # --- Model 1: spectral masking ---
+        spec = np.fft.rfft(block)  # complex128, shape (257,)
+        mag = np.abs(spec).reshape(1, 1, _N_BINS).astype(np.float32)
+
+        out1 = self._sess1.run(None, {
+            self._m1_in_mag: mag,
+            self._m1_in_state: self._state1,
+        })
+        mask = out1[0].reshape(_N_BINS)   # (257,)
+        self._state1 = out1[1]
+
+        # Apply mask in frequency domain, reconstruct time domain
+        enhanced_spec = mask * spec       # element-wise, preserves phase
+        enhanced_time = np.fft.irfft(enhanced_spec, n=_BLOCK_LEN).astype(np.float32)
+
+        # --- Model 2: time-domain refinement ---
+        time_in = enhanced_time.reshape(1, 1, _BLOCK_LEN).astype(np.float32)
+        out2 = self._sess2.run(None, {
+            self._m2_in_time: time_in,
+            self._m2_in_state: self._state2,
+        })
+        denoised = out2[0].reshape(_BLOCK_LEN)  # (512,)
+        self._state2 = out2[1]
+
+        return denoised

livekit_plugins_dtln-0.1.0/src/livekit_plugins_dtln.egg-info/PKG-INFO

@@ -0,0 +1,228 @@
+Metadata-Version: 2.4
+Name: livekit-plugins-dtln
+Version: 0.1.0
+Summary: DTLN noise suppression plugin for LiveKit Agents — self-hosted, in-process, no cloud API
+Author-email: "Aloware, Inc." <dev@aloware.com>
+License: MIT License
+        
+        Copyright (c) 2024 Aloware, Inc.
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Project-URL: Homepage, https://github.com/aloware/livekit-plugins-dtln
+Project-URL: Repository, https://github.com/aloware/livekit-plugins-dtln
+Project-URL: Bug Tracker, https://github.com/aloware/livekit-plugins-dtln/issues
+Project-URL: Demo, https://aloware.github.io/livekit-plugins-dtln/
+Keywords: livekit,noise-suppression,noise-cancellation,dtln,onnx,speech,audio,voip
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Multimedia :: Sound/Audio :: Speech
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: livekit>=1.1.0
+Requires-Dist: livekit-agents>=1.4.4
+Requires-Dist: onnxruntime>=1.17.0
+Requires-Dist: numpy>=1.26.0
+Dynamic: license-file
+
+# livekit-plugins-dtln
+
+Python [LiveKit](https://livekit.io) plugin for **DTLN** (Dual-Signal Transformation LSTM Network) noise suppression — a fully self-hosted, open-source alternative to cloud-based noise cancellation services like Krisp or AI-coustics.
+
+Runs entirely **in-process** using [ONNX Runtime](https://onnxruntime.ai). No cloud API, no per-minute fees, no proprietary binaries. Works with **self-hosted LiveKit** servers.
+
+> Based on [Westhausen & Meyer, "Noise Reduction with DTLN", Interspeech 2020](https://www.isca-archive.org/interspeech_2020/westhausen20_interspeech.html)
+> Original implementation: [github.com/breizhn/DTLN](https://github.com/breizhn/DTLN)
+
+**[Live audio comparison demo →](https://aloware.github.io/livekit-plugins-dtln/)**
+
+---
+
+## Why DTLN?
+
+| | DTLN (this plugin) | Krisp / AI-coustics |
+|---|---|---|
+| **Hosting** | Self-hosted, in-process | Cloud API required |
+| **Cost** | Free (open weights) | Per-minute billing |
+| **LiveKit** | Works with self-hosted | Requires LiveKit Cloud |
+| **Latency** | ~8 ms (one block shift) | Network round-trip |
+| **Privacy** | Audio never leaves your server | Audio sent to third party |
+| **Real-time factor** | ~0.05× (20× faster than real-time) | Varies |
+
+---
+
+## Installation
+
+**pip:**
+
+```bash
+pip install livekit-plugins-dtln
+```
+
+**requirements.txt:**
+
+```
+livekit-plugins-dtln
+```
+
+**From source:**
+
+```bash
+git clone https://github.com/aloware/livekit-plugins-dtln.git
+pip install -e ./livekit-plugins-dtln
+```
+
+> The pretrained ONNX model weights (~4 MB) are bundled in the PyPI wheel — no separate download step needed.
+
+---
+
+## Usage
+
+### Session pipeline (recommended)
+
+```python
+from livekit.agents import room_io
+from livekit.plugins import dtln
+
+await session.start(
+    # ...,
+    room_options=room_io.RoomOptions(
+        audio_input=room_io.AudioInputOptions(
+            noise_cancellation=dtln.noise_suppression(),
+        ),
+    ),
+)
+```
+
+### Custom AudioStream
+
+```python
+from livekit import rtc
+from livekit.plugins import dtln
+
+stream = rtc.AudioStream.from_track(
+    track=track,
+    noise_cancellation=dtln.noise_suppression(),
+)
+```
+
+> **Note:** Create one `dtln.noise_suppression()` instance **per session**. Each instance holds stateful LSTM hidden states that must be scoped to a single call.
+
+> **Note:** DTLN is trained on raw microphone audio. Do not chain it with another noise cancellation model — applying two models in series produces unexpected results.
+
+### Custom model paths
+
+```python
+dtln.noise_suppression(
+    model_1_path="/path/to/model_1.onnx",
+    model_2_path="/path/to/model_2.onnx",
+)
+```
+
+---
+
+## Requirements
+
+- Python >= 3.10
+- livekit >= 1.1.0
+- livekit-agents >= 1.4.4
+- onnxruntime >= 1.17.0
+- numpy >= 1.26.0
+
+---
+
+## How It Works
+
+DTLN uses two sequential LSTM-based models:
+
+1. **Model 1 — Spectral masking**: Computes the magnitude spectrum of a 32 ms window, runs it through an LSTM to produce a spectral mask, applies the mask in the frequency domain (preserving phase), and reconstructs the time-domain signal via IFFT.
+
+2. **Model 2 — Time-domain refinement**: Refines the output of Model 1 with a second LSTM that operates directly on the waveform, capturing residual artifacts that spectral processing misses.
+
+The two models are chained: Model 1's output feeds Model 2. Both LSTMs are stateful — their hidden states persist across audio frames, giving the network temporal context across the full duration of a call.
+
+**Signal flow:**
+
+```
+Input frame (any sample rate, any channels)
+  → downsample to 16 kHz mono
+  → overlap-add loop (512-sample window, 128-sample shift)
+      → FFT → magnitude → Model 1 (spectral mask) → masked IFFT
+      → Model 2 (time-domain refinement)
+  → upsample back to original sample rate
+  → restore original channel count
+→ Denoised output frame
+```
+
+The overlap-add synthesis uses 75% overlap (512-sample window, 128-sample shift), identical to the original DTLN paper. This gives ~8 ms of algorithmic latency at 16 kHz.
+
+---
+
+## Performance
+
+Benchmarked on Apple M3 Pro, processing 16 kHz mono audio:
+
+| Metric | Value |
+|---|---|
+| Steady-state latency per block | ~0.7 ms |
+| Real-time factor | ~0.05× |
+| Headroom vs real-time | ~20× |
+| Cold-start (first inference) | ~500 ms (amortized by warmup in `__init__`) |
+
+The `__init__` method runs a dummy forward pass to trigger ONNX Runtime's JIT compilation before the first real audio frame arrives, eliminating the cold-start stall.
+
+---
+
+## Models
+
+Pretrained weights are the official DTLN models published by the original authors:
+
+| File | Source |
+|---|---|
+| `model_1.onnx` | [breizhn/DTLN · pretrained_model/](https://github.com/breizhn/DTLN/tree/master/pretrained_model) |
+| `model_2.onnx` | [breizhn/DTLN · pretrained_model/](https://github.com/breizhn/DTLN/tree/master/pretrained_model) |
+
+The models are not bundled in this repository (to keep it lightweight). They are downloaded automatically by `python agent.py download-files` or by calling `download_models()` directly.
+
+---
+
+## References
+
+- **Original DTLN paper**: [Westhausen & Meyer, "Noise Reduction with DTLN", Interspeech 2020](https://www.isca-archive.org/interspeech_2020/westhausen20_interspeech.html)
+- **Original DTLN implementation & pretrained models**: [github.com/breizhn/DTLN](https://github.com/breizhn/DTLN)
+- **DataDog engineering article** — the inspiration for this plugin: [Building a Real-Time Noise Suppression Library](https://www.datadoghq.com/blog/engineering/noise-suppression-library/)
+- **LiveKit noise cancellation overview**: [docs.livekit.io — Noise Cancellation](https://docs.livekit.io/transport/media/noise-cancellation/)
+- **LiveKit Agents SDK**: [github.com/livekit/agents](https://github.com/livekit/agents)
+- **ONNX Runtime**: [onnxruntime.ai](https://onnxruntime.ai)
+
+---
+
+## License
+
+The plugin code in this repository is released under the **MIT License**.
+
+The pretrained DTLN model weights are published by the original authors under the **MIT License** — see [breizhn/DTLN](https://github.com/breizhn/DTLN/blob/master/LICENSE).

livekit_plugins_dtln-0.1.0/src/livekit_plugins_dtln.egg-info/SOURCES.txt

@@ -0,0 +1,12 @@
+LICENSE
+README.md
+pyproject.toml
+src/livekit/plugins/dtln/__init__.py
+src/livekit/plugins/dtln/noise_suppressor.py
+src/livekit/plugins/dtln/models/model_1.onnx
+src/livekit/plugins/dtln/models/model_2.onnx
+src/livekit_plugins_dtln.egg-info/PKG-INFO
+src/livekit_plugins_dtln.egg-info/SOURCES.txt
+src/livekit_plugins_dtln.egg-info/dependency_links.txt
+src/livekit_plugins_dtln.egg-info/requires.txt
+src/livekit_plugins_dtln.egg-info/top_level.txt

livekit_plugins_dtln-0.1.0/src/livekit_plugins_dtln.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

livekit_plugins_dtln-0.1.0/src/livekit_plugins_dtln.egg-info/requires.txt

@@ -0,0 +1,4 @@
+livekit>=1.1.0
+livekit-agents>=1.4.4
+onnxruntime>=1.17.0
+numpy>=1.26.0

livekit_plugins_dtln-0.1.0/src/livekit_plugins_dtln.egg-info/top_level.txt

@@ -0,0 +1 @@
+livekit