senko 0.1.0rc1__tar.gz

senko-0.1.0rc1/.github/FUNDING.yml

@@ -0,0 +1,15 @@
+# These are supported funding model platforms
+
+github: [hamzaq2000] # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2]
+patreon: # Replace with a single Patreon username
+open_collective: # Replace with a single Open Collective username
+ko_fi: # Replace with a single Ko-fi username
+tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
+community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
+liberapay: # Replace with a single Liberapay username
+issuehunt: # Replace with a single IssueHunt username
+lfx_crowdfunding: # Replace with a single LFX Crowdfunding project-name e.g., cloud-foundry
+polar: # Replace with a single Polar username
+buy_me_a_coffee: # Replace with a single Buy Me a Coffee username
+thanks_dev: # Replace with a single thanks.dev username
+custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']

senko-0.1.0rc1/.gitignore

@@ -0,0 +1,16 @@
+**/__pycache__/
+**/.DS_Store
+
+.vscode/
+
+dist/
+build/
+env/
+.venv/
+
+result/
+
+*.wav
+*.m4a
+*.webm
+*.mp3

senko-0.1.0rc1/.python-version

@@ -0,0 +1 @@
+3.11.13

senko-0.1.0rc1/BENCHMARK.md

@@ -0,0 +1,10 @@
+# Benchmarking
+To get the figures listed in the README for time taken to diarize 1 hour of audio on an RTX 4090 + Ryzen 9 7950X machine and an M3 MacBook Air, as well as the CPU performance figure in the FAQ section using a Ryzen 9 9950X, the audio from [this](https://www.youtube.com/watch?v=GT_sXIUJPUo) video was used. It was downloaded and decompressed into a 16 KHz mono wav file in the following manner.
+```
+yt-dlp -f bestaudio GT_sXIUJPUo -o "temp_GT_sXIUJPUo.%(ext)s"
+ffmpeg -i "temp_GT_sXIUJPUo."* -acodec pcm_s16le -ac 1 -ar 16000 cowen.wav
+rm "temp_GT_sXIUJPUo."*
+```
+Then the `examples/diarize.py` script was run, and `cowen.wav` was entered after the warmup.
+
+For the sake of reproducibility, the exact `cowen.wav` file used can be downloaded from [here](https://www.dropbox.com/scl/fi/77kgl6luhmsm6k30x1muf/cowen.wav?rlkey=n6goatgi3pjpgn7glna654f2a&dl=1).

senko-0.1.0rc1/CMakeLists.txt

@@ -0,0 +1,18 @@
+cmake_minimum_required(VERSION 3.18)
+project(senko LANGUAGES CXX)
+
+# Set C++ standard
+set(CMAKE_CXX_STANDARD 20)
+set(CMAKE_CXX_STANDARD_REQUIRED ON)
+
+# Find Python
+find_package(Python COMPONENTS Interpreter Development.Module REQUIRED)
+
+# Add the fbank_extractor subdirectory. This is where the C++ library is defined, built, and its installation is configured
+add_subdirectory(senko/fbank_extractor)
+
+# Only install in wheel builds (skip editable and sdist)
+if(SKBUILD_STATE STREQUAL "wheel")
+    install(TARGETS fbank_extractor DESTINATION senko)
+    install(DIRECTORY models DESTINATION senko)
+endif()

senko-0.1.0rc1/DEV_SETUP.md

@@ -0,0 +1,46 @@
+# Senko Development Setup Guide
+## Pre-requisites
+- `clang` if on Linux or WSL
+    - `sudo apt update; sudo apt install clang`
+- `git`
+    - `sudo apt update; sudo apt install git` on Linux/WSL
+    - On macOS, should already have it if you have the Xcode Command Line Tools installed
+- [`uv`](https://docs.astral.sh/uv/#installation)
+
+## Development Setup Steps
+First create and activate a Python virtual environment
+```
+uv venv --python 3.11.13 .venv
+source .venv/bin/activate
+```
+Clone the Senko repository
+```
+git clone https://github.com/narcotic-sh/senko.git
+```
+Then install using editable mode (`-e`)
+```bash
+# For NVIDIA GPUs with CUDA compute capability >= 7.5 (~GTX 16 series and newer)
+uv pip install -e "/path/to/cloned/senko[nvidia]"
+
+# For NVIDIA GPUs with CUDA compute capability < 7.5 (~GTX 10 series and older)
+uv pip install -e "/path/to/cloned/senko[nvidia-old]"
+
+# For Macs (mps) and cpu execution on all other platforms
+uv pip install -e "/path/to/cloned/senko"
+```
+For NVIDIA, make sure the installed driver is CUDA 12 capable (should see `CUDA Version: 12.x` in the top right of `nvidia-smi`)
+
+Now you can modify the Senko code in the cloned repository folder; changes will be reflected immediately.
+
+Use Senko like normal in scripts:
+```python
+from senko import Diarizer
+import json
+
+diarizer = Diarizer(torch_device='auto', warmup=True, quiet=False)
+result = diarizer.diarize('audio.wav', generate_colors=False) # 16 KHz mono wav
+
+with open('./audio_diarized.json', 'w') as f:
+    json.dump(result["merged_segments"], f, indent=2)
+```
+Also see `examples/diarize.py`.

senko-0.1.0rc1/DOCS.md

@@ -0,0 +1,97 @@
+# Senko Documentation
+
+### `Diarizer`
+```python
+from senko import Diarizer
+diarizer = Diarizer(torch_device='auto', warmup=True, quiet=True)
+```
+- `torch_device`: Device to use for PyTorch operations (`auto`, `cuda`, `mps`, `cpu`)
+    - `auto` automatically selects `cuda` if available, if not, then `mps` (Apple Silicon), if not, then `cpu`
+- `warmup`: Warm up CAM++ embeddings model and clustering objects during initialization
+    - If warmup is not done, the first few runs of the pipeline will be a bit slower
+- `quiet`: Suppress progress updates and all other output to stdout
+
+### `diarize()`
+```python
+result_data = diarizer.diarize(wav_path='audio.wav', generate_colors=False)
+```
+#### Parameters
+- `wav_path`: Path to the audio file (16 KHz mono WAV format)
+- `generate_colors`: Whether to generate speaker color sets for visualization
+
+#### Returns
+Dictionary (`result_data`) containing keys:
+- `raw_segments`: Raw diarization output
+    - A list of speaking segments (dictionaries) with keys `start`, `end`, `speaker`
+- `merged_segments`: Cleaned diarization output
+    - Same format as `raw_segments`
+    - Segments <= 0.78 seconds in length are removed
+    - Adjacent segments of the same speaker that have a silence in between them of <= 4 seconds are merged into one segment
+- `speaker_centroids`: Voice fingerprints for each detected speaker
+    - Dictionary mapping speaker IDs to 192-dimensional numpy arrays
+    - Each centroid is the mean of all audio embeddings for that speaker
+    - Can be used for speaker comparison/identification across different audio files
+- `timing_stats`: Dictionary of how long each stage of the pipeline took in seconds, as well as the total time
+    - Keys: `total_time`, `vad_time`, `fbank_time`, `embeddings_time`, `clustering_time`
+- `speaker_color_sets`: 10 sets of speaker colors (if requested)
+
+### `speaker_similarity()`
+```python
+from senko import speaker_similarity
+if speaker_similarity(centroid1, centroid2) >= 0.875:
+    print('Speakers are the same')
+```
+Calculate cosine similarity between two speaker centroids (voice fingerprints).
+#### Parameters
+- `centroid1`: First speaker centroid (192-dimensional numpy array)
+- `centroid2`: Second speaker centroid (192-dimensional numpy array)
+
+#### Returns
+- `float`: Cosine similarity score between -1 and 1 (<1 rarely if ever happens with speaker embeddings)
+
+### Output Format
+Speaker segments (`raw_segments`/`merged_segments`):
+```
+[
+  {
+    "start": 0.0,
+    "end": 5.2,
+    "speaker": "SPEAKER_01"
+  },
+  {
+    "start": 5.2,
+    "end": 10.8,
+    "speaker": "SPEAKER_02"
+  },
+  ...
+]
+```
+Speaker centroids (`speaker_centroids`):
+```
+{
+  "SPEAKER_01": array([0.123, -0.456, 0.789, ...]),  # 192-dimensional numpy array
+  "SPEAKER_02": array([-0.234, 0.567, -0.890, ...]), # 192-dimensional numpy array
+  ...
+}
+```
+Color sets (`speaker_color_sets`):
+```
+{
+    "0": {
+      "SPEAKER_01": "#ea759c",
+      "SPEAKER_02": "#579c3a",
+      "SPEAKER_03": "#100058",
+    },
+    "1": {
+      "SPEAKER_01": "#97de7b",
+      "SPEAKER_02": "#4c56b6",
+      "SPEAKER_03": "#480000",
+    },
+    "2": {
+      "SPEAKER_01": "#8393f9",
+      "SPEAKER_02": "#bf5d01",
+      "SPEAKER_03": "#003a38",
+    },
+    ...
+}
+```

senko-0.1.0rc1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Narcotic Software
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

senko-0.1.0rc1/PKG-INFO

@@ -0,0 +1,162 @@
+Metadata-Version: 2.4
+Name: senko
+Version: 0.1.0rc1
+Summary: Very fast speaker diarization
+Keywords: speaker-diarization,audio-processing,speech-processing,voice-activity-detection,speaker-verification,speech-analysis,cuda,gpu,machine-learning,audio-ai,pyannote,silero-vad,speaker-embeddings
+Author-Email: Hamza Qayyum <mhamzaqayyum@icloud.com>
+License-Expression: MIT
+License-File: LICENSE
+License-File: THIRD_PARTY_LICENSES
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Intended Audience :: Education
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: C++
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Topic :: Multimedia :: Sound/Audio :: Analysis
+Classifier: Topic :: Multimedia :: Sound/Audio :: Speech
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Environment :: GPU :: NVIDIA CUDA
+Project-URL: Homepage, https://github.com/narcotic-sh/senko
+Project-URL: Repository, https://github.com/narcotic-sh/senko
+Project-URL: Issues, https://github.com/narcotic-sh/senko/issues
+Requires-Python: <3.12,>=3.9
+Requires-Dist: numpy
+Requires-Dist: scikit-learn
+Requires-Dist: umap-learn
+Requires-Dist: hdbscan
+Requires-Dist: numba
+Requires-Dist: llvmlite
+Requires-Dist: pyyaml
+Requires-Dist: soundfile
+Requires-Dist: torch
+Requires-Dist: silero-vad
+Requires-Dist: termcolor
+Requires-Dist: psutil
+Requires-Dist: colour-science
+Provides-Extra: nvidia-old
+Requires-Dist: torchaudio; extra == "nvidia-old"
+Requires-Dist: pyannote.audio; extra == "nvidia-old"
+Provides-Extra: nvidia
+Requires-Dist: torchaudio; extra == "nvidia"
+Requires-Dist: pyannote.audio; extra == "nvidia"
+Requires-Dist: cudf-cu12==25.8.*; extra == "nvidia"
+Requires-Dist: dask-cudf-cu12==25.8.*; extra == "nvidia"
+Requires-Dist: cuml-cu12==25.8.*; extra == "nvidia"
+Requires-Dist: cugraph-cu12==25.8.*; extra == "nvidia"
+Requires-Dist: nx-cugraph-cu12==25.8.*; extra == "nvidia"
+Requires-Dist: cuxfilter-cu12==25.8.*; extra == "nvidia"
+Requires-Dist: cucim-cu12==25.8.*; extra == "nvidia"
+Requires-Dist: pylibraft-cu12==25.8.*; extra == "nvidia"
+Requires-Dist: raft-dask-cu12==25.8.*; extra == "nvidia"
+Requires-Dist: cuvs-cu12==25.8.*; extra == "nvidia"
+Description-Content-Type: text/markdown
+
+# Senko
+> 閃光 (senkō) - a flash of light
+
+A very fast speaker diarization pipeline.
+
+1 hour of audio processed in 5 seconds (RTX 4090, Ryzen 9 7950X). ~17x faster than [Pyannote 3.1](https://huggingface.co/pyannote/speaker-diarization-3.1).
+
+On M3 Macbook Air, 1 hour in 23.5 seconds (~14x faster).
+
+This pipeline is used in the [Zanshin](https://github.com/narcotic-sh/zanshin) media player.
+
+## Usage
+```python
+from senko import Diarizer
+import json
+
+diarizer = Diarizer(torch_device='auto', warmup=True, quiet=False)
+result = diarizer.diarize('audio.wav', generate_colors=False) # 16 KHz mono wav
+
+with open('./audio_diarized.json', 'w') as f:
+    json.dump(result["merged_segments"], f, indent=2)
+```
+See `examples/diarize.py` and `DOCS.md`.
+
+## Installation
+Senko has been tested to work on Linux, macOS, and WSL, with Python version `3.11.13`.
+
+The official install method is using [uv](https://docs.astral.sh/uv/#installation). Have `clang` installed as well if on Linux/WSL.
+```bash
+# For NVIDIA GPUs with CUDA compute capability >= 7.5 (~GTX 16 series and newer)
+uv pip install "senko[nvidia]"
+
+# For NVIDIA GPUs with CUDA compute capability < 7.5 (~GTX 10 series and older)
+uv pip install "senko[nvidia-old]"
+
+# For Macs (mps) and cpu execution on all other platforms
+uv pip install senko
+```
+For NVIDIA, make sure the installed driver is CUDA 12 capable (should see `CUDA Version: 12.x` in `nvidia-smi`).
+
+For setting up Senko for development, see `DEV_SETUP.md`.
+
+## Technical Details
+Senko is a modified version of the speaker diarization pipeline found in the [3D-Speaker](https://github.com/modelscope/3D-Speaker/tree/main/egs/3dspeaker/speaker-diarization) project.
+It consists of four stages: VAD (voice activity detection), Fbank feature extraction, speaker embeddings generation, and clustering (spectral or UMAP+HDBSCAN).
+
+The following modifications have been made:
+- VAD model has been swapped from FSMN-VAD to either Pyannote [segmentation-3.0](https://huggingface.co/pyannote/segmentation-3.0) (when running on NVIDIA) or [Silero VAD](https://github.com/snakers4/silero-vad) (non-NVIDIA)
+- Fbank feature extraction is done fully upfront, in C++, using all available CPU cores
+- Batched inference of CAM++ embeddings model
+- Clustering when on NVIDIA (with a GPU of CUDA compute capability 7.0+) is done through [RAPIDS](https://docs.rapids.ai/api/cuml/stable/zero-code-change/)
+
+## FAQ
+<details>
+<summary>Is there any way to visualize the output diarization data?</summary>
+<br>
+Absolutely. The <a href="https://github.com/narcotic-sh/zanshin">Zanshin</a> media player is entirely made for this purpose. Zanshin is powered by Senko, so the easiest way to visualize the diarization data is by simply using it. It's currently available for Mac (Apple Silicon) with packaging. It also works on Windows and Linux, but without packaging (coming soon); you'll need to clone the repo and launch it through the terminal. See <a href="https://github.com/narcotic-sh/zanshin/blob/main/DEV_SETUP.md">here</a> for instructions.
+<br>
+<br>
+You can also load in the diarization data that Senko generates manually into Zanshin if you want. First, turn off diarization in Zanshin by going into Settings and turning off <code>Identify Speakers</code>. Then, after you add a media item, click on it and on the player page press the <code>H</code> key. In the textbox that appears, paste the contents of the output JSON file that <code>examples/diarize.py</code> generates.
+</details>
+<details>
+<summary>Are overlapping speaker segments detected correctly?</summary>
+<br>
+The current output will not have any overlapping speaker segments; i.e. only one speaker max is reported to be speaking at any given time. However, despite this, the current pipeline still performs great in determining who the dominant speaker is at any given time in chaotic audio with speakers talking over each other (example: casual podcasts). That said, detecting overlapping speaker segments is a planned feature thanks to the Pyannote segmentation-3.0 model (which we currently only use for VAD) supporting it.
+</details>
+<details>
+<summary>How fast is the pipeline on CPU (<code>cpu</code>)?</summary>
+<br>
+On a Ryzen 9 9950X it takes 42 seconds to process 1 hour of audio.
+</details>
+<details>
+<summary>Does the entire pipeline run fully on the GPU, if available?</summary>
+<br>
+With <code>cuda</code>, all parts of the pipeline except Fbank feature extraction (which always runs on the CPU) do run on the GPU. However, CPU performance still significantly impacts overall speed even for the GPU-accelerated stages.
+<br><br>
+During the embeddings generation phase, for example, while the actual model inference happens on the GPU with minimal CPU-GPU memory transfers (just input/output), the CPU handles all the orchestration work: Python loops for batching, tensor preparation, padding operations dispatch, and managing the inference pipeline. All this orchestration runs single-threaded on the CPU. This means a faster CPU will improve performance even when using a powerful GPU, as the CPU coordinates all the GPU operations.
+<br><br>
+Therefore, for optimal performance, pair a fast GPU with a fast CPU. The CPU bottleneck becomes more noticeable with very fast GPUs (ex. RTX 4090) where the GPU can execute the batch preparation and inference faster than the CPU can orchestrate/dispatch these operations.
+<br><br>
+As for <code>mps</code>, the only part of the pipeline that runs on the GPU is the embeddings gen phase. All other parts run on the CPU.
+</details>
+<details>
+<summary>Known limitations?</summary>
+<br>
+- The pipeline works best when the audio recording quality is good. Ideal setting: professional podcast studio. Heavy background noise, background music, or a generally low fidelity recording will degrade the diarization performance significantly.
+<br><br>
+- It is rare but possible that voices that sound very similar get clustered as one voice. This can happen if the voices are genuinely extremely similar, or, more commonly, if the audio recording fidelity is low.
+<br><br>
+- The same voice but recorded with >1 microphones or in >1 recording settings (within the same audio file) will often get detected as >1 speakers.
+<br><br>
+- Diarization performance (as in quality, not just speed) on NVIDIA is a bit better than on Mac and CPU. This is due to RAPIDS clustering yielding slightly better results than CPU clustering.
+</details>
+
+## Future Improvements & Directions
+- Overlapping speaker segments support
+- Improve speaker colors generation algorithm
+- Support for Intel and AMD GPUs
+- Experiment with `torch.compile()` (faster than current JIT tracing TorchScript approach?)
+- Experiment with Modular [MAX](https://www.modular.com/blog/bring-your-own-pytorch-model) engine (faster CPU inference speed?)
+- Measure DER (diarization error rate) of pipeline
+- Experiment with [Resonate](https://alexandrefrancois.org/Resonate/) for superior audio feature extraction
+- Background noise removal ([DeepFilterNet](https://github.com/Rikorose/DeepFilterNet))
+- Live progress reporting

senko-0.1.0rc1/README.md

@@ -0,0 +1,104 @@
+# Senko
+> 閃光 (senkō) - a flash of light
+
+A very fast speaker diarization pipeline.
+
+1 hour of audio processed in 5 seconds (RTX 4090, Ryzen 9 7950X). ~17x faster than [Pyannote 3.1](https://huggingface.co/pyannote/speaker-diarization-3.1).
+
+On M3 Macbook Air, 1 hour in 23.5 seconds (~14x faster).
+
+This pipeline is used in the [Zanshin](https://github.com/narcotic-sh/zanshin) media player.
+
+## Usage
+```python
+from senko import Diarizer
+import json
+
+diarizer = Diarizer(torch_device='auto', warmup=True, quiet=False)
+result = diarizer.diarize('audio.wav', generate_colors=False) # 16 KHz mono wav
+
+with open('./audio_diarized.json', 'w') as f:
+    json.dump(result["merged_segments"], f, indent=2)
+```
+See `examples/diarize.py` and `DOCS.md`.
+
+## Installation
+Senko has been tested to work on Linux, macOS, and WSL, with Python version `3.11.13`.
+
+The official install method is using [uv](https://docs.astral.sh/uv/#installation). Have `clang` installed as well if on Linux/WSL.
+```bash
+# For NVIDIA GPUs with CUDA compute capability >= 7.5 (~GTX 16 series and newer)
+uv pip install "senko[nvidia]"
+
+# For NVIDIA GPUs with CUDA compute capability < 7.5 (~GTX 10 series and older)
+uv pip install "senko[nvidia-old]"
+
+# For Macs (mps) and cpu execution on all other platforms
+uv pip install senko
+```
+For NVIDIA, make sure the installed driver is CUDA 12 capable (should see `CUDA Version: 12.x` in `nvidia-smi`).
+
+For setting up Senko for development, see `DEV_SETUP.md`.
+
+## Technical Details
+Senko is a modified version of the speaker diarization pipeline found in the [3D-Speaker](https://github.com/modelscope/3D-Speaker/tree/main/egs/3dspeaker/speaker-diarization) project.
+It consists of four stages: VAD (voice activity detection), Fbank feature extraction, speaker embeddings generation, and clustering (spectral or UMAP+HDBSCAN).
+
+The following modifications have been made:
+- VAD model has been swapped from FSMN-VAD to either Pyannote [segmentation-3.0](https://huggingface.co/pyannote/segmentation-3.0) (when running on NVIDIA) or [Silero VAD](https://github.com/snakers4/silero-vad) (non-NVIDIA)
+- Fbank feature extraction is done fully upfront, in C++, using all available CPU cores
+- Batched inference of CAM++ embeddings model
+- Clustering when on NVIDIA (with a GPU of CUDA compute capability 7.0+) is done through [RAPIDS](https://docs.rapids.ai/api/cuml/stable/zero-code-change/)
+
+## FAQ
+<details>
+<summary>Is there any way to visualize the output diarization data?</summary>
+<br>
+Absolutely. The <a href="https://github.com/narcotic-sh/zanshin">Zanshin</a> media player is entirely made for this purpose. Zanshin is powered by Senko, so the easiest way to visualize the diarization data is by simply using it. It's currently available for Mac (Apple Silicon) with packaging. It also works on Windows and Linux, but without packaging (coming soon); you'll need to clone the repo and launch it through the terminal. See <a href="https://github.com/narcotic-sh/zanshin/blob/main/DEV_SETUP.md">here</a> for instructions.
+<br>
+<br>
+You can also load in the diarization data that Senko generates manually into Zanshin if you want. First, turn off diarization in Zanshin by going into Settings and turning off <code>Identify Speakers</code>. Then, after you add a media item, click on it and on the player page press the <code>H</code> key. In the textbox that appears, paste the contents of the output JSON file that <code>examples/diarize.py</code> generates.
+</details>
+<details>
+<summary>Are overlapping speaker segments detected correctly?</summary>
+<br>
+The current output will not have any overlapping speaker segments; i.e. only one speaker max is reported to be speaking at any given time. However, despite this, the current pipeline still performs great in determining who the dominant speaker is at any given time in chaotic audio with speakers talking over each other (example: casual podcasts). That said, detecting overlapping speaker segments is a planned feature thanks to the Pyannote segmentation-3.0 model (which we currently only use for VAD) supporting it.
+</details>
+<details>
+<summary>How fast is the pipeline on CPU (<code>cpu</code>)?</summary>
+<br>
+On a Ryzen 9 9950X it takes 42 seconds to process 1 hour of audio.
+</details>
+<details>
+<summary>Does the entire pipeline run fully on the GPU, if available?</summary>
+<br>
+With <code>cuda</code>, all parts of the pipeline except Fbank feature extraction (which always runs on the CPU) do run on the GPU. However, CPU performance still significantly impacts overall speed even for the GPU-accelerated stages.
+<br><br>
+During the embeddings generation phase, for example, while the actual model inference happens on the GPU with minimal CPU-GPU memory transfers (just input/output), the CPU handles all the orchestration work: Python loops for batching, tensor preparation, padding operations dispatch, and managing the inference pipeline. All this orchestration runs single-threaded on the CPU. This means a faster CPU will improve performance even when using a powerful GPU, as the CPU coordinates all the GPU operations.
+<br><br>
+Therefore, for optimal performance, pair a fast GPU with a fast CPU. The CPU bottleneck becomes more noticeable with very fast GPUs (ex. RTX 4090) where the GPU can execute the batch preparation and inference faster than the CPU can orchestrate/dispatch these operations.
+<br><br>
+As for <code>mps</code>, the only part of the pipeline that runs on the GPU is the embeddings gen phase. All other parts run on the CPU.
+</details>
+<details>
+<summary>Known limitations?</summary>
+<br>
+- The pipeline works best when the audio recording quality is good. Ideal setting: professional podcast studio. Heavy background noise, background music, or a generally low fidelity recording will degrade the diarization performance significantly.
+<br><br>
+- It is rare but possible that voices that sound very similar get clustered as one voice. This can happen if the voices are genuinely extremely similar, or, more commonly, if the audio recording fidelity is low.
+<br><br>
+- The same voice but recorded with >1 microphones or in >1 recording settings (within the same audio file) will often get detected as >1 speakers.
+<br><br>
+- Diarization performance (as in quality, not just speed) on NVIDIA is a bit better than on Mac and CPU. This is due to RAPIDS clustering yielding slightly better results than CPU clustering.
+</details>
+
+## Future Improvements & Directions
+- Overlapping speaker segments support
+- Improve speaker colors generation algorithm
+- Support for Intel and AMD GPUs
+- Experiment with `torch.compile()` (faster than current JIT tracing TorchScript approach?)
+- Experiment with Modular [MAX](https://www.modular.com/blog/bring-your-own-pytorch-model) engine (faster CPU inference speed?)
+- Measure DER (diarization error rate) of pipeline
+- Experiment with [Resonate](https://alexandrefrancois.org/Resonate/) for superior audio feature extraction
+- Background noise removal ([DeepFilterNet](https://github.com/Rikorose/DeepFilterNet))
+- Live progress reporting