vocal-helper 0.4.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- vocal_helper-0.4.1/LICENSE +29 -0
- vocal_helper-0.4.1/PKG-INFO +374 -0
- vocal_helper-0.4.1/README.md +307 -0
- vocal_helper-0.4.1/pyproject.toml +199 -0
- vocal_helper-0.4.1/setup.cfg +4 -0
- vocal_helper-0.4.1/tests/test_api.py +65 -0
- vocal_helper-0.4.1/tests/test_asr.py +191 -0
- vocal_helper-0.4.1/tests/test_cli.py +270 -0
- vocal_helper-0.4.1/tests/test_diar_engines.py +72 -0
- vocal_helper-0.4.1/tests/test_integration_youtube.py +403 -0
- vocal_helper-0.4.1/tests/test_lid.py +164 -0
- vocal_helper-0.4.1/tests/test_llm.py +103 -0
- vocal_helper-0.4.1/tests/test_mcp.py +44 -0
- vocal_helper-0.4.1/tests/test_offline_regression.py +246 -0
- vocal_helper-0.4.1/tests/test_pipeline.py +193 -0
- vocal_helper-0.4.1/tests/test_pipeline_observability.py +152 -0
- vocal_helper-0.4.1/tests/test_ported_patterns.py +168 -0
- vocal_helper-0.4.1/tests/test_public_api_stability.py +146 -0
- vocal_helper-0.4.1/tests/test_settings.py +90 -0
- vocal_helper-0.4.1/tests/test_smoke.py +52 -0
- vocal_helper-0.4.1/tests/test_sources.py +180 -0
- vocal_helper-0.4.1/vocal_helper/__init__.py +169 -0
- vocal_helper-0.4.1/vocal_helper/_settings.py +234 -0
- vocal_helper-0.4.1/vocal_helper/api.py +279 -0
- vocal_helper-0.4.1/vocal_helper/asr.py +504 -0
- vocal_helper-0.4.1/vocal_helper/cli.py +42 -0
- vocal_helper-0.4.1/vocal_helper/cli_argparse.py +442 -0
- vocal_helper-0.4.1/vocal_helper/cli_click.py +466 -0
- vocal_helper-0.4.1/vocal_helper/diar.py +1204 -0
- vocal_helper-0.4.1/vocal_helper/eot.py +292 -0
- vocal_helper-0.4.1/vocal_helper/eot_bench.py +201 -0
- vocal_helper-0.4.1/vocal_helper/lid.py +549 -0
- vocal_helper-0.4.1/vocal_helper/llm.py +382 -0
- vocal_helper-0.4.1/vocal_helper/mcp.py +87 -0
- vocal_helper-0.4.1/vocal_helper/parallel_pipelines.py +170 -0
- vocal_helper-0.4.1/vocal_helper/pipeline.py +698 -0
- vocal_helper-0.4.1/vocal_helper/py.typed +0 -0
- vocal_helper-0.4.1/vocal_helper/sources.py +367 -0
- vocal_helper-0.4.1/vocal_helper/types.py +108 -0
- vocal_helper-0.4.1/vocal_helper/vad.py +271 -0
- vocal_helper-0.4.1/vocal_helper.egg-info/PKG-INFO +374 -0
- vocal_helper-0.4.1/vocal_helper.egg-info/SOURCES.txt +44 -0
- vocal_helper-0.4.1/vocal_helper.egg-info/dependency_links.txt +1 -0
- vocal_helper-0.4.1/vocal_helper.egg-info/entry_points.txt +4 -0
- vocal_helper-0.4.1/vocal_helper.egg-info/requires.txt +48 -0
- vocal_helper-0.4.1/vocal_helper.egg-info/top_level.txt +1 -0
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
BSD 3-Clause License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026, Warith HARCHAOUI
|
|
4
|
+
All rights reserved.
|
|
5
|
+
|
|
6
|
+
Redistribution and use in source and binary forms, with or without
|
|
7
|
+
modification, are permitted provided that the following conditions are met:
|
|
8
|
+
|
|
9
|
+
1. Redistributions of source code must retain the above copyright notice, this
|
|
10
|
+
list of conditions and the following disclaimer.
|
|
11
|
+
|
|
12
|
+
2. Redistributions in binary form must reproduce the above copyright notice,
|
|
13
|
+
this list of conditions and the following disclaimer in the documentation
|
|
14
|
+
and/or other materials provided with the distribution.
|
|
15
|
+
|
|
16
|
+
3. Neither the name of the copyright holder nor the names of its
|
|
17
|
+
contributors may be used to endorse or promote products derived from
|
|
18
|
+
this software without specific prior written permission.
|
|
19
|
+
|
|
20
|
+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
|
|
21
|
+
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
22
|
+
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
|
|
23
|
+
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
|
|
24
|
+
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
25
|
+
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
|
|
26
|
+
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
|
|
27
|
+
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
|
|
28
|
+
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
|
29
|
+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
@@ -0,0 +1,374 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: vocal-helper
|
|
3
|
+
Version: 0.4.1
|
|
4
|
+
Summary: Vocal Helper — async producer/consumer pipeline turning a live PCM stream into diarized, transcribed utterances and (optionally) a rolling LLM summary. Stages: VAD (Silero) → online speaker diarization (Hungarian-style cosine clustering on backend embeddings) → STT (pywhispercpp turbo) → optional analyst (Gemma 3 4b via Ollama).
|
|
5
|
+
Author-email: Warith HARCHAOUI <warith@deraison.ai>
|
|
6
|
+
License: BSD-3-Clause
|
|
7
|
+
Project-URL: Homepage, https://github.com/warith-harchaoui/vocal-helper
|
|
8
|
+
Project-URL: Documentation, https://github.com/warith-harchaoui/vocal-helper#readme
|
|
9
|
+
Project-URL: Repository, https://github.com/warith-harchaoui/vocal-helper
|
|
10
|
+
Project-URL: Issues, https://github.com/warith-harchaoui/vocal-helper/issues
|
|
11
|
+
Project-URL: Changelog, https://github.com/warith-harchaoui/vocal-helper/blob/main/CHANGELOG.md
|
|
12
|
+
Project-URL: Deployment guide, https://github.com/warith-harchaoui/vocal-helper/blob/main/TECHNICAL_STACK.md
|
|
13
|
+
Project-URL: Study log, https://github.com/warith-harchaoui/vocal-helper/tree/main/studies
|
|
14
|
+
Keywords: voice,speech,asr,diarization,vad,whisper,pywhispercpp,silero,streaming,producer-consumer,asyncio,ollama,gemma,live
|
|
15
|
+
Classifier: Development Status :: 3 - Alpha
|
|
16
|
+
Classifier: Intended Audience :: Developers
|
|
17
|
+
Classifier: License :: OSI Approved :: BSD License
|
|
18
|
+
Classifier: Operating System :: OS Independent
|
|
19
|
+
Classifier: Programming Language :: Python :: 3
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
21
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
22
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
23
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
24
|
+
Classifier: Topic :: Multimedia :: Sound/Audio :: Speech
|
|
25
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
26
|
+
Requires-Python: <3.14,>=3.10
|
|
27
|
+
Description-Content-Type: text/markdown
|
|
28
|
+
License-File: LICENSE
|
|
29
|
+
Requires-Dist: os-helper>=1.5.0
|
|
30
|
+
Requires-Dist: audio-helper>=1.5.5
|
|
31
|
+
Requires-Dist: numpy>=1.24
|
|
32
|
+
Requires-Dist: scipy>=1.15
|
|
33
|
+
Requires-Dist: silero-vad>=5.1
|
|
34
|
+
Requires-Dist: pywhispercpp>=1.2
|
|
35
|
+
Provides-Extra: mic
|
|
36
|
+
Requires-Dist: capture-helper>=0.2.2; extra == "mic"
|
|
37
|
+
Provides-Extra: stream
|
|
38
|
+
Requires-Dist: podcast-helper>=0.3.3; extra == "stream"
|
|
39
|
+
Provides-Extra: pyannote
|
|
40
|
+
Requires-Dist: pyannote.audio>=3.3; extra == "pyannote"
|
|
41
|
+
Provides-Extra: nemo
|
|
42
|
+
Requires-Dist: torch<3,>=2.5; extra == "nemo"
|
|
43
|
+
Requires-Dist: torchaudio<3,>=2.5; extra == "nemo"
|
|
44
|
+
Requires-Dist: nemo-toolkit[asr]>=2.0; extra == "nemo"
|
|
45
|
+
Provides-Extra: llm
|
|
46
|
+
Requires-Dist: ollama>=0.4; extra == "llm"
|
|
47
|
+
Provides-Extra: cli
|
|
48
|
+
Requires-Dist: click<9,>=8.1; extra == "cli"
|
|
49
|
+
Provides-Extra: api
|
|
50
|
+
Requires-Dist: fastapi>=0.115; extra == "api"
|
|
51
|
+
Requires-Dist: uvicorn[standard]>=0.30; extra == "api"
|
|
52
|
+
Requires-Dist: python-multipart>=0.0.9; extra == "api"
|
|
53
|
+
Provides-Extra: mcp
|
|
54
|
+
Requires-Dist: fastapi-mcp>=0.3; extra == "mcp"
|
|
55
|
+
Provides-Extra: lid
|
|
56
|
+
Requires-Dist: speechbrain>=1.0; extra == "lid"
|
|
57
|
+
Requires-Dist: torchaudio>=2.0; extra == "lid"
|
|
58
|
+
Provides-Extra: all
|
|
59
|
+
Requires-Dist: vocal-helper[api,cli,lid,llm,mcp,mic,nemo,pyannote,stream]; extra == "all"
|
|
60
|
+
Provides-Extra: dev
|
|
61
|
+
Requires-Dist: pytest>=8; extra == "dev"
|
|
62
|
+
Requires-Dist: click<9,>=8.1; extra == "dev"
|
|
63
|
+
Requires-Dist: fastapi>=0.115; extra == "dev"
|
|
64
|
+
Requires-Dist: httpx>=0.27; extra == "dev"
|
|
65
|
+
Requires-Dist: python-multipart>=0.0.9; extra == "dev"
|
|
66
|
+
Dynamic: license-file
|
|
67
|
+
|
|
68
|
+
# Vocal Helper
|
|
69
|
+
|
|
70
|
+
[🇫🇷](https://github.com/warith-harchaoui/vocal-helper/blob/main/LISEZMOI.md) · [🇬🇧](https://github.com/warith-harchaoui/vocal-helper/blob/main/README.md)
|
|
71
|
+
|
|
72
|
+
[](https://github.com/warith-harchaoui/vocal-helper/actions/workflows/ci.yml)
|
|
73
|
+
[](LICENSE)
|
|
74
|
+
[](#)
|
|
75
|
+
[](https://github.com/astral-sh/ruff)
|
|
76
|
+
[](.github/PULL_REQUEST_TEMPLATE.md)
|
|
77
|
+
|
|
78
|
+
`Vocal Helper` belongs to a collection of libraries called `AI Helpers` developed for building Artificial Intelligence.
|
|
79
|
+
|
|
80
|
+
[🌍 AI Helpers](https://harchaoui.org/warith/ai-helpers)
|
|
81
|
+
|
|
82
|
+
[](https://harchaoui.org/warith/ai-helpers)
|
|
83
|
+
|
|
84
|
+
Vocal Helper is an **async producer/consumer pipeline** turning audio into diarized, transcribed utterances — and (optionally) a rolling LLM summary of the conversation. Two paths ship :
|
|
85
|
+
|
|
86
|
+
- **Online** (`voh.Pipeline`) — live PCM stream → live transcript + live summary. Each stage runs at its own cadence, decoupled by bounded queues. The STT stage warms up on start so the first caption doesn't stall on whisper's cold inference.
|
|
87
|
+
- **Offline** (`voh.OfflinePipeline`) — full audio buffer → highest-quality diarization (pyannote 3.1 runs the whole meeting in one call — the 2026-07-14 offline map-reduce study found whole-buffer strictly best for DER; chunk-and-stitch survives only as a memory backstop past ~1 h) → **full-throttle batched transcript** (consecutive segments concatenated into ≤ 24 s whisper calls — ~6.5× lower RTF at better WER per the 2026-07-09 sweep) → summary. Opt back into per-segment ASR with `OfflinePipelineConfig(asr={"batch": False})`.
|
|
88
|
+
|
|
89
|
+
## Pipelines
|
|
90
|
+
|
|
91
|
+
Every edge is a bounded `asyncio.Queue` ; every stage is its own
|
|
92
|
+
coroutine. Colours follow the
|
|
93
|
+
[AI Helpers palette](https://harchaoui.org/warith/colors/).
|
|
94
|
+
|
|
95
|
+
### Online (streaming)
|
|
96
|
+
|
|
97
|
+
```mermaid
|
|
98
|
+
flowchart LR
|
|
99
|
+
S([Source<br/><i>PCM frames</i>]):::source
|
|
100
|
+
--> V[VAD<br/><i>Silero v5 ONNX</i>]:::vad
|
|
101
|
+
--> D[Online Diar<br/><i>TitaNet · cosine clustering</i>]:::diar
|
|
102
|
+
--> A[STT<br/><i>whisper.cpp turbo</i>]:::asr
|
|
103
|
+
-.-> L[LLM analyst<br/><i>gemma3:4b · rolling summary</i>]:::llm
|
|
104
|
+
|
|
105
|
+
classDef source fill:#CCE4FF,stroke:#007AFF,stroke-width:2px,color:#0b3d91
|
|
106
|
+
classDef vad fill:#00ffef,stroke:#79dbdc,stroke-width:2px,color:#003b3c
|
|
107
|
+
classDef diar fill:#EFDCF8,stroke:#AF52DE,stroke-width:2px,color:#4a1063
|
|
108
|
+
classDef asr fill:#FFEACC,stroke:#FF9500,stroke-width:2px,color:#5a3300
|
|
109
|
+
classDef llm fill:#D4F5D9,stroke:#28CD41,stroke-width:2px,color:#144d1e,stroke-dasharray: 5 5
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
The dashed edge marks the analyst as optional (`llm=None` disables it).
|
|
113
|
+
|
|
114
|
+
### Offline (batch)
|
|
115
|
+
|
|
116
|
+
```mermaid
|
|
117
|
+
flowchart LR
|
|
118
|
+
S([Source<br/><i>full PCM buffer</i>]):::source
|
|
119
|
+
--> D[Offline Diar<br/><i>pyannote 3.1<br/>whole-buffer</i>]:::diar
|
|
120
|
+
--> A[STT<br/><i>whisper.cpp turbo</i>]:::asr
|
|
121
|
+
-.-> L[LLM analyst<br/><i>gemma3:4b · rolling summary</i>]:::llm
|
|
122
|
+
|
|
123
|
+
classDef source fill:#CCE4FF,stroke:#007AFF,stroke-width:2px,color:#0b3d91
|
|
124
|
+
classDef diar fill:#EFDCF8,stroke:#AF52DE,stroke-width:2px,color:#4a1063
|
|
125
|
+
classDef asr fill:#FFEACC,stroke:#FF9500,stroke-width:2px,color:#5a3300
|
|
126
|
+
classDef llm fill:#D4F5D9,stroke:#28CD41,stroke-width:2px,color:#144d1e,stroke-dasharray: 5 5
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
No VAD in the offline path — the diarizer consumes the whole buffer
|
|
130
|
+
and does its own segmentation.
|
|
131
|
+
|
|
132
|
+
| Stage | Backend | Notes |
|
|
133
|
+
|---|---|---|
|
|
134
|
+
| **VAD** *(online only)* | Silero v5 ONNX (CPU) | 32 ms window, `activity_threshold=0.5`, default `min_silence_ms=300`. |
|
|
135
|
+
| **Online diarization** | `nvidia/titanet_large` (NeMo, default) or `pyannote/embedding` | Per-segment embedding + cosine-distance running-mean clustering, `join_threshold=0.30`. Default backend switched to NeMo by the 2026-06-30 embedding sweep (`studies/diar_embedding_backend.py`): TitaNet has **+76 % separability margin** (inter − intra median cosine = 0.354 vs pyannote 0.201) on AMI dev-slice, at 7× per-call latency (45 ms vs 6 ms — still negligible per voiced segment). Pass `backend='pyannote'` to skip the ~ 5 GB NeMo install. |
|
|
136
|
+
| **Offline diarization** | `pyannote/speaker-diarization-3.1` (default) or `nvidia/diar_sortformer_v1` (NeMo) | Whole-buffer call. Inputs longer than `ideal_duration_s` (**3600 s** for pyannote — effectively whole-buffer, chunking is a memory backstop only; **60 s** for NeMo, forced by its Sortformer 90 s cap) are auto-chunked with 10 s overlap and stitched by cosine AHC at `stitch_threshold=0.35`. The 2026-07-14 offline map-reduce study found whole-buffer strictly best for DER (0.143 vs 0.170 at 300 s). |
|
|
137
|
+
| **STT** | [`pywhispercpp`](https://github.com/abdeladim-s/pywhispercpp) turbo | `large-v3-turbo-q5_0` by default. Word timestamps on. Runs in a thread pool so the event loop never stalls. **Strongly recommended: supply `initial_prompt` (domain bias)** — cuts WER 15-25 pp and saves up to 39 % RTF per the 2026-06-30 sweep (`studies/whisper_prompt_lang_lock.py`). |
|
|
138
|
+
| **LLM analyst** *(optional)* | Ollama-served Gemma 3 4b (`gemma3:4b`) | Rolling summary of everything **older than 60 s**. The recent 60 s window is kept verbatim. Summary refreshes every **60 s of evicted content** (`flush_every_s=60`). Default model `gemma3:4b` selected by the 2026-06-30 7-model Pareto sweep (`studies/llm_model_size_sweep.py`): it dominates `gemma4:e4b-mlx` on BOTH RTF (0.099 vs 0.313, **3× faster**) AND cos_sim (0.466 vs 0.420). Pareto front also exposes `gemma4:12b-mlx` (RTF 2.45, cos_sim 0.496) for offline-batch quality runs, and `qwen2.5:3b` (RTF 0.043) for tight RTF budgets. |
|
|
139
|
+
|
|
140
|
+
## Quickstart
|
|
141
|
+
|
|
142
|
+
> **More recipes?** See [`EXAMPLES.md`](https://github.com/warith-harchaoui/vocal-helper/blob/main/EXAMPLES.md)
|
|
143
|
+
> for a self-contained, copy-runnable cookbook of the common workflows
|
|
144
|
+
> (live mic, URL replay, offline batch, subscribers, multi-surface exposure).
|
|
145
|
+
|
|
146
|
+
> **Deploying on a GPU server?** See [TECHNICAL_STACK.md](https://github.com/warith-harchaoui/vocal-helper/blob/main/TECHNICAL_STACK.md)
|
|
147
|
+
> for the full stack recipe : CUDA + PyTorch, whisper.cpp with `GGML_CUDA=on`,
|
|
148
|
+
> pyannote 3.1 on MPS/CUDA, Ollama systemd, Docker Compose template,
|
|
149
|
+
> expected RTFs per GPU, and a 10-step reproducible install manifest
|
|
150
|
+
> covering the whole AI Helpers suite (os-helper, audio-helper,
|
|
151
|
+
> podcast-helper, youtube-helper, vocal-helper, music-helper).
|
|
152
|
+
|
|
153
|
+
### Install
|
|
154
|
+
|
|
155
|
+
**Prerequisites** — **Python 3.10–3.13** and **git**, **ffmpeg**, **PortAudio**, cross-platform:
|
|
156
|
+
|
|
157
|
+
- 🍎 **macOS** ([Homebrew](https://brew.sh)): `brew install python git ffmpeg portaudio`
|
|
158
|
+
- 🐧 **Ubuntu/Debian**: `sudo apt update && sudo apt install -y python3 python3-pip git ffmpeg portaudio19-dev`
|
|
159
|
+
- 🪟 **Windows** (PowerShell): `winget install Python.Python.3.12 Git.Git Gyan.FFmpeg` (PortAudio ships inside the Python wheels)
|
|
160
|
+
|
|
161
|
+
Then install the package:
|
|
162
|
+
|
|
163
|
+
|
|
164
|
+
```bash
|
|
165
|
+
pip install 'vocal-helper[all]'
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
The `[all]` extra brings the mic source, both diarization backends (NeMo — the default — and pyannote), and Ollama. Pick à la carte if you don't need everything :
|
|
169
|
+
|
|
170
|
+
| Extra | Brings | Required when |
|
|
171
|
+
|---|---|---|
|
|
172
|
+
| (none) | `pywhispercpp`, `silero-vad`, `audio-helper` | File / numpy sources, no diarization |
|
|
173
|
+
| `[mic]` | `capture-helper` | Live microphone source |
|
|
174
|
+
| `[pyannote]` | `pyannote.audio` | `diar={'backend': 'pyannote'}` (lighter ~500 MB fallback) |
|
|
175
|
+
| `[nemo]` | `torch`, `nemo-toolkit[asr]` | `diar={'backend': 'nemo'}` (default — TitaNet, ~5 GB) |
|
|
176
|
+
| `[llm]` | `ollama` | `llm={'model': 'gemma3:4b'}` (default) |
|
|
177
|
+
| `[all]` | All of the above | One-line install |
|
|
178
|
+
|
|
179
|
+
You also need [Ollama](https://ollama.com) running locally if you enable the LLM analyst :
|
|
180
|
+
|
|
181
|
+
```bash
|
|
182
|
+
ollama pull gemma3:4b # default (or gemma4:12b-mlx for max quality, qwen2.5:3b for min RTF)
|
|
183
|
+
ollama serve # usually launched at install time
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
### Model weights — no HuggingFace needed
|
|
187
|
+
|
|
188
|
+
All model weights ship in a single self-hosted **diarization-engines
|
|
189
|
+
bundle** (offline pyannote 3.1, NeMo Sortformer, the online
|
|
190
|
+
`pyannote/embedding` embedder, SpeechBrain VoxLingua107). Point
|
|
191
|
+
`vocal-helper` at it once and the whole stack runs **HuggingFace-free** —
|
|
192
|
+
no token, no gated downloads, `HF_HUB_OFFLINE=1` safe.
|
|
193
|
+
|
|
194
|
+
Configure it in `settings.yaml` (the only config the project needs):
|
|
195
|
+
|
|
196
|
+
```bash
|
|
197
|
+
cp settings.yaml.example settings.yaml
|
|
198
|
+
# settings.yaml already contains:
|
|
199
|
+
# engines:
|
|
200
|
+
# diarization_url: https://deraison.ai/diarization-engines.zip
|
|
201
|
+
# settings.yaml is git-ignored.
|
|
202
|
+
```
|
|
203
|
+
|
|
204
|
+
#### What the URL is
|
|
205
|
+
|
|
206
|
+
`https://deraison.ai/diarization-engines.zip` is a **self-hosted ZIP**
|
|
207
|
+
(~800 MB) that mirrors every gated/hub-hosted model the pipeline needs, so
|
|
208
|
+
the project never has to authenticate against HuggingFace. It contains:
|
|
209
|
+
|
|
210
|
+
| Folder | Weights | Used by |
|
|
211
|
+
|---|---|---|
|
|
212
|
+
| `pyannote-3.1/` | segmentation-3.0 + wespeaker `.bin` + a local `config.yaml` | offline diarization |
|
|
213
|
+
| `nemo-sortformer/` | `diar_sortformer_4spk-v1.nemo` | offline diarization (NeMo) |
|
|
214
|
+
| `pyannote-embedding/` | embedding `.bin` | online diarization |
|
|
215
|
+
| `speechbrain-voxlingua107/` | ECAPA VoxLingua107 snapshot | language-ID cross-check |
|
|
216
|
+
| `manifest.json` | sha256 + sizes | integrity check on download |
|
|
217
|
+
|
|
218
|
+
On first use it is downloaded once, verified against `manifest.json`, and
|
|
219
|
+
cached under `~/.cache/vocal-helper`; later runs load straight from the
|
|
220
|
+
cache. Set `$VH_DIARIZATION_ENGINES` to a local directory (or your own
|
|
221
|
+
mirror URL) for air-gapped / self-hosted deploys. TitaNet (the default
|
|
222
|
+
online-diar embedder) loads from NVIDIA NGC, also without HuggingFace.
|
|
223
|
+
|
|
224
|
+
### Live microphone → terminal
|
|
225
|
+
|
|
226
|
+
```bash
|
|
227
|
+
# No token, no HuggingFace — weights come from the diarization-engines bundle.
|
|
228
|
+
vocal-helper mic --llm
|
|
229
|
+
```
|
|
230
|
+
|
|
231
|
+
### Python API
|
|
232
|
+
|
|
233
|
+
```python
|
|
234
|
+
import asyncio
|
|
235
|
+
import vocal_helper as voh
|
|
236
|
+
|
|
237
|
+
async def main():
|
|
238
|
+
pipeline = voh.Pipeline(
|
|
239
|
+
source=lambda: voh.sources.from_microphone(),
|
|
240
|
+
config=voh.PipelineConfig(
|
|
241
|
+
diar={"backend": "pyannote"},
|
|
242
|
+
asr={"model": "large-v3-turbo-q5_0", "language": "fr"},
|
|
243
|
+
llm={"model": "gemma3:4b"}, # remove to disable
|
|
244
|
+
),
|
|
245
|
+
)
|
|
246
|
+
async for ev in pipeline.run():
|
|
247
|
+
if "text" in ev:
|
|
248
|
+
print(f"[{ev['t0']:.1f} {ev['speaker']}] {ev['text']}")
|
|
249
|
+
elif "summary" in ev:
|
|
250
|
+
print(f"--- rolling summary ---\n{ev['summary']}")
|
|
251
|
+
|
|
252
|
+
asyncio.run(main())
|
|
253
|
+
```
|
|
254
|
+
|
|
255
|
+
### Replay a WAV through the **online** pipeline
|
|
256
|
+
|
|
257
|
+
```bash
|
|
258
|
+
vocal-helper file path/to/conversation.wav --llm
|
|
259
|
+
```
|
|
260
|
+
|
|
261
|
+
The file source preserves real-time pacing by default ; pass `--no-real-time` for as-fast-as-possible batch processing.
|
|
262
|
+
|
|
263
|
+
### **Offline** batch on a WAV (full-buffer pyannote 3.1)
|
|
264
|
+
|
|
265
|
+
```python
|
|
266
|
+
import asyncio, vocal_helper as voh
|
|
267
|
+
|
|
268
|
+
async def main():
|
|
269
|
+
pipeline = voh.OfflinePipeline(
|
|
270
|
+
source=lambda: voh.sources.from_wav_file(
|
|
271
|
+
"meeting.wav", real_time=False
|
|
272
|
+
),
|
|
273
|
+
config=voh.OfflinePipelineConfig(
|
|
274
|
+
diar={"backend": "pyannote"}, # or "nemo" for ≤ 60 s clips
|
|
275
|
+
asr={"language": "en"},
|
|
276
|
+
llm={"model": "gemma3:4b"}, # remove to disable
|
|
277
|
+
),
|
|
278
|
+
)
|
|
279
|
+
async for ev in pipeline.run():
|
|
280
|
+
if "text" in ev:
|
|
281
|
+
print(f"[{ev['t0']:.1f} {ev['speaker']}] {ev['text']}")
|
|
282
|
+
elif "summary" in ev:
|
|
283
|
+
print(f"--- digest ---\n{ev['summary']}")
|
|
284
|
+
|
|
285
|
+
asyncio.run(main())
|
|
286
|
+
```
|
|
287
|
+
|
|
288
|
+
When to use which :
|
|
289
|
+
|
|
290
|
+
| Use-case | Pipeline | Why |
|
|
291
|
+
|---|---|---|
|
|
292
|
+
| Live mic / live stream | `Pipeline` | Real-time diarization + transcript at RTF ≈ 0.03 (NeMo) or RTF ≈ 0.06 (pyannote). |
|
|
293
|
+
| Meeting recording / podcast / lecture / voicemail batch | `OfflinePipeline` | Pyannote 3.1 on the full audio is the highest-DER answer per the pdbms 2026-06-29 canonical study — median DER 0.116 on AMI dev-slice, inside Bredin 2023's band. |
|
|
294
|
+
| ≤ 60 s clips, fast turn-around | `OfflinePipeline(backend='nemo')` | NeMo Sortformer end-to-end attribution gives conf ≈ 0 ; RTF ≈ 0.004. |
|
|
295
|
+
|
|
296
|
+
## Multi-surface exposure
|
|
297
|
+
|
|
298
|
+
`vocal-helper` exposes the same pipeline through four coherent surfaces so it can slot into any host — a shell, a Python script, a container behind a reverse proxy, or an MCP-aware agent — without re-implementing the wiring anywhere else.
|
|
299
|
+
|
|
300
|
+
| Surface | Entry point | Extra | Kind of use |
|
|
301
|
+
|---|---|---|---|
|
|
302
|
+
| argparse CLI | `vocal-helper` | (none — ships with the base install) | Shell scripts, cron, headless CI, pipes to `jq`. |
|
|
303
|
+
| click CLI | `vocal-helper-click` | `[cli]` | Rich `--help`, shell completion, chained sub-commands. |
|
|
304
|
+
| FastAPI HTTP | `uvicorn vocal_helper.api:app` | `[api]` | Behind a reverse proxy — upload a file, get a transcript / a full event list, `GET /docs` for the OpenAPI. |
|
|
305
|
+
| MCP tools | `vocal-helper-mcp` | `[api,mcp]` | Any MCP-aware host — agent runtimes, IDE integrations — publishes `transcribe` and `pipeline` as first-class tools. |
|
|
306
|
+
|
|
307
|
+
```bash
|
|
308
|
+
# argparse
|
|
309
|
+
vocal-helper transcribe clip.wav --language en
|
|
310
|
+
vocal-helper file meeting.wav --offline --language en --llm
|
|
311
|
+
|
|
312
|
+
# click twin
|
|
313
|
+
vocal-helper-click transcribe clip.wav --language en
|
|
314
|
+
|
|
315
|
+
# HTTP surface
|
|
316
|
+
uvicorn vocal_helper.api:app --host 0.0.0.0 --port 8000 &
|
|
317
|
+
curl -F 'file=@clip.wav' -F 'language=en' http://localhost:8000/transcribe
|
|
318
|
+
curl -F 'file=@meeting.wav' -F 'llm=true' http://localhost:8000/pipeline
|
|
319
|
+
|
|
320
|
+
# MCP surface (same FastAPI app + /mcp endpoint mounted)
|
|
321
|
+
vocal-helper-mcp
|
|
322
|
+
```
|
|
323
|
+
|
|
324
|
+
A one-line container recipe ships in `Dockerfile` — `docker build -t vocal-helper .` gets you an image serving both HTTP and MCP on `:8000`. See `GUI.md` for the (WIP) visual-product plan.
|
|
325
|
+
|
|
326
|
+
## Subscribers — fan-out without owning the loop
|
|
327
|
+
|
|
328
|
+
Every stage can be observed without consuming the merged output stream :
|
|
329
|
+
|
|
330
|
+
```python
|
|
331
|
+
async def on_voiced(seg): print("VAD:", seg["t0"], seg["t1"])
|
|
332
|
+
async def on_diar(seg): print(" → ", seg["speaker"], seg["t0"], seg["t1"])
|
|
333
|
+
|
|
334
|
+
pipeline.subscribe_voiced(on_voiced)
|
|
335
|
+
pipeline.subscribe_diarized(on_diar)
|
|
336
|
+
|
|
337
|
+
async for ev in pipeline.run():
|
|
338
|
+
...
|
|
339
|
+
```
|
|
340
|
+
|
|
341
|
+
Useful for WebSocket / SSE relays, live UI updates, or JSONL persistence.
|
|
342
|
+
|
|
343
|
+
## Diarization choice — why **online cosine clustering**
|
|
344
|
+
|
|
345
|
+
The `pdbms` study (2026-06-29, N=2089 per system) ranks the online streaming diarizers as :
|
|
346
|
+
|
|
347
|
+
| Mode | Recommended | DER (clean) |
|
|
348
|
+
|---|---|---|
|
|
349
|
+
| Streaming ≤ 300 s | `hungarian_nemo` (w=20 s) | 0.13 – 0.20 |
|
|
350
|
+
| Streaming > 300 s | `hungarian_pyannote` (w=30 s) | 0.30 – 0.45 |
|
|
351
|
+
|
|
352
|
+
Vocal Helper specialises that decision : since the VAD already isolates each voiced segment for us, the sliding-window machinery collapses to per-segment embedding + cosine-distance running-mean clustering. The default `join_threshold=0.30` is the value selected on AMI dev-slice N=8 in the 2026-06-30 `pyannote_stitch_threshold_sweep`.
|
|
353
|
+
|
|
354
|
+
## Roadmap
|
|
355
|
+
|
|
356
|
+
- v0.2 — JSONL output writer + standard WebSocket relay (mirroring `capture-helper`'s publish path).
|
|
357
|
+
- v0.2 — language-locked Whisper rejection for ASR hallucinations on silence.
|
|
358
|
+
- v0.2 — `SemanticEOTStage` enabled-by-default after the 2026-06-30 EOT study validates the false-cut reduction on AMI (LiveKit-style ; see `studies/eot_semantic_vs_silero.py`).
|
|
359
|
+
- v0.2 — auto LLM-engine selector : Ollama+MLX on Apple Silicon, vLLM on Linux+NVIDIA, llama.cpp gguf on CPU fallback.
|
|
360
|
+
- v0.3 — out of scope : speaker ID anchoring via pre-enrolled voiceprints (excluded by user's industrial deployment compliance constraints — IDs stay anonymous `S0`, `S1`, … within a session).
|
|
361
|
+
- v0.3 — replace the in-stage `_PyannoteEmbedder` with the overlap-aware variant from `pdbms.diar.backends.pyannote.embed_overlap_aware` for noisy mixes.
|
|
362
|
+
- v0.3 — Pipecat-style typed Frame events with SystemFrame priority queue (clean shutdown / out-of-band control signals that bypass DataFrame queues).
|
|
363
|
+
|
|
364
|
+
## Author
|
|
365
|
+
|
|
366
|
+
[Warith HARCHAOUI](https://linkedin.com/in/warith-harchaoui) — `warith@deraison.ai`
|
|
367
|
+
|
|
368
|
+
## Acknowledgements
|
|
369
|
+
|
|
370
|
+
Special thanks to
|
|
371
|
+
[Mohamed Chelali](https://mchelali.github.io),
|
|
372
|
+
[Bachir Zerroug](https://www.linkedin.com/in/bachirzerroug)
|
|
373
|
+
and
|
|
374
|
+
[Edmond Jacoupeau](https://www.crunchbase.com/person/edmond-jacoupeau).
|