realtimevad 0.0.4__tar.gz

realtimevad-0.0.4/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Kolja Beigel
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

realtimevad-0.0.4/PKG-INFO

@@ -0,0 +1,174 @@
+Metadata-Version: 2.4
+Name: realtimevad
+Version: 0.0.4
+Summary: Fast speech-end detection for low-latency voice assistants
+Author: Kolja Beigel
+License: MIT
+Project-URL: Homepage, https://github.com/KoljaB/RealtimeVAD
+Project-URL: Issues, https://github.com/KoljaB/RealtimeVAD/issues
+Keywords: vad,voice activity detection,endpointing,turn detection,voice assistant,realtime
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Multimedia :: Sound/Audio :: Speech
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.21
+Requires-Dist: onnxruntime>=1.16
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Dynamic: license-file
+
+# RealtimeVAD
+
+*Know that the speaker stopped, about 110 ms before Silero does.*
+
+[RealtimeSTT](https://github.com/KoljaB/RealtimeSTT) · [RealtimeTTS](https://github.com/KoljaB/RealtimeTTS) · [Live demo](https://github.com/KoljaB/RealtimeVAD/blob/main/demo/README.md) · [Spec](https://github.com/KoljaB/RealtimeVAD/blob/main/SPEC.md)
+
+A voice assistant can't do anything until it knows the user has stopped
+talking. Speech recognition, the LLM call and the voice output all wait for
+that one decision, and a normal voice activity detector takes a few hundred
+milliseconds to make it.
+
+RealtimeVAD makes it sooner. It keeps [Silero VAD](https://github.com/snakers4/silero-vad)
+to decide what is speech, and adds a second, faster check: every 10 ms it
+compares the audio level with the room's noise. The moment the audio goes
+quiet, you get an event, long before Silero's probability has fallen.
+
+- **Input:** 16 kHz mono audio in chunks of any size (float32, int16 or int16 bytes).
+- **Output:** `start`, `pause` and `end` events, plus the audio of the running segment at any time.
+- **Lead:** speech-off a median 110 ms before Silero's probability drops, and 214 ms before Silero's `VADIterator` reports the end, on 3,238 real turns.
+- **Price:** it also fires in short gaps between words, about ten times as often as Silero.
+- **Runs on:** one CPU thread, far faster than realtime. Needs only numpy and onnxruntime; the Silero model is bundled, no torch.
+- **Status:** alpha (0.0.4). The API may still change.
+
+## Live demo
+
+https://github.com/user-attachments/assets/d01900dd-9601-4f3d-8c88-db55e8bc83a1
+
+The video uses a synthetic voice and shows a median lead of 92 ms. The 110 ms above is the median on real voice-assistant turns.
+
+[demo/](https://github.com/KoljaB/RealtimeVAD/blob/main/demo/README.md) runs this page on your own microphone. It shows, turn by turn, how much sooner
+RealtimeVAD knew that you stopped than Silero alone, on the same audio.
+
+## Quick start
+
+```bash
+pip install realtimevad
+```
+
+```python
+from RealtimeVAD import VAD
+
+vad = VAD()
+for chunk in microphone_chunks():
+    for event in vad.feed(chunk):
+        if event.kind == "start":
+            stream_to_asr(vad.audio())       # includes the audio just before speech began
+        elif event.kind == "pause":
+            speculate(vad.audio())           # the speaker may be done; this isn't the end yet
+        elif event.kind == "end":
+            finalize(event.segment.audio)
+```
+
+## Events
+
+| Event | Meaning | When |
+|---|---|---|
+| `start` | Speech has begun | After about 64 ms of confident speech, so clicks don't trigger it |
+| `pause` | The speaker may have stopped (speech-off) | Within about 10 ms of the audio going quiet |
+| `end` | The segment is over | 0.24 s after speech-off, unless speech resumes. Never later than Silero alone |
+| `onset` | Speech may have begun | At the first confident frame, before `start` confirms it. Off by default: `VAD(onset_events=True)` |
+
+`vad.audio()` returns the running segment at any time, starting about 0.3 s
+before the speech. `end` carries the finished segment with its audio, start
+time, end time and the time the speech stopped.
+
+If your assistant talks while the microphone is open, pass
+`vad.feed(chunk, playback=True)` for those chunks so its own voice is handled
+as possible echo. Every threshold and duration can be configured;
+[SPEC.md](https://github.com/KoljaB/RealtimeVAD/blob/main/SPEC.md) lists what is guaranteed.
+
+## How to use `pause`
+
+`pause` is early and often wrong, on purpose. It fires on every breath, every
+short gap between words and every stop consonant, so most `pause` events are
+followed by more speech.
+
+- **Use it to start work you can cancel:** finalize the transcript, call the
+  LLM, prepare the voice output. Cancel when speech continues. A wrong `pause`
+  then costs almost nothing, and a right one saves hundreds of milliseconds.
+- **Never treat it as the end of the turn.** Only `end` closes a segment.
+- **Can't cancel quickly?** Ignore `pause` and wait for `end`. The extra
+  false alarms would cost you more than they save.
+
+## Already using Silero?
+
+Change the import:
+
+```python
+from silero_vad import load_silero_vad
+from RealtimeVAD import VADIterator          # was: from silero_vad import VADIterator
+
+vad_iterator = VADIterator(load_silero_vad(), sampling_rate=16000)
+speech_dict = vad_iterator(chunk, return_seconds=True)   # {'start': ...} / {'end': ...} / None
+```
+
+The constructor and the results stay the same, but `end` arrives as soon as the
+audio goes quiet. Without a model argument, it uses the bundled Silero model and
+doesn't need torch.
+
+## Results
+
+Measured on 3,238 turns from a live voice assistant, each with its trailing
+silence, with default settings. Silero's decisions are computed from the very
+same frame probabilities, so both see identical input. "Speech-off" is
+RealtimeVAD's last `pause` before the turn ended, against the first frame of
+Silero's final run below its threshold.
+
+| RealtimeVAD speech-off earlier than … | median | p25 – p75 | p90 | earlier in |
+|---|---|---|---|---|
+| Silero, probability below 0.5 (its default threshold) | 110 ms | 84 – 190 ms | 242 ms | 98 % |
+| Silero, probability below 0.35 (`VADIterator`'s end threshold) | 120 ms | 98 – 200 ms | 252 ms | 98 % |
+| Silero `VADIterator` end, default 100 ms minimum silence | 214 ms | 192 – 298 ms | 356 ms | 99 % |
+
+| Speech-off signals followed by more speech | per second of speech | turns without any |
+|---|---|---|
+| RealtimeVAD `pause` | 1.6 | 16 % |
+| Silero, probability below 0.5 | 0.15 | 85 % |
+
+Reproduce with your own recordings (16 kHz mono WAVs, one turn each, with
+trailing silence):
+
+```bash
+python benchmarks/speech_off_lead.py <wav dir> out.json
+```
+
+## Limits
+
+- The numbers come from one speaker and one capture chain: browser capture
+  with noise suppression and echo cancellation, mostly quiet rooms.
+- The lead needs audio that actually goes quiet after a word, and noise
+  suppression helps that. In the 147 noisier turns (room level above −70 dBFS)
+  the median lead was 92–132 ms. Without noise suppression in a loud room,
+  expect less.
+- Earlier is not the same as correct: whether a speech-off sometimes lands
+  inside a soft word ending was not checked against word timings.
+- The lead is for speech-off, not for the turn end. `end` waits 0.24 s after
+  speech-off in both cases; its own gain over Silero is smaller and depends on
+  the noise.
+- 16 kHz mono only.
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+pytest
+```
+
+## License and credits
+
+MIT. The bundled speech model is [Silero VAD](https://github.com/snakers4/silero-vad) (MIT).

realtimevad-0.0.4/README.md

@@ -0,0 +1,150 @@
+# RealtimeVAD
+
+*Know that the speaker stopped, about 110 ms before Silero does.*
+
+[RealtimeSTT](https://github.com/KoljaB/RealtimeSTT) · [RealtimeTTS](https://github.com/KoljaB/RealtimeTTS) · [Live demo](https://github.com/KoljaB/RealtimeVAD/blob/main/demo/README.md) · [Spec](https://github.com/KoljaB/RealtimeVAD/blob/main/SPEC.md)
+
+A voice assistant can't do anything until it knows the user has stopped
+talking. Speech recognition, the LLM call and the voice output all wait for
+that one decision, and a normal voice activity detector takes a few hundred
+milliseconds to make it.
+
+RealtimeVAD makes it sooner. It keeps [Silero VAD](https://github.com/snakers4/silero-vad)
+to decide what is speech, and adds a second, faster check: every 10 ms it
+compares the audio level with the room's noise. The moment the audio goes
+quiet, you get an event, long before Silero's probability has fallen.
+
+- **Input:** 16 kHz mono audio in chunks of any size (float32, int16 or int16 bytes).
+- **Output:** `start`, `pause` and `end` events, plus the audio of the running segment at any time.
+- **Lead:** speech-off a median 110 ms before Silero's probability drops, and 214 ms before Silero's `VADIterator` reports the end, on 3,238 real turns.
+- **Price:** it also fires in short gaps between words, about ten times as often as Silero.
+- **Runs on:** one CPU thread, far faster than realtime. Needs only numpy and onnxruntime; the Silero model is bundled, no torch.
+- **Status:** alpha (0.0.4). The API may still change.
+
+## Live demo
+
+https://github.com/user-attachments/assets/d01900dd-9601-4f3d-8c88-db55e8bc83a1
+
+The video uses a synthetic voice and shows a median lead of 92 ms. The 110 ms above is the median on real voice-assistant turns.
+
+[demo/](https://github.com/KoljaB/RealtimeVAD/blob/main/demo/README.md) runs this page on your own microphone. It shows, turn by turn, how much sooner
+RealtimeVAD knew that you stopped than Silero alone, on the same audio.
+
+## Quick start
+
+```bash
+pip install realtimevad
+```
+
+```python
+from RealtimeVAD import VAD
+
+vad = VAD()
+for chunk in microphone_chunks():
+    for event in vad.feed(chunk):
+        if event.kind == "start":
+            stream_to_asr(vad.audio())       # includes the audio just before speech began
+        elif event.kind == "pause":
+            speculate(vad.audio())           # the speaker may be done; this isn't the end yet
+        elif event.kind == "end":
+            finalize(event.segment.audio)
+```
+
+## Events
+
+| Event | Meaning | When |
+|---|---|---|
+| `start` | Speech has begun | After about 64 ms of confident speech, so clicks don't trigger it |
+| `pause` | The speaker may have stopped (speech-off) | Within about 10 ms of the audio going quiet |
+| `end` | The segment is over | 0.24 s after speech-off, unless speech resumes. Never later than Silero alone |
+| `onset` | Speech may have begun | At the first confident frame, before `start` confirms it. Off by default: `VAD(onset_events=True)` |
+
+`vad.audio()` returns the running segment at any time, starting about 0.3 s
+before the speech. `end` carries the finished segment with its audio, start
+time, end time and the time the speech stopped.
+
+If your assistant talks while the microphone is open, pass
+`vad.feed(chunk, playback=True)` for those chunks so its own voice is handled
+as possible echo. Every threshold and duration can be configured;
+[SPEC.md](https://github.com/KoljaB/RealtimeVAD/blob/main/SPEC.md) lists what is guaranteed.
+
+## How to use `pause`
+
+`pause` is early and often wrong, on purpose. It fires on every breath, every
+short gap between words and every stop consonant, so most `pause` events are
+followed by more speech.
+
+- **Use it to start work you can cancel:** finalize the transcript, call the
+  LLM, prepare the voice output. Cancel when speech continues. A wrong `pause`
+  then costs almost nothing, and a right one saves hundreds of milliseconds.
+- **Never treat it as the end of the turn.** Only `end` closes a segment.
+- **Can't cancel quickly?** Ignore `pause` and wait for `end`. The extra
+  false alarms would cost you more than they save.
+
+## Already using Silero?
+
+Change the import:
+
+```python
+from silero_vad import load_silero_vad
+from RealtimeVAD import VADIterator          # was: from silero_vad import VADIterator
+
+vad_iterator = VADIterator(load_silero_vad(), sampling_rate=16000)
+speech_dict = vad_iterator(chunk, return_seconds=True)   # {'start': ...} / {'end': ...} / None
+```
+
+The constructor and the results stay the same, but `end` arrives as soon as the
+audio goes quiet. Without a model argument, it uses the bundled Silero model and
+doesn't need torch.
+
+## Results
+
+Measured on 3,238 turns from a live voice assistant, each with its trailing
+silence, with default settings. Silero's decisions are computed from the very
+same frame probabilities, so both see identical input. "Speech-off" is
+RealtimeVAD's last `pause` before the turn ended, against the first frame of
+Silero's final run below its threshold.
+
+| RealtimeVAD speech-off earlier than … | median | p25 – p75 | p90 | earlier in |
+|---|---|---|---|---|
+| Silero, probability below 0.5 (its default threshold) | 110 ms | 84 – 190 ms | 242 ms | 98 % |
+| Silero, probability below 0.35 (`VADIterator`'s end threshold) | 120 ms | 98 – 200 ms | 252 ms | 98 % |
+| Silero `VADIterator` end, default 100 ms minimum silence | 214 ms | 192 – 298 ms | 356 ms | 99 % |
+
+| Speech-off signals followed by more speech | per second of speech | turns without any |
+|---|---|---|
+| RealtimeVAD `pause` | 1.6 | 16 % |
+| Silero, probability below 0.5 | 0.15 | 85 % |
+
+Reproduce with your own recordings (16 kHz mono WAVs, one turn each, with
+trailing silence):
+
+```bash
+python benchmarks/speech_off_lead.py <wav dir> out.json
+```
+
+## Limits
+
+- The numbers come from one speaker and one capture chain: browser capture
+  with noise suppression and echo cancellation, mostly quiet rooms.
+- The lead needs audio that actually goes quiet after a word, and noise
+  suppression helps that. In the 147 noisier turns (room level above −70 dBFS)
+  the median lead was 92–132 ms. Without noise suppression in a loud room,
+  expect less.
+- Earlier is not the same as correct: whether a speech-off sometimes lands
+  inside a soft word ending was not checked against word timings.
+- The lead is for speech-off, not for the turn end. `end` waits 0.24 s after
+  speech-off in both cases; its own gain over Silero is smaller and depends on
+  the noise.
+- 16 kHz mono only.
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+pytest
+```
+
+## License and credits
+
+MIT. The bundled speech model is [Silero VAD](https://github.com/snakers4/silero-vad) (MIT).

realtimevad-0.0.4/RealtimeVAD/__init__.py

@@ -0,0 +1,8 @@
+"""RealtimeVAD: fast speech-end detection for low-latency voice assistants."""
+
+from .iterator import VADIterator
+from .vad import VAD, Event, Segment
+
+__version__ = "0.0.4"
+
+__all__ = ["VAD", "VADIterator", "Event", "Segment", "__version__"]

realtimevad-0.0.4/RealtimeVAD/assets/LICENSE-silero-vad.txt

@@ -0,0 +1,23 @@
+silero_vad_v6.2.1.onnx comes from Silero VAD (https://github.com/snakers4/silero-vad).
+
+MIT License
+
+Copyright (c) 2020-present Silero Team
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

realtimevad-0.0.4/RealtimeVAD/iterator.py

@@ -0,0 +1,84 @@
+"""Drop-in for ``silero_vad.VADIterator``: same constructor, call and results."""
+
+from __future__ import annotations
+
+from typing import Optional
+
+import numpy as np
+
+from .silero import SAMPLE_RATE
+from .vad import FRAME_S, VAD
+
+
+class _SileroModel:
+    """Adapts a model from ``silero_vad.load_silero_vad()`` to a frame -> probability call."""
+
+    def __init__(self, model) -> None:
+        self._model = model
+        try:
+            import torch
+
+            self._tensor = torch.from_numpy
+        except ImportError:
+            self._tensor = lambda a: a
+
+    def __call__(self, frame: np.ndarray) -> float:
+        p = self._model(self._tensor(np.ascontiguousarray(frame)), SAMPLE_RATE)
+        return float(p.item() if hasattr(p, "item") else p)
+
+    def reset(self) -> None:
+        self._model.reset_states()
+
+
+class VADIterator:
+    """Like Silero's ``VADIterator``, but ``end`` comes as soon as the audio goes quiet.
+
+    ``model`` may be a Silero model (used as given) or None (the bundled one).
+    Extra keyword arguments go to :class:`VAD`. Chunks may have any size.
+    """
+
+    def __init__(
+        self,
+        model=None,
+        threshold: float = 0.5,
+        sampling_rate: int = 16000,
+        min_silence_duration_ms: int = 100,
+        speech_pad_ms: int = 30,
+        neg_threshold: Optional[float] = None,
+        **options,
+    ) -> None:
+        if sampling_rate != SAMPLE_RATE:
+            raise ValueError("RealtimeVAD supports 16000 Hz only")
+        if neg_threshold is None:
+            neg_threshold = max(threshold - 0.15, 0.01)
+        self.sampling_rate = sampling_rate
+        self.speech_pad_samples = sampling_rate * speech_pad_ms / 1000
+        self.vad = VAD(
+            start_threshold=threshold,
+            continue_threshold=neg_threshold,
+            start_s=FRAME_S,  # Silero starts on one frame
+            end_s=min_silence_duration_ms / 1000,
+            preroll_s=speech_pad_ms / 1000,
+            probability=None if model is None else _SileroModel(model),
+            **options,
+        )
+
+    @property
+    def triggered(self) -> bool:
+        return self.vad.in_speech
+
+    def reset_states(self) -> None:
+        self.vad.reset()
+
+    def __call__(self, x, return_seconds: bool = False, time_resolution: int = 1) -> Optional[dict]:
+        result = {}
+        for event in self.vad.feed(np.asarray(x)):
+            if event.kind == "start":  # Silero: start of the first speech frame, minus the pad
+                sample = (event.time - FRAME_S) * self.sampling_rate - self.speech_pad_samples
+            elif event.kind == "end":  # Silero: where speech stopped, plus the pad
+                sample = event.segment.speech_end * self.sampling_rate + self.speech_pad_samples
+            else:
+                continue
+            sample = max(0, int(round(sample)))
+            result[event.kind] = round(sample / self.sampling_rate, time_resolution) if return_seconds else sample
+        return result or None

realtimevad-0.0.4/RealtimeVAD/silero.py

@@ -0,0 +1,76 @@
+"""Stateful Silero v6 ONNX inference on 512-sample, 16 kHz frames."""
+
+from __future__ import annotations
+
+import math
+from pathlib import Path
+
+import numpy as np
+
+SAMPLE_RATE = 16_000
+FRAME = 512
+MODEL = Path(__file__).parent / "assets" / "silero_vad_v6.2.1.onnx"
+
+
+class Silero:
+    """Silero v6 with recovery from its "deaf" state.
+
+    On a long stream the recurrent state can drift into an attractor where the
+    probability stays near 0 even through loud speech (low babble, near-silent
+    pauses). After ``restore_s`` of p < 0.1 the state is put back to where the
+    model last heard confident speech (p >= 0.9), or zeroed before any. A zero
+    state is the worse restart point: it scores the next utterance lower and
+    later. Costs no extra inference; 0 disables it.
+    """
+
+    def __init__(self, model: str | Path = MODEL, threads: int = 1, restore_s: float = 1.0) -> None:
+        import onnxruntime as ort
+
+        options = ort.SessionOptions()
+        options.intra_op_num_threads = threads
+        options.inter_op_num_threads = 1
+        options.log_severity_level = 3
+        self._session = ort.InferenceSession(
+            str(model), sess_options=options, providers=["CPUExecutionProvider"]
+        )
+        self._sr = np.asarray(SAMPLE_RATE, dtype=np.int64)
+        self._restore_frames = math.ceil(restore_s * SAMPLE_RATE / FRAME - 1e-9) if restore_s > 0 else 0
+        self.restores = 0
+        self.reset()
+
+    def reset(self) -> None:
+        """Start a new stream: zero state and context, forget the speech snapshot."""
+        self._context = np.zeros((1, 64), dtype=np.float32)
+        self._state = np.zeros((2, 1, 128), dtype=np.float32)
+        self._snapshot = None
+        self._quiet = 0
+
+    def __call__(self, frame: np.ndarray) -> float:
+        audio = np.asarray(frame, dtype=np.float32).reshape(1, FRAME)
+        p, self._state = self._session.run(
+            None,
+            {"input": np.concatenate((self._context, audio), axis=1), "sr": self._sr, "state": self._state},
+        )
+        self._context = audio[:, -64:].copy()
+        p = float(np.asarray(p).reshape(-1)[0])
+        if self._restore_frames:  # standalone rule; inside VAD, VAD decides both (see vad.py)
+            if p >= 0.9:
+                self.snapshot()
+            self._quiet = self._quiet + 1 if p < 0.1 else 0
+            if self._quiet >= self._restore_frames:
+                self.restore()
+        return p
+
+    def snapshot(self) -> None:
+        """Remember the current state as the restore point (fresh arrays each frame)."""
+        self._snapshot = (self._state, self._context)
+
+    def restore(self) -> None:
+        """Put the state back to the last confident speech (zeros before any); affects the next frame."""
+        self._quiet = 0
+        self.restores += 1
+        if self._snapshot is None:
+            self._context = np.zeros((1, 64), dtype=np.float32)
+            self._state = np.zeros((2, 1, 128), dtype=np.float32)
+        else:
+            self._state, self._context = (a.copy() for a in self._snapshot)