streaming-vits 0.1.0__tar.gz

streaming_vits-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Abrar Ahmed
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

streaming_vits-0.1.0/PKG-INFO

@@ -0,0 +1,266 @@
+Metadata-Version: 2.4
+Name: streaming-vits
+Version: 0.1.0
+Summary: Streaming inference for VITS/piper TTS — a drop-in replacement for sherpa-onnx-offline-tts that starts playing before synthesis finishes
+License: MIT
+Keywords: tts,vits,piper,streaming,onnx,sherpa-onnx,speech
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Multimedia :: Sound/Audio :: Speech
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.21
+Requires-Dist: onnx>=1.14
+Requires-Dist: onnxruntime>=1.15
+Provides-Extra: play
+Requires-Dist: sounddevice>=0.4; extra == "play"
+Provides-Extra: fast-phonemes
+Requires-Dist: piper-phonemize>=1.1; extra == "fast-phonemes"
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Dynamic: license-file
+
+# streaming-vits
+
+Streaming inference for VITS / [piper](https://github.com/rhasspy/piper) TTS models.
+
+Feed in a whole paragraph, get audio out incrementally — and the result is
+**numerically identical** to a full-paragraph forward pass. No sentence
+splitting, no prosody loss.
+
+A drop-in replacement for `sherpa-onnx-offline-tts`: same model file, same flags,
+same WAV. It just starts producing audio before synthesis finishes.
+
+```diff
+- sherpa-onnx-offline-tts \
++ streaming-vits-offline-tts \
+      --vits-model=./en_US-libritts_r-medium.onnx \
+      --vits-tokens=./tokens.txt \
+      --vits-data-dir=./espeak-ng-data \
+      --num-threads=4 --sid=0 \
+      --output-filename=./test-0.wav \
+      "The quick brown fox jumped skillfully over the lazy dog..."
+```
+
+```
+Elapsed seconds: 1.012
+Audio duration: 9.532 s
+Real-time factor (RTF): 1.012/9.532 = 0.106
+Time to first audio: 141 ms          <- 9.5 s of audio, first sound in 141 ms
+```
+
+Add `--play` to hear it live as it generates.
+
+## Why this is possible
+
+VITS is not autoregressive, so streaming looks like it shouldn't work. But
+`SynthesizerTrn.infer()` splits cleanly at the monotonic alignment:
+
+| half | what it does | context needed | share of compute |
+|---|---|---|---|
+| **frontend** | text encoder → duration predictor → alignment → prior expansion → `z_p` | global, over text | ~2% |
+| **decoder** | flow → HiFi-GAN | **finite receptive field over frames** | ~98% |
+
+Everything that carries paragraph-level prosody lives in the cheap frontend,
+which runs once on the whole input. The expensive half is a plain CNN over the
+frame axis — no recurrence, no global attention — so it can be evaluated in
+chunks, and with enough overlap context the chunks concatenate *exactly*.
+
+This is the opposite trade-off from sentence chunking. Splitting text throws away
+the global context that produces good prosody. Splitting in *frame* space costs
+nothing, because the frame-space network cannot see beyond its receptive field
+anyway.
+
+One detail makes it exact rather than approximate: both `RandomNormalLike` nodes
+(duration noise and prior noise) land in the frontend, so all sampling happens
+once, for every frame, before any chunk is decoded. Two overlapping chunk decodes
+necessarily see identical noise on the frames they share.
+
+## Install
+
+```bash
+pip install streaming-vits
+
+# phonemization backend (one of):
+brew install espeak-ng          # macOS
+apt install espeak-ng           # debian/ubuntu
+pip install piper-phonemize     # faster, no subprocess
+
+pip install 'streaming-vits[play]'   # optional: live playback
+```
+
+Point it at any piper voice — the same tarballs sherpa-onnx uses:
+
+```bash
+wget https://github.com/k2-fsa/sherpa-onnx/releases/download/tts-models/vits-piper-en_US-libritts_r-medium.tar.bz2
+tar xf vits-piper-en_US-libritts_r-medium.tar.bz2
+```
+
+The monolithic `.onnx` is split into a frontend and a decoder graph on first use
+and cached next to the model. You don't have to do anything.
+
+## Calibrate for your device
+
+The chunk schedule is the whole game for latency, and the right one depends on
+how fast the device is. Run this once per device:
+
+```bash
+streaming-vits calibrate \
+    --vits-model=./en_US-libritts_r-medium.onnx \
+    --vits-tokens=./tokens.txt \
+    --vits-data-dir=./espeak-ng-data \
+    --num-threads=4
+```
+
+It measures two different things:
+
+**Margins** are a property of the model's weights — how much context does the
+decoder actually need before its output stops changing? Swept empirically,
+because it is not guessable: trained weights use far more of their nominal
+receptive field than an untrained network suggests.
+
+```
+[1/3] receptive field (model property, device independent)
+  seed 1234 -- deterministic across runs
+  lookahead  rel RMS err
+          4     32.835%
+          8     11.672%
+         12      4.197%
+         16      1.047%
+         20      0.174%
+         24      0.006%
+  -> left 34, right 28 (325 ms lookahead)
+```
+
+The frontend sampling is pinned to a fixed seed for this sweep (`--seed`), so
+repeated calibrations of the same model give the same margins. Normal synthesis
+is unaffected and stays stochastic.
+
+**Schedule** is a property of the device. An affine cost model
+`decode(n) = α + β·(n + margins)` is fitted, then the schedule that reaches the
+speaker soonest without ever starving playback is solved for:
+
+```
+[2/3] decode cost on this device
+  fit: 1.26 ms/call + 0.442 ms/frame (a frame is 11.61 ms of audio)
+
+  predicted time to first audio  162 ms
+  steady-state RTF per chunk     0.052
+  survives a device              2.0x slower than this one
+  steady-state decode overhead   1.21x (the ramp costs more early)
+```
+
+`--safety N` sets how much slower than measured to plan for; the prebuffer is
+sized so headroom is a guarantee rather than an observation. Raise it on devices
+with contended CPUs.
+
+Margins are reproducible run to run, but the *schedule* is not quite: it depends
+on wall-clock timings, which move with machine load. Calibrate on an otherwise
+idle device, and treat `--safety` as the thing that absorbs the rest.
+
+The profile is written next to the model and picked up automatically.
+
+## Results
+
+`en_US-libritts_r-medium`, 4 threads, M-series Mac, 15.2 s utterance:
+
+```
+non-streaming  0.701s  RTF 0.046   <- you wait this long before any sound
+
+time to first audio  0.163s   (4.3x sooner than non-streaming)
+total wall clock     1.301s  RTF 0.085  (1.86x the work)
+playback             no underrun
+```
+
+Time-to-first-audio for the non-streaming path scales with the length of the
+paragraph. For the streamed path it is constant, so the gap widens the more you
+ask it to say.
+
+Equivalence, on the real weights:
+
+```
+$ streaming-vits verify --vits-model=en_US-libritts_r-medium.onnx ...
+  max |streamed - full| = 8.00e-05  OK
+
+$ streaming-vits verify --vits-model=es_MX-claude-high.onnx ...
+  max |streamed - full| = 1.79e-05  OK
+```
+
+## Commands
+
+| command | what it does |
+|---|---|
+| `streaming-vits-offline-tts …` | drop-in for `sherpa-onnx-offline-tts` |
+| `streaming-vits speak … --play` | synthesise, optionally play live |
+| `streaming-vits calibrate …` | benchmark this device, write a profile |
+| `streaming-vits bench …` | streamed vs non-streaming latency |
+| `streaming-vits verify …` | check chunked output still equals a full decode |
+| `streaming-vits info …` | show split graphs and active profile |
+| `streaming-vits clear-cache …` | delete cached split graphs |
+
+Accepted-and-ignored sherpa flags (`--vits-lexicon`, `--vits-dict-dir`,
+`--tts-rule-fsts`, `--max-num-sentences`) print a note rather than failing, so
+existing command lines keep working.
+
+## Python API
+
+```python
+from streaming_vits import StreamingTTS
+
+tts = StreamingTTS("en_US-libritts_r-medium.onnx", tokens_path="tokens.txt")
+
+for chunk in tts.stream("Hello there. This plays before it has finished."):
+    speaker.write(chunk.audio)      # float32, mono, tts.sample_rate
+    print(chunk.index, chunk.frames, chunk.ready_at)
+
+audio = tts.synthesize("Same thing, one array.")
+```
+
+## Limits
+
+- **Extra work.** Each chunk re-decodes its margin frames: ~1.2× in steady
+  state, more during the ramp. Cached-state streaming convolutions would cut
+  this to ~1.05× but are considerably more implementation.
+- **TTFA is dominated by the frontend**, not by chunk size. On the numbers above
+  the frontend is ~130 ms of a 162 ms budget; smaller first chunks will not help.
+  Speeding up the stochastic duration predictor is the next lever.
+- **Streaming text *in* is a separate problem.** The duration predictor needs the
+  whole utterance before any audio exists. If text arrives from an LLM token
+  stream you still chunk at clause level — but you can feed the previous clause
+  as context and discard its audio to soften the seam.
+- **The bundled phonemizer is pragmatic.** It shells out to `espeak-ng` per
+  sentence and re-inserts punctuation, where piper drives libespeak-ng directly
+  and gets clause terminators back. Install `piper-phonemize` for the faithful
+  path. This does not affect the streaming claim — `verify` compares on identical
+  phoneme ids.
+- Verified on two piper voices: `en_US-libritts_r-medium` (904 speakers) and
+  `es_MX-claude-high` (single speaker, different exporter version). Other VITS
+  exports should work if the graph has the same seam; `streaming-vits info` will
+  tell you before you rely on it.
+
+## How the split is found
+
+`streaming_vits/graph.py` locates the seam structurally rather than by hardcoded
+node names: find the last `RandomNormalLike` (the prior noise), walk forward to
+the `Add` that forms `z_p`, then collect whatever the downstream subgraph still
+needs from upstream. It also reads the decoder config back off the weights —
+piper's `config.json` does not record it, and it varies by voice
+(`libritts_r-medium` uses 3 upsample stages, `[8,8,4]`, not the 4 in the
+reference VITS config).
+
+Two things are deliberately not read from node names, because those vary between
+piper exporter versions even when the weights do not: the upsample config comes
+from `dec.ups.N.weight`, and the seam comes from graph topology. Single-speaker
+voices have no speaker embedding and no `sid` input, so the decoder takes two
+inputs instead of three; that is detected rather than configured.
+
+`research/` holds the original proof of concept, including `verify_streaming.py`,
+which proves the same equivalence against the reference PyTorch VITS
+implementation independently of ONNX.
+
+## License
+
+MIT

streaming_vits-0.1.0/README.md

@@ -0,0 +1,241 @@
+# streaming-vits
+
+Streaming inference for VITS / [piper](https://github.com/rhasspy/piper) TTS models.
+
+Feed in a whole paragraph, get audio out incrementally — and the result is
+**numerically identical** to a full-paragraph forward pass. No sentence
+splitting, no prosody loss.
+
+A drop-in replacement for `sherpa-onnx-offline-tts`: same model file, same flags,
+same WAV. It just starts producing audio before synthesis finishes.
+
+```diff
+- sherpa-onnx-offline-tts \
++ streaming-vits-offline-tts \
+      --vits-model=./en_US-libritts_r-medium.onnx \
+      --vits-tokens=./tokens.txt \
+      --vits-data-dir=./espeak-ng-data \
+      --num-threads=4 --sid=0 \
+      --output-filename=./test-0.wav \
+      "The quick brown fox jumped skillfully over the lazy dog..."
+```
+
+```
+Elapsed seconds: 1.012
+Audio duration: 9.532 s
+Real-time factor (RTF): 1.012/9.532 = 0.106
+Time to first audio: 141 ms          <- 9.5 s of audio, first sound in 141 ms
+```
+
+Add `--play` to hear it live as it generates.
+
+## Why this is possible
+
+VITS is not autoregressive, so streaming looks like it shouldn't work. But
+`SynthesizerTrn.infer()` splits cleanly at the monotonic alignment:
+
+| half | what it does | context needed | share of compute |
+|---|---|---|---|
+| **frontend** | text encoder → duration predictor → alignment → prior expansion → `z_p` | global, over text | ~2% |
+| **decoder** | flow → HiFi-GAN | **finite receptive field over frames** | ~98% |
+
+Everything that carries paragraph-level prosody lives in the cheap frontend,
+which runs once on the whole input. The expensive half is a plain CNN over the
+frame axis — no recurrence, no global attention — so it can be evaluated in
+chunks, and with enough overlap context the chunks concatenate *exactly*.
+
+This is the opposite trade-off from sentence chunking. Splitting text throws away
+the global context that produces good prosody. Splitting in *frame* space costs
+nothing, because the frame-space network cannot see beyond its receptive field
+anyway.
+
+One detail makes it exact rather than approximate: both `RandomNormalLike` nodes
+(duration noise and prior noise) land in the frontend, so all sampling happens
+once, for every frame, before any chunk is decoded. Two overlapping chunk decodes
+necessarily see identical noise on the frames they share.
+
+## Install
+
+```bash
+pip install streaming-vits
+
+# phonemization backend (one of):
+brew install espeak-ng          # macOS
+apt install espeak-ng           # debian/ubuntu
+pip install piper-phonemize     # faster, no subprocess
+
+pip install 'streaming-vits[play]'   # optional: live playback
+```
+
+Point it at any piper voice — the same tarballs sherpa-onnx uses:
+
+```bash
+wget https://github.com/k2-fsa/sherpa-onnx/releases/download/tts-models/vits-piper-en_US-libritts_r-medium.tar.bz2
+tar xf vits-piper-en_US-libritts_r-medium.tar.bz2
+```
+
+The monolithic `.onnx` is split into a frontend and a decoder graph on first use
+and cached next to the model. You don't have to do anything.
+
+## Calibrate for your device
+
+The chunk schedule is the whole game for latency, and the right one depends on
+how fast the device is. Run this once per device:
+
+```bash
+streaming-vits calibrate \
+    --vits-model=./en_US-libritts_r-medium.onnx \
+    --vits-tokens=./tokens.txt \
+    --vits-data-dir=./espeak-ng-data \
+    --num-threads=4
+```
+
+It measures two different things:
+
+**Margins** are a property of the model's weights — how much context does the
+decoder actually need before its output stops changing? Swept empirically,
+because it is not guessable: trained weights use far more of their nominal
+receptive field than an untrained network suggests.
+
+```
+[1/3] receptive field (model property, device independent)
+  seed 1234 -- deterministic across runs
+  lookahead  rel RMS err
+          4     32.835%
+          8     11.672%
+         12      4.197%
+         16      1.047%
+         20      0.174%
+         24      0.006%
+  -> left 34, right 28 (325 ms lookahead)
+```
+
+The frontend sampling is pinned to a fixed seed for this sweep (`--seed`), so
+repeated calibrations of the same model give the same margins. Normal synthesis
+is unaffected and stays stochastic.
+
+**Schedule** is a property of the device. An affine cost model
+`decode(n) = α + β·(n + margins)` is fitted, then the schedule that reaches the
+speaker soonest without ever starving playback is solved for:
+
+```
+[2/3] decode cost on this device
+  fit: 1.26 ms/call + 0.442 ms/frame (a frame is 11.61 ms of audio)
+
+  predicted time to first audio  162 ms
+  steady-state RTF per chunk     0.052
+  survives a device              2.0x slower than this one
+  steady-state decode overhead   1.21x (the ramp costs more early)
+```
+
+`--safety N` sets how much slower than measured to plan for; the prebuffer is
+sized so headroom is a guarantee rather than an observation. Raise it on devices
+with contended CPUs.
+
+Margins are reproducible run to run, but the *schedule* is not quite: it depends
+on wall-clock timings, which move with machine load. Calibrate on an otherwise
+idle device, and treat `--safety` as the thing that absorbs the rest.
+
+The profile is written next to the model and picked up automatically.
+
+## Results
+
+`en_US-libritts_r-medium`, 4 threads, M-series Mac, 15.2 s utterance:
+
+```
+non-streaming  0.701s  RTF 0.046   <- you wait this long before any sound
+
+time to first audio  0.163s   (4.3x sooner than non-streaming)
+total wall clock     1.301s  RTF 0.085  (1.86x the work)
+playback             no underrun
+```
+
+Time-to-first-audio for the non-streaming path scales with the length of the
+paragraph. For the streamed path it is constant, so the gap widens the more you
+ask it to say.
+
+Equivalence, on the real weights:
+
+```
+$ streaming-vits verify --vits-model=en_US-libritts_r-medium.onnx ...
+  max |streamed - full| = 8.00e-05  OK
+
+$ streaming-vits verify --vits-model=es_MX-claude-high.onnx ...
+  max |streamed - full| = 1.79e-05  OK
+```
+
+## Commands
+
+| command | what it does |
+|---|---|
+| `streaming-vits-offline-tts …` | drop-in for `sherpa-onnx-offline-tts` |
+| `streaming-vits speak … --play` | synthesise, optionally play live |
+| `streaming-vits calibrate …` | benchmark this device, write a profile |
+| `streaming-vits bench …` | streamed vs non-streaming latency |
+| `streaming-vits verify …` | check chunked output still equals a full decode |
+| `streaming-vits info …` | show split graphs and active profile |
+| `streaming-vits clear-cache …` | delete cached split graphs |
+
+Accepted-and-ignored sherpa flags (`--vits-lexicon`, `--vits-dict-dir`,
+`--tts-rule-fsts`, `--max-num-sentences`) print a note rather than failing, so
+existing command lines keep working.
+
+## Python API
+
+```python
+from streaming_vits import StreamingTTS
+
+tts = StreamingTTS("en_US-libritts_r-medium.onnx", tokens_path="tokens.txt")
+
+for chunk in tts.stream("Hello there. This plays before it has finished."):
+    speaker.write(chunk.audio)      # float32, mono, tts.sample_rate
+    print(chunk.index, chunk.frames, chunk.ready_at)
+
+audio = tts.synthesize("Same thing, one array.")
+```
+
+## Limits
+
+- **Extra work.** Each chunk re-decodes its margin frames: ~1.2× in steady
+  state, more during the ramp. Cached-state streaming convolutions would cut
+  this to ~1.05× but are considerably more implementation.
+- **TTFA is dominated by the frontend**, not by chunk size. On the numbers above
+  the frontend is ~130 ms of a 162 ms budget; smaller first chunks will not help.
+  Speeding up the stochastic duration predictor is the next lever.
+- **Streaming text *in* is a separate problem.** The duration predictor needs the
+  whole utterance before any audio exists. If text arrives from an LLM token
+  stream you still chunk at clause level — but you can feed the previous clause
+  as context and discard its audio to soften the seam.
+- **The bundled phonemizer is pragmatic.** It shells out to `espeak-ng` per
+  sentence and re-inserts punctuation, where piper drives libespeak-ng directly
+  and gets clause terminators back. Install `piper-phonemize` for the faithful
+  path. This does not affect the streaming claim — `verify` compares on identical
+  phoneme ids.
+- Verified on two piper voices: `en_US-libritts_r-medium` (904 speakers) and
+  `es_MX-claude-high` (single speaker, different exporter version). Other VITS
+  exports should work if the graph has the same seam; `streaming-vits info` will
+  tell you before you rely on it.
+
+## How the split is found
+
+`streaming_vits/graph.py` locates the seam structurally rather than by hardcoded
+node names: find the last `RandomNormalLike` (the prior noise), walk forward to
+the `Add` that forms `z_p`, then collect whatever the downstream subgraph still
+needs from upstream. It also reads the decoder config back off the weights —
+piper's `config.json` does not record it, and it varies by voice
+(`libritts_r-medium` uses 3 upsample stages, `[8,8,4]`, not the 4 in the
+reference VITS config).
+
+Two things are deliberately not read from node names, because those vary between
+piper exporter versions even when the weights do not: the upsample config comes
+from `dec.ups.N.weight`, and the seam comes from graph topology. Single-speaker
+voices have no speaker embedding and no `sid` input, so the decoder takes two
+inputs instead of three; that is detected rather than configured.
+
+`research/` holds the original proof of concept, including `verify_streaming.py`,
+which proves the same equivalence against the reference PyTorch VITS
+implementation independently of ONNX.
+
+## License
+
+MIT

streaming_vits-0.1.0/pyproject.toml

@@ -0,0 +1,39 @@
+[build-system]
+requires = ["setuptools>=64"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "streaming-vits"
+version = "0.1.0"
+description = "Streaming inference for VITS/piper TTS — a drop-in replacement for sherpa-onnx-offline-tts that starts playing before synthesis finishes"
+readme = "README.md"
+requires-python = ">=3.9"
+license = { text = "MIT" }
+keywords = ["tts", "vits", "piper", "streaming", "onnx", "sherpa-onnx", "speech"]
+classifiers = [
+  "Development Status :: 4 - Beta",
+  "Intended Audience :: Developers",
+  "License :: OSI Approved :: MIT License",
+  "Programming Language :: Python :: 3",
+  "Topic :: Multimedia :: Sound/Audio :: Speech",
+]
+dependencies = [
+  "numpy>=1.21",
+  "onnx>=1.14",
+  "onnxruntime>=1.15",
+]
+
+[project.optional-dependencies]
+play = ["sounddevice>=0.4"]
+fast-phonemes = ["piper-phonemize>=1.1"]
+dev = ["pytest>=7"]
+
+[project.scripts]
+streaming-vits = "streaming_vits.cli:main"
+streaming-vits-offline-tts = "streaming_vits.cli:main_offline_tts"
+
+[tool.setuptools.packages.find]
+include = ["streaming_vits*"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

streaming_vits-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

streaming_vits-0.1.0/streaming_vits/__init__.py

@@ -0,0 +1,17 @@
+"""Streaming inference for VITS / piper TTS models.
+
+Whole paragraph in, audio out incrementally, numerically identical to a
+full-paragraph forward pass -- no sentence splitting, no prosody loss.
+
+    from streaming_vits import StreamingTTS
+
+    tts = StreamingTTS("en_US-libritts_r-medium.onnx", tokens_path="tokens.txt")
+    for chunk in tts.stream("Hello there, this plays before it finishes."):
+        play(chunk.audio)
+"""
+
+from .engine import Chunk, StreamingTTS
+from .profile import Profile
+
+__version__ = "0.1.0"
+__all__ = ["StreamingTTS", "Chunk", "Profile", "__version__"]