@qvac/tts-ggml 0.9.2 → 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +317 -33
- package/docs/pocket-tts.md +201 -0
- package/examples/moss-dialogue-tts.js +113 -0
- package/examples/moss-tts.js +154 -0
- package/examples/pocket-tts.js +51 -0
- package/examples/supertonic-chunk-stream-tts.js +156 -0
- package/examples/supertonic-sentence-stream-tts.js +4 -4
- package/index.d.ts +352 -29
- package/index.js +698 -52
- package/lib/fit.d.ts +127 -0
- package/lib/fit.js +29 -0
- package/lib/pocketConfig.d.ts +10 -0
- package/lib/pocketConfig.js +145 -0
- package/package.json +14 -9
- package/test/integration/addon.test.js +5 -0
- package/test/integration/audio8.test.js +7 -0
- package/test/integration/chatterbox-mtl.test.js +6 -0
- package/test/integration/chatterbox-speed.test.js +13 -2
- package/test/integration/cosyvoice3-clone.test.js +16 -1
- package/test/integration/cosyvoice3-lavasr.test.js +6 -0
- package/test/integration/cosyvoice3.test.js +19 -0
- package/test/integration/fit.test.js +174 -0
- package/test/integration/gpu-smoke.test.js +13 -0
- package/test/integration/lavasr-enhancer.test.js +104 -3
- package/test/integration/moss.test.js +282 -0
- package/test/integration/multiple-runs.test.js +18 -2
- package/test/integration/output-sample-rate.test.js +6 -0
- package/test/integration/parler-wer.test.js +7 -1
- package/test/integration/parler.test.js +27 -3
- package/test/integration/pocket.integration.test.js +156 -0
- package/test/integration/supertonic-mtl.test.js +2 -0
- package/test/integration/supertonic.test.js +70 -0
- package/test/integration/supertonic3-quant.test.js +2 -0
- package/test/integration/voice-controls.test.js +56 -0
- package/test/mobile/integration.auto.cjs +16 -1
- package/test/mobile/pocket-worklet-host.cpp +92 -0
- package/test/mobile/pocket-worklet.cjs +47 -0
- package/test/mobile/test-groups.json +8 -4
- package/test/utils/runCosyvoiceTTS.js +1 -0
- package/test/utils/runTTS.js +6 -11
- package/tts.d.ts +13 -2
- package/tts.js +3 -2
package/README.md
CHANGED
|
@@ -7,8 +7,10 @@ v1/v2 still loadable), **Parler** (mini/large English + indic 21-language,
|
|
|
7
7
|
description-conditioned with voice/emotion templates), **CosyVoice3**
|
|
8
8
|
(Fun-CosyVoice3-0.5B, instruct-conditioned, 24 kHz, CPU with opt-in
|
|
9
9
|
Metal on Apple, CUDA/Vulkan on Linux, Vulkan on Windows, and OpenCL/Adreno
|
|
10
|
-
GPU offload on Android),
|
|
11
|
-
(DualAR + neural codec, in-process voice cloning, desktop GPU),
|
|
10
|
+
GPU offload on Android), **Audio8**
|
|
11
|
+
(DualAR + neural codec, in-process voice cloning, desktop GPU), and **MOSS**
|
|
12
|
+
(MOSS-TTS Delay, 24 kHz, in-process voice cloning, native chunk streaming,
|
|
13
|
+
desktop), plus optional
|
|
12
14
|
LavaSR neural denoise + 48 kHz bandwidth-extension enhancement. Unsure
|
|
13
15
|
which checkpoint to stage? Start with [Choosing a model](#choosing-a-model).
|
|
14
16
|
|
|
@@ -24,7 +26,8 @@ backend per its own per-vendor allowlist (Adreno → OpenCL,
|
|
|
24
26
|
Xclipse/Mali → Vulkan). Parler supports Apple/Metal, linux CUDA, and the
|
|
25
27
|
validated Android paths, including Vulkan on ARM Mali (see
|
|
26
28
|
[Backends & GPU acceleration](#backends--gpu-acceleration)). Audio8 supports
|
|
27
|
-
CUDA/Vulkan offload on Linux and Vulkan on Windows.
|
|
29
|
+
CUDA/Vulkan offload on Linux and Vulkan on Windows. MOSS GPU offload is
|
|
30
|
+
validated on Apple/Metal.
|
|
28
31
|
|
|
29
32
|
[qvac-tts-cpp]: https://github.com/tetherto/qvac-fabric-speech.cpp/tree/master/engines/tts
|
|
30
33
|
|
|
@@ -44,6 +47,9 @@ CUDA/Vulkan offload on Linux and Vulkan on Windows.
|
|
|
44
47
|
`config.useGPU: true` on GPU-capable hosts — including Android, where
|
|
45
48
|
`tts-cpp` selects the GPU backend per its per-vendor allowlist (see
|
|
46
49
|
[Backends & GPU acceleration](#backends--gpu-acceleration)).
|
|
50
|
+
- **Apple Core ML sidecars** for the Supertonic vocoder and the Audio8 codec
|
|
51
|
+
on macOS / iOS, picked up when staged next to the model (see
|
|
52
|
+
[Core ML sidecars on Apple](#core-ml-sidecars-on-apple)).
|
|
47
53
|
- **Dynamic backend loading on Android** — per-arch CPU + Vulkan +
|
|
48
54
|
OpenCL `.so` files ship under `prebuilds/<bare-target>/qvac__tts-ggml/`
|
|
49
55
|
and are picked up at runtime via the new `backendsDir` option (see
|
|
@@ -66,6 +72,8 @@ obvious for each job.
|
|
|
66
72
|
| English + voice cloning + low first-audio latency | `chatterbox-t3-turbo.gguf` + `chatterbox-s3gen.gguf` | Reference-wav / voice-profile cloning; native `streamChunkTokens` chunk streaming. Native 24 kHz. |
|
|
67
73
|
| Multilingual + voice cloning (EU / CJK) | `chatterbox-t3-mtl.gguf` + `chatterbox-s3gen-mtl.gguf` | en/es/fr/de/pt/it/zh/ja/ko/…; same cloning + streaming surface as Turbo. |
|
|
68
74
|
| Voice cloning at 44.1 kHz with nothing pre-baked | `audio8-lm-q8_0.gguf` + `audio8-codec-decoder-q8_0.gguf` (+ `audio8-codec-encoder-q8_0.gguf` to clone) | Clones from a reference wav **and its transcript**, encoded in-process — no enrolment step, no voice profile. Whole-utterance only (no native chunk streaming). |
|
|
75
|
+
| Multilingual speech with native chunk streaming and cloning on a desktop | `moss-tts-delay-f16.gguf` + `moss-codec-decoder-f16.gguf` (+ `moss-codec-encoder-f16.gguf` to clone) | 8B backbone, desktop only. Clones from a reference wav encoded in-process (no transcript needed); native `streamChunkTokens` chunk streaming. Native 24 kHz. |
|
|
76
|
+
| Multi-speaker dialogue that continues each speaker's reference voice | `moss-ttsd-f16.gguf` + `moss-codec-decoder-f16.gguf` + `moss-codec-encoder-f16.gguf` | 8B backbone, desktop only. One 24 kHz reference per speaker; the text opens with their transcripts. Native 24 kHz. |
|
|
69
77
|
| Indic languages | `parler-indic-q8_0.gguf` | 21 Indic languages; voice / emotion templates. Emotion is officially tested on 10 languages — see [Parler descriptions & emotions](#parler-descriptions--emotions) (or the upstream model card). Native 44.1 kHz. |
|
|
70
78
|
| Chinese dialects (Cantonese, Sichuan, Shanghai, …) | CosyVoice3 dir (`cosyvoice3-llm-*.gguf` + flow / hift / `voice.gguf`) | Instruct-conditioned; 17 dialects via `instruct: { dialect: '…' }`. CPU, with opt-in GPU offload (Metal on Apple, Vulkan on Linux/Windows, OpenCL/Adreno on Android); native 24 kHz. |
|
|
71
79
|
| Zero-shot / cross-lingual cloning (multilingual) | CosyVoice3 dir + `cosyvoice3-s3tok-*.gguf` + `cosyvoice3-campplus-*.gguf` | Clones from a reference wav; its transcript (`promptText`) selects zero-shot, omitting it selects cross-lingual (timbre only, any target language). Composes with `instruct`. |
|
|
@@ -88,6 +96,8 @@ here, so this guide does not go stale when backends change.
|
|
|
88
96
|
| `parler-mini-v1-q8_0` | English | ~1.2 GB | 44.1 kHz | Description / templates | Sentence streaming |
|
|
89
97
|
| CosyVoice3 (`cosyvoice3/`) | Instruct-led (strong on Chinese + dialects) | ~2.3 GB dir (+ ~300 MB to clone) | 24 kHz | Reference wav (zero-shot / cross-lingual) + `instruct` (dialect / emotion / speed / volume / style) | Native chunk opts |
|
|
90
98
|
| `audio8-lm-q8_0` + codec halves | Text-led (no `language` option) | ~0.7 GB (+ ~120 MB to clone) | 44.1 kHz | Reference wav + transcript, per call too | Sentence streaming |
|
|
99
|
+
| `moss-tts-delay-f16` + codec halves | Multilingual (`language` hint) | ~18.8 GB (+ ~1.8 GB to clone) | 24 kHz | Reference wav, fixed per instance | Sentence + native chunk |
|
|
100
|
+
| `moss-ttsd-f16` + codec halves | Multilingual (`language` hint) | ~20.5 GB | 24 kHz | One reference wav per speaker, fixed per instance | Native chunk |
|
|
91
101
|
|
|
92
102
|
### Legacy Supertonic v1 / v2
|
|
93
103
|
|
|
@@ -162,8 +172,8 @@ dependency, pinned to the exact `@qvac/tts-ggml` version:
|
|
|
162
172
|
|
|
163
173
|
## Model files
|
|
164
174
|
|
|
165
|
-
|
|
166
|
-
CosyVoice3, Audio8), each with its own GGUF layout under `models/`:
|
|
175
|
+
Six engine families are wrapped (Chatterbox, Supertonic, Parler,
|
|
176
|
+
CosyVoice3, Audio8, MOSS), each with its own GGUF layout under `models/`:
|
|
167
177
|
|
|
168
178
|
```
|
|
169
179
|
# Chatterbox turbo (English)
|
|
@@ -177,6 +187,9 @@ chatterbox-s3gen-mtl.gguf (~1.0 GB)
|
|
|
177
187
|
# Supertonic 3 (Supertone/supertonic-3; 31 languages) — preferred Supertonic
|
|
178
188
|
# checkpoint; published per quant tier (auto-detected from modelDir)
|
|
179
189
|
supertonic3-q4_0.gguf (~80 MB; also -q8_0 ~126 MB / -f16 ~191 MB / -f32)
|
|
190
|
+
supertonic3-vocoder.mlmodelc/ (optional, macOS / iOS: Apple Core ML sidecar for
|
|
191
|
+
the vocoder, used by the 8-bit and wider tiers; see
|
|
192
|
+
Core ML sidecars on Apple)
|
|
180
193
|
|
|
181
194
|
# Legacy Supertonic (not recommended for new integrations — see Choosing a model)
|
|
182
195
|
supertonic.gguf (~263 MB) — v1 English only
|
|
@@ -195,8 +208,9 @@ parler-indic-q8_0.gguf (~1.3 GB; also -f16 / -f32; 21 Indic languages)
|
|
|
195
208
|
# from the cosyvoice3-llm-*.gguf file
|
|
196
209
|
cosyvoice3/
|
|
197
210
|
cosyvoice3-llm-*.gguf (~973 MB q8_0 — Qwen2.5 speech LM)
|
|
198
|
-
cosyvoice3-flow-*.gguf (~1.3 GB f32 — DiT conditional-flow-matching
|
|
199
|
-
|
|
211
|
+
cosyvoice3-flow-*.gguf (~1.3 GB f32 — DiT conditional-flow-matching; also
|
|
212
|
+
-f16 / -bf16 / -q8_0 / -q4_0)
|
|
213
|
+
cosyvoice3-hift-*.gguf (~83 MB f32 — CausalHiFT vocoder; also -f16)
|
|
200
214
|
voice.gguf (baked default voice: timbre + prompt tensors)
|
|
201
215
|
vocab.json merges.txt (Qwen2 BPE tokenizer)
|
|
202
216
|
cosyvoice3-s3tok-*.gguf (~275 MB q8_0 / ~497 MB f16 — speech tokenizer;
|
|
@@ -208,6 +222,17 @@ cosyvoice3/
|
|
|
208
222
|
audio8-lm-q8_0.gguf (~0.6 GB; also -f16 / -q4_0 / -f32)
|
|
209
223
|
audio8-codec-decoder-q8_0.gguf (~110 MB; also -f16 / -f32)
|
|
210
224
|
audio8-codec-encoder-q8_0.gguf (~120 MB; also -f16 / -f32; cloning only)
|
|
225
|
+
audio8-codec-decoder.mlmodelc/ (optional, macOS / iOS: Apple Core ML sidecar for
|
|
226
|
+
the codec synthesis stack, picked up when it sits
|
|
227
|
+
next to the decoder GGUF; one export serves every
|
|
228
|
+
quant tier)
|
|
229
|
+
|
|
230
|
+
# MOSS (OpenMOSS MOSS-TTS v1.5 Delay; 24 kHz, 8B backbone + RVQ codec); the
|
|
231
|
+
# encoder is only needed to clone a voice or run a dialogue
|
|
232
|
+
moss-tts-delay-f16.gguf (~17 GB)
|
|
233
|
+
moss-ttsd-f16.gguf (~17 GB; MOSS-TTSD dialogue backbone, optional)
|
|
234
|
+
moss-codec-decoder-f16.gguf (~1.8 GB)
|
|
235
|
+
moss-codec-encoder-f16.gguf (~1.8 GB; cloning and dialogue only)
|
|
211
236
|
```
|
|
212
237
|
|
|
213
238
|
Download the registry-published Chatterbox, Supertonic, and Parler models into
|
|
@@ -219,10 +244,15 @@ npm run download-models:registry -- --group chatterbox,supertonic3
|
|
|
219
244
|
npm run download-models:registry -- --output /path/to/models
|
|
220
245
|
```
|
|
221
246
|
|
|
222
|
-
CosyVoice3 and
|
|
223
|
-
Stage their layouts shown above from local converted artifacts.
|
|
224
|
-
|
|
225
|
-
|
|
247
|
+
CosyVoice3, Audio8 and MOSS are not currently included in that registry command.
|
|
248
|
+
Stage their layouts shown above from local converted artifacts. CosyVoice3
|
|
249
|
+
resolves each component by filename prefix and does not rank quantizations, so
|
|
250
|
+
stage one file per component or name it explicitly with
|
|
251
|
+
`files.cosyvoiceFlowModelPath` / `cosyvoiceHiftModelPath`. Which flow tier is
|
|
252
|
+
fastest depends on the backend — `q8_0` on Vulkan / CUDA, `f16` on Metal,
|
|
253
|
+
`bf16` on AVX512-BF16 CPUs and `f16` on other CPUs — and the `f16` HiFT is the
|
|
254
|
+
recommendation everywhere. The package converts upstream Chatterbox,
|
|
255
|
+
Supertonic, and Parler checkpoints via a Python venv pipeline:
|
|
226
256
|
|
|
227
257
|
```bash
|
|
228
258
|
npm run setup-models # creates ./venv, installs requirements.txt, runs convert-models.sh
|
|
@@ -238,7 +268,9 @@ npm run convert-models
|
|
|
238
268
|
The Audio8 GGUFs are produced by the converters in
|
|
239
269
|
qvac-fabric-speech.cpp (`engines/tts/scripts/convert-audio8-lm-to-gguf.py`
|
|
240
270
|
and `convert-audio8-codec-to-gguf.py`) until they are published to the model
|
|
241
|
-
registry alongside the other engines.
|
|
271
|
+
registry alongside the other engines. The MOSS GGUFs come from
|
|
272
|
+
`convert-moss-delay-to-gguf.py` and `convert-moss-codec-to-gguf.py` in the same
|
|
273
|
+
directory.
|
|
242
274
|
|
|
243
275
|
Point the addon at a custom location via `files.modelDir` (engine
|
|
244
276
|
auto-detected from the gguf filenames present), or pass explicit
|
|
@@ -247,7 +279,9 @@ auto-detected from the gguf filenames present), or pass explicit
|
|
|
247
279
|
`files.cosyvoiceModelDir` (CosyVoice3 — a directory, see
|
|
248
280
|
[CosyVoice3 instruct](#cosyvoice3-instruct)) /
|
|
249
281
|
`files.audio8Lm` + `files.audio8CodecDecoder` (+ `files.audio8CodecEncoder`
|
|
250
|
-
to clone) (Audio8)
|
|
282
|
+
to clone) (Audio8) /
|
|
283
|
+
`files.mossBackbone` + `files.mossCodecDecoder` (+ `files.mossCodecEncoder`
|
|
284
|
+
to clone) (MOSS).
|
|
251
285
|
|
|
252
286
|
## Quick start
|
|
253
287
|
|
|
@@ -320,7 +354,8 @@ Full runnable demo (with streaming playback):
|
|
|
320
354
|
|
|
321
355
|
Use when you want the fastest possible first-audio-out **within a
|
|
322
356
|
single utterance**. The C++ engine splits each synthesis into chunks
|
|
323
|
-
of `streamChunkTokens` speech tokens (25 ≈ 1 s of audio
|
|
357
|
+
of `streamChunkTokens` speech tokens (25 ≈ 1 s of audio; on MOSS the unit
|
|
358
|
+
is a codec frame, so 25 ≈ 2 s) and emits
|
|
324
359
|
audio per chunk, keeping HiFT's source cache phase-continuous across
|
|
325
360
|
seams so the joins are inaudible.
|
|
326
361
|
|
|
@@ -345,6 +380,28 @@ await response.onUpdate(data => {
|
|
|
345
380
|
Full runnable demo (with gapless playback via `sox` or `ffplay`):
|
|
346
381
|
`bare examples/chatterbox-chunk-stream-tts.js`
|
|
347
382
|
|
|
383
|
+
Long utterances re-synthesize the whole emitted prefix for every chunk by
|
|
384
|
+
default; `streamLeftContextTokens` (typically 25-50) keeps only that many
|
|
385
|
+
tokens of left context per chunk, which bounds the per-chunk cost.
|
|
386
|
+
|
|
387
|
+
Supertonic streams the same way, but its chunks are measured in text tokens
|
|
388
|
+
(Unicode code points; about 50 for English, 25-30 for CJK). The engine snaps
|
|
389
|
+
each boundary to a sentence end, clause or space within
|
|
390
|
+
`streamChunkTolerancePct` of the target and never emits a chunk shorter than
|
|
391
|
+
`streamMinChunkTokens`. Supertonic native streaming cannot be combined with
|
|
392
|
+
the LavaSR enhancer or denoiser; use sentence-level streaming for enhanced
|
|
393
|
+
output.
|
|
394
|
+
|
|
395
|
+
```js
|
|
396
|
+
const model = new TTSGgml({
|
|
397
|
+
files: { supertonicModel: './models/supertonic3-q8_0.gguf' },
|
|
398
|
+
streamChunkTokens: 50, // text tokens per chunk
|
|
399
|
+
streamFirstChunkTokens: 20 // smaller first chunk
|
|
400
|
+
})
|
|
401
|
+
```
|
|
402
|
+
|
|
403
|
+
Full runnable demo (with streaming playback): `bare examples/supertonic-chunk-stream-tts.js`
|
|
404
|
+
|
|
348
405
|
## Voice cloning
|
|
349
406
|
|
|
350
407
|
### Chatterbox
|
|
@@ -415,6 +472,14 @@ across calls skips the encoder. Per-call fields also ride on the
|
|
|
415
472
|
`runStream` / `runStreaming` options, pinned for the whole response so the
|
|
416
473
|
cache stays hot across chunks.
|
|
417
474
|
|
|
475
|
+
### MOSS
|
|
476
|
+
|
|
477
|
+
MOSS clones from `referenceAudio` alone, with no transcript. The recording
|
|
478
|
+
must be sampled at 24 kHz (there is no resampling; multichannel input is
|
|
479
|
+
downmixed), needs `files.mossCodecEncoder`, and is fixed for the instance.
|
|
480
|
+
For a multi-speaker dialogue, pass one recording per speaker in
|
|
481
|
+
`dialogueReferences` instead. See [MOSS](#moss).
|
|
482
|
+
|
|
418
483
|
### CosyVoice3
|
|
419
484
|
|
|
420
485
|
CosyVoice3 clones **zero-shot or cross-lingual**, selected by whether the
|
|
@@ -579,6 +644,10 @@ Notes:
|
|
|
579
644
|
- Denoiser + native chunk streaming (`streamChunkTokens > 0`) is rejected up
|
|
580
645
|
front for every engine — a stateful streaming denoiser is the follow-up. Use
|
|
581
646
|
batch synthesis, or drop the denoiser for streaming.
|
|
647
|
+
- The denoiser follows the engine's resolved device, like the enhancer: a GPU
|
|
648
|
+
engine runs its ggml graph on the GPU (falling back to the ggml CPU backend
|
|
649
|
+
when no GPU backend initialises), a CPU engine keeps the scalar core.
|
|
650
|
+
`denoiserBackendDevice` / `denoiserBackendId` in the stats report where it ran.
|
|
582
651
|
- The tts-cpp UL-UNAS forward is implemented in
|
|
583
652
|
[qvac-fabric-speech.cpp#78](https://github.com/tetherto/qvac-fabric-speech.cpp/pull/78)
|
|
584
653
|
(scalar CPU port, validated bit-close to the ONNX reference); it requires a
|
|
@@ -605,6 +674,46 @@ On hosts where more than one backend is usable, `TTS_CPP_GPU_BACKEND`
|
|
|
605
674
|
and fails loudly when that backend cannot be resolved; unset (or empty)
|
|
606
675
|
keeps the automatic preference above.
|
|
607
676
|
|
|
677
|
+
### Core ML sidecars on Apple
|
|
678
|
+
|
|
679
|
+
The macOS / iOS prebuilds carry the Apple Core ML (Neural Engine) sidecars
|
|
680
|
+
for the Supertonic vocoder and the Audio8 codec. They are presence-driven:
|
|
681
|
+
each stage runs on a compiled `.mlmodelc` found next to its model file
|
|
682
|
+
(`supertonic3-q8_0.gguf` -> `supertonic3-vocoder.mlmodelc`) and falls back to
|
|
683
|
+
the ggml graph when it is absent, so a model directory without sidecars
|
|
684
|
+
behaves exactly as before. The sidecar name drops the quantization suffix, so
|
|
685
|
+
one sidecar serves every tier of a model. Sidecars are not part of the
|
|
686
|
+
published model set yet; supply your own to opt in.
|
|
687
|
+
|
|
688
|
+
| Model | Sidecar next to the GGUF | Stage on Core ML | Runs on ggml instead | Force ggml |
|
|
689
|
+
| --- | --- | --- | --- | --- |
|
|
690
|
+
| Supertonic 1 / 2 / 3 | `<model>-vocoder.mlmodelc` | vocoder, in 64-latent-frame windows | GGUFs whose vocoder weights are stored below 8 bits (`q4_0`) | `SUPERTONIC_COREML_DISABLE=1` |
|
|
691
|
+
| Audio8 | `audio8-codec-decoder.mlmodelc`, beside the codec decoder GGUF | codec synthesis stack (upsampling + DAC decoder), in 64-post-frame windows; the language model stays on the ggml backend | a call that fails on the sidecar, which also retires it for every later call on that instance | `AUDIO8_COREML_DISABLE=1` |
|
|
692
|
+
| Chatterbox, Parler, CosyVoice3, MOSS, LavaSR | none | — | always | — |
|
|
693
|
+
|
|
694
|
+
Set the force-ggml variables in the process environment before `load()`.
|
|
695
|
+
Audio8 reports its codec path in `response.stats`: `codecSidecarLoaded` is 1
|
|
696
|
+
while the sidecar is attached and `codecOnCoreml` is 1 when that synthesis
|
|
697
|
+
ran its codec on it (see [Response shape](#response-shape)). The Supertonic
|
|
698
|
+
vocoder path is not reported in the stats.
|
|
699
|
+
|
|
700
|
+
Worth it where the GPU is consumer-class: on an Apple M4 the Supertonic
|
|
701
|
+
vocoder runs 1.6-2.9x faster on the Neural Engine than on Metal (1.06-1.13x
|
|
702
|
+
end to end). On workstation parts the GPU wins — an M3 Ultra is 0.6-0.9x —
|
|
703
|
+
so do not stage a sidecar there. `q4_0` models ignore the vocoder sidecar:
|
|
704
|
+
it carries full-precision weights and would substitute a different vocoder
|
|
705
|
+
rather than accelerate the quantized one.
|
|
706
|
+
|
|
707
|
+
Export the sidecars from the model GGUFs with
|
|
708
|
+
`engines/tts/scripts/export-supertonic-coreml.py` and
|
|
709
|
+
`engines/tts/scripts/export-audio8-codec-coreml.py` from the
|
|
710
|
+
[`qvac-fabric-speech.cpp`](https://github.com/tetherto/qvac-fabric-speech.cpp)
|
|
711
|
+
tree at the ref `speech-cpp` pins. Its
|
|
712
|
+
[Supertonic](https://github.com/tetherto/qvac-fabric-speech.cpp/blob/master/engines/tts/docs/supertonic.md#core-ml-vocoder-sidecar)
|
|
713
|
+
and
|
|
714
|
+
[Audio8](https://github.com/tetherto/qvac-fabric-speech.cpp/blob/master/engines/tts/docs/audio8.md#core-ml-codec-sidecar)
|
|
715
|
+
guides cover export, placement, and measurements.
|
|
716
|
+
|
|
608
717
|
When the addon is built with `ENABLE_CUDA` — on in the published linux-x64
|
|
609
718
|
prebuilds, opt-in on linux-arm64 and win32-x64 (`npm run build:cuda` or
|
|
610
719
|
`bare-make generate -D ENABLE_CUDA=ON`) — the CUDA backend ships as a
|
|
@@ -676,7 +785,9 @@ and they are set the same way: the constructor and `reload()` everywhere, plus
|
|
|
676
785
|
per call on the engines that can change them per call. The vocabulary is owned
|
|
677
786
|
by tts-cpp (`include/tts-cpp/voice_controls.h`) and mirrored here; each engine
|
|
678
787
|
declares the subset it supports, and an unsupported value throws naming that
|
|
679
|
-
engine's set.
|
|
788
|
+
engine's set. `TTSGgml.getVoiceControls()` returns tts-cpp's vocabulary and
|
|
789
|
+
each engine's subset straight from the native library, without loading a model
|
|
790
|
+
(`{ emotions, paces, engines: { parler: { emotions, paces }, … } }`).
|
|
680
791
|
|
|
681
792
|
| engine | `emotion` | `pace` | per call | exact rate knob |
|
|
682
793
|
|---|---|---|---|---|
|
|
@@ -685,6 +796,7 @@ engine's set.
|
|
|
685
796
|
| Supertonic | not supported | slow / moderate / fast | no | `speed` |
|
|
686
797
|
| Chatterbox | not supported | not supported | — | `speed` |
|
|
687
798
|
| Audio8 | not supported | not supported | — | — |
|
|
799
|
+
| MOSS | not supported | not supported | — | — |
|
|
688
800
|
|
|
689
801
|
The 12 canonical emotions (case-insensitive): `command`, `anger`, `narration`,
|
|
690
802
|
`conversation`, `disgust`, `fear`, `happy`, `neutral`, `proper noun`, `news`,
|
|
@@ -819,6 +931,89 @@ available, the engine falls back to CPU and sets
|
|
|
819
931
|
`response.stats.gpuUnsupported`. Voice cloning uses the same backend and adds
|
|
820
932
|
a one-off encode when a new reference recording is supplied.
|
|
821
933
|
|
|
934
|
+
## MOSS
|
|
935
|
+
|
|
936
|
+
MOSS Delay is an autoregressive model that predicts 32 RVQ codebooks per 80 ms
|
|
937
|
+
frame on a delay pattern, and a transformer codec turns those codes back into
|
|
938
|
+
24 kHz audio. The backbone and the codec's synthesis half run on every
|
|
939
|
+
synthesis, the analysis half only to clone a voice or run a dialogue, so a
|
|
940
|
+
text-only deployment can omit `files.mossCodecEncoder`. The backbones have 8B
|
|
941
|
+
parameters, so MOSS targets desktop hosts; mobile is not supported.
|
|
942
|
+
|
|
943
|
+
```js
|
|
944
|
+
const model = new TTSGgml({
|
|
945
|
+
engine: TTSGgml.ENGINE_MOSS,
|
|
946
|
+
files: { modelDir: './models' },
|
|
947
|
+
referenceAudio: './voice.wav', // optional; 24 kHz, needs moss-codec-encoder-*.gguf
|
|
948
|
+
streamChunkTokens: 25, // optional; codec frames per chunk (2 s)
|
|
949
|
+
config: { language: 'en' },
|
|
950
|
+
opts: { stats: true }
|
|
951
|
+
})
|
|
952
|
+
await model.load()
|
|
953
|
+
await model.run({ input: 'Hello from a fully on-device pipeline.' })
|
|
954
|
+
```
|
|
955
|
+
|
|
956
|
+
`config.language` is passed to the model as a prompt hint. The reference
|
|
957
|
+
recording is encoded once at load and stays fixed for the instance, so
|
|
958
|
+
changing voices means a new instance. `streamChunkTokens > 0` delivers audio in
|
|
959
|
+
chunks of that many codec frames (12.5 per second) while the backbone is still
|
|
960
|
+
generating; `streamFirstChunkTokens` is not supported. MOSS emits 24 kHz only,
|
|
961
|
+
so `config.outputSampleRate` is rejected unless it equals `24000`, and the
|
|
962
|
+
LavaSR enhancer / denoiser are not supported.
|
|
963
|
+
|
|
964
|
+
The speech is directable from the text itself: `[pause 2.0s]` markers insert
|
|
965
|
+
a silence of roughly that length, and inline Pinyin (`ni3 hao3`) or IPA
|
|
966
|
+
(`/həloʊ/`) steers pronunciation. `durationTokens` asks for a target length
|
|
967
|
+
in codec frames (12.5 per second, so `38` is about 3 s; `0` keeps the length
|
|
968
|
+
free). It is reloadable, and the engine rejects a target that does not fit in
|
|
969
|
+
its generation budget.
|
|
970
|
+
|
|
971
|
+
```js
|
|
972
|
+
const model = new TTSGgml({
|
|
973
|
+
engine: TTSGgml.ENGINE_MOSS,
|
|
974
|
+
files: { modelDir: './models' },
|
|
975
|
+
durationTokens: 38,
|
|
976
|
+
config: { language: 'en' }
|
|
977
|
+
})
|
|
978
|
+
await model.load()
|
|
979
|
+
await model.run({ input: 'Hold on [pause 1.0s] here it comes.' })
|
|
980
|
+
```
|
|
981
|
+
|
|
982
|
+
Dialogue needs the MOSS-TTSD backbone: with `dialogueReferences` set, a
|
|
983
|
+
`modelDir` must hold `moss-ttsd-*.gguf` (or name the file with
|
|
984
|
+
`files.mossBackbone`). A `modelDir` holding only the TTSD backbone also serves
|
|
985
|
+
plain synthesis. `dialogueReferences` takes one 24 kHz recording per
|
|
986
|
+
speaker, in the order the text tags them with `[S1]`, `[S2]`, and so on. The
|
|
987
|
+
model continues the references, so the text must open with what each
|
|
988
|
+
recording says, under its speaker tag, followed by the lines to generate; only
|
|
989
|
+
the new lines come out as audio. For the same reason a dialogue cannot be split
|
|
990
|
+
into sentences: `runStream()`, `runStreaming()` and `run({ streamOutput: true })`
|
|
991
|
+
are rejected; use `run()`, with `streamChunkTokens` for chunked audio. The references need
|
|
992
|
+
`files.mossCodecEncoder`, exclude `referenceAudio`, and are fixed for the
|
|
993
|
+
instance. They stay in the codec as causal history, so the first generated
|
|
994
|
+
words continue the reference voices without a seam.
|
|
995
|
+
|
|
996
|
+
```js
|
|
997
|
+
const dialogue = new TTSGgml({
|
|
998
|
+
engine: TTSGgml.ENGINE_MOSS,
|
|
999
|
+
files: { modelDir: './models' },
|
|
1000
|
+
dialogueReferences: ['./alice.wav', './bob.wav'],
|
|
1001
|
+
config: { language: 'en' }
|
|
1002
|
+
})
|
|
1003
|
+
await dialogue.load()
|
|
1004
|
+
await dialogue.run({
|
|
1005
|
+
input:
|
|
1006
|
+
'[S1] What alice.wav says. [S2] What bob.wav says. ' +
|
|
1007
|
+
'[S1] Did the build finish? [S2] Yes, every test passed.'
|
|
1008
|
+
})
|
|
1009
|
+
```
|
|
1010
|
+
|
|
1011
|
+
MOSS runs on CPU by default. `config.useGPU: true` (or a non-zero
|
|
1012
|
+
`nGpuLayers`) asks the engine for a GPU backend; when none is available it
|
|
1013
|
+
falls back to CPU and sets `response.stats.gpuUnsupported`. `backendsDir`
|
|
1014
|
+
reaches MOSS like every other engine, so builds that load ggml backends at
|
|
1015
|
+
runtime find them.
|
|
1016
|
+
|
|
822
1017
|
## API overview
|
|
823
1018
|
|
|
824
1019
|
### Constructor — `new TTSGgml(options)`
|
|
@@ -834,47 +1029,65 @@ a one-off encode when a new reference recording is supplied.
|
|
|
834
1029
|
| `files.cosyvoiceS3tokModel` | string | — | CosyVoice3 speech_tokenizer_v3 GGUF; needed only with `referenceAudio` (auto-discovered in the model dir as `cosyvoice3-s3tok-*.gguf`) |
|
|
835
1030
|
| `files.cosyvoiceCampplusModel` | string | — | CosyVoice3 CAM++ speaker-encoder GGUF; needed only with `referenceAudio` (auto-discovered as `cosyvoice3-campplus-*.gguf`) |
|
|
836
1031
|
| `files.cosyvoiceLlmModelPath` / `cosyvoiceFlowModelPath` / `cosyvoiceHiftModelPath` | string | — | Per-component overrides for the CosyVoice3 model dir |
|
|
1032
|
+
| `files.cosyvoiceVocab` / `cosyvoiceMerges` / `cosyvoiceVoiceModel` | string | — | CosyVoice3 text frontend (`vocab.json`, `merges.txt`) and baked voice (`voice.gguf`) overrides; each resolves under the model dir when unset. `cosyvoiceVoiceModel` selects a different baked voice without a separate model dir (`*Path` aliases accepted) |
|
|
837
1033
|
| `files.audio8Lm` | string | — | Audio8 DualAR language model GGUF (overrides `modelDir`) |
|
|
838
1034
|
| `files.audio8CodecDecoder`| string | — | Audio8 codec synthesis half — codes to wav (overrides `modelDir`) |
|
|
839
1035
|
| `files.audio8CodecEncoder`| string | — | Audio8 codec analysis half — wav to codes; only needed to clone a voice |
|
|
1036
|
+
| `files.mossBackbone` | string | — | MOSS Delay backbone GGUF: MOSS-TTS or the MOSS-TTSD dialogue checkpoint (overrides `modelDir`) |
|
|
1037
|
+
| `files.mossCodecDecoder` | string | — | MOSS codec synthesis half — codes to wav (overrides `modelDir`) |
|
|
1038
|
+
| `files.mossCodecEncoder` | string | — | MOSS codec analysis half — wav to codes; only needed to clone a voice or for `dialogueReferences` |
|
|
840
1039
|
| `files.lavasrEnhancer` | string | — | LavaSR enhancer GGUF — supplying it turns on 48 kHz enhancement |
|
|
841
1040
|
| `files.lavasrDenoiser` | string | — | LavaSR denoiser GGUF — supplying it turns on denoising (batch only) |
|
|
842
|
-
| `engine` | string | auto | Force `'chatterbox'`, `'supertonic'`, `'cosyvoice3'`, `'parler'` or `'
|
|
843
|
-
| `referenceAudio` | string | — | Wav to clone (Chatterbox: mono, ≥ 5 s; CosyVoice3: 0.5-30 s, multichannel downmixed to mono, needs the s3tok + campplus GGUFs; Audio8: also needs `referenceText`). Audio8 accepts it per call too |
|
|
1041
|
+
| `engine` | string | auto | Force `'chatterbox'`, `'supertonic'`, `'cosyvoice3'`, `'parler'`, `'audio8'` or `'moss'` (`TTSGgml.ENGINE_CHATTERBOX` / `ENGINE_SUPERTONIC` / `ENGINE_COSYVOICE3` / `ENGINE_PARLER` / `ENGINE_AUDIO8` / `ENGINE_MOSS`); auto-detected from the GGUFs present otherwise |
|
|
1042
|
+
| `referenceAudio` | string | — | Wav to clone (Chatterbox: mono, ≥ 5 s; CosyVoice3: 0.5-30 s, multichannel downmixed to mono, needs the s3tok + campplus GGUFs; Audio8: also needs `referenceText`; MOSS: needs `files.mossCodecEncoder`). Audio8 accepts it per call too |
|
|
844
1043
|
| `referenceText` | string | — | Audio8-only: what `referenceAudio` says, verbatim. Required whenever a reference is set; accepted per call |
|
|
1044
|
+
| `dialogueReferences` | string[] | — | MOSS-only: one 24 kHz recording per speaker (`[S1]`, `[S2]`, ...) for dialogue synthesis; the text must open with each recording's transcript under its tag; needs `files.mossCodecEncoder`, excludes `referenceAudio`, fixed per instance |
|
|
1045
|
+
| `durationTokens` | number | `0` | MOSS-only: target length in codec frames (12.5 per second); `0` keeps the length free; reloadable |
|
|
845
1046
|
| `voiceDir` | string | — | Pre-baked voice profile |
|
|
846
|
-
| `seed` | number | 42 | RNG seed (CFM noise + sampling) |
|
|
1047
|
+
| `seed` | number | 42 | RNG seed (CFM noise + sampling); MOSS defaults to its engine's 1234 |
|
|
847
1048
|
| `nGpuLayers` | number | 0 | Layers offloaded to GPU (mirrors `useGPU`; pass `99` to offload all) |
|
|
848
1049
|
| `nCtx` | number | 4096 | Chatterbox T3 context limit for the prompt plus generated speech tokens (25 tokens ≈ 1 s of audio). The KV cache is allocated up front at this length, so the 4096-token default directly bounds memory. Pass `0` to use the GGUF metadata value |
|
|
849
1050
|
| `kvCacheType` | string | `f16` | T3 KV-cache dtype: `f32` \| `f16` \| `q8_0`. `f16` (~50% of f32) is the safe cross-backend default. `q8_0` stores the cache at ~27% of f32 and decodes 20-30% faster on Metal, but only works on backends with a q8_0 CONT op (CPU, CUDA) — it hard-aborts the multilingual model on Metal, so it is opt-in. Turbo greedy decoding is byte-identical across all three (upstream-validated). Pass `f32` for bit-exact pre-quantisation behaviour |
|
|
850
|
-
| `threads` | number | hw.concurrency capped at 4 | |
|
|
851
|
-
| `streamChunkTokens` | number | 0 | **>0 enables native chunk streaming** |
|
|
1051
|
+
| `threads` | number | hw.concurrency capped at 4 | MOSS: engine default of 4; `0` keeps it |
|
|
1052
|
+
| `streamChunkTokens` | number | 0 | **>0 enables native chunk streaming** (speech tokens per chunk; Supertonic: text tokens; MOSS: codec frames per chunk, 12.5 per second) |
|
|
852
1053
|
| `streamFirstChunkTokens` | number | = streamChunkTokens | Smaller first chunk for low first-audio-out |
|
|
853
|
-
| `
|
|
1054
|
+
| `streamChunkTolerancePct` | number | 20 | Supertonic-only: boundary-snap window for clause / space chunk boundaries, in percent of the chunk target |
|
|
1055
|
+
| `streamMinChunkTokens` | number | 30 | Supertonic-only: floor on every streamed chunk's size in text tokens; shorter trailing chunks merge into the previous one |
|
|
1056
|
+
| `streamLeftContextTokens` | number | 0 | Chatterbox: left-context speech tokens S3Gen keeps per streamed chunk (typically 25-50), bounding per-chunk cost on long utterances; `0` = the whole emitted prefix. CosyVoice3: reserved — the pinned engine accepts but does not read it |
|
|
1057
|
+
| `cfmSteps` | number | 2 | Chatterbox native-streaming CFM steps: 1 = faster (halved CFM cost) |
|
|
1058
|
+
| `batchCfmSteps` | number | engine default | Chatterbox-only batch CFM steps (`0` = 2-step meanflow on Turbo, the model's full schedule on multilingual). The multilingual engine does not floor it, so values well below its schedule under-integrate |
|
|
1059
|
+
| `nPredict` | number | 1000 | Chatterbox-only cap on speech tokens per synthesis (25 ≈ 1 s, so ~40 s by default). Longer single calls also need `nCtx` to fit, or use `maxSentenceChars` |
|
|
1060
|
+
| `maxSentenceChars` | number | 0 | Chatterbox-only sentence auto-split: T3 + S3Gen run per segment of at most this many bytes (the reference CLI uses 180), so `nPredict` applies per segment |
|
|
1061
|
+
| `crossfadeMs` | number | 30 | Chatterbox-only raised-cosine crossfade between auto-split batch segments; native streaming stays gapless |
|
|
1062
|
+
| `repeatPenalty` | number | 1.2 | Chatterbox-only T3 repetition penalty (`1` = off) |
|
|
1063
|
+
| `exaggeration` / `cfgWeight` / `minP` | number | 0.5 / 0.5 / 0 | Chatterbox multilingual only (Turbo ignores them): prosody intensity (> 0 more expressive, < 0 flatter), T3 classifier-free guidance scale (`0` = off; `cfgRate` is the separate S3Gen rate), and the min-p sampling threshold |
|
|
854
1064
|
| `speed` | number | 1.0 | Speaking-rate multiplier, bounded `[0.25, 4.0]` (`< 1` slower, `> 1` faster). Both engines |
|
|
855
1065
|
| `voice` / `voiceName` | string | — | Supertonic voice id (e.g. `'F1'`, `'M1'`); Parler template speaker name (e.g. `'Laura'`, `'Rohit'`) |
|
|
1066
|
+
| `voiceJsonPath` | string | — | Supertonic-only: external voice JSON (`{ style_ttl, style_dp }`, e.g. a cloned voice) that overrides `voice`; the engine checks the tensor sizes against the model |
|
|
1067
|
+
| `prewarmText` | string | — | Supertonic-only: text synthesized once at load so GPU pipelines compile, and a Core ML vocoder sidecar specializes, before the first `run()` (skipped on a plain CPU run); use a representative length. Wins over the `vulkanCacheDir` default pre-warm |
|
|
1068
|
+
| `vulkanDevice` | number | 0 | Supertonic / CosyVoice3 Vulkan adapter, also used by their LavaSR enhancer: `0` = first, `N` = the Nth, `-1` = auto-pick by free VRAM preferring a discrete GPU |
|
|
1069
|
+
| `flowCutPrompt` | boolean | `false` | CosyVoice3-only: attention-only prompt frames in the flow — faster, with output that deviates slightly from the reference |
|
|
856
1070
|
| `steps` / `numInferenceSteps` | number | GGUF default | Supertonic vector-estimator CFM steps (`0` = GGUF default) |
|
|
857
1071
|
| `noiseNpyPath` | string | — | Supertonic: optional fixed CFM noise `.npy` for reproducibility |
|
|
858
1072
|
| `description` / `voiceDescription` | string | fallback caption | Parler-only: full free-text voice description (mutually exclusive with the template fields) |
|
|
859
1073
|
| `emotion`, `pace` | string | — | Cross-engine conditioning (see [Emotion & pace](#emotion--pace-cross-engine)); an unsupported value errors listing what that engine supports |
|
|
860
1074
|
| `pitch`, `expressivity`, `noise`, `reverb`, `quality` | string | — | Parler-only template fields (see [Parler descriptions](#parler-descriptions)); invalid values error listing the valid set |
|
|
861
|
-
| `temperature` / `topK` / `topP` | number | engine defaults | Parler
|
|
1075
|
+
| `temperature` / `topK` / `topP` | number | engine defaults | Parler, Audio8 and Chatterbox sampling knobs (omit for the engine's own defaults — Parler temp 1.0 / top-k 50, Audio8 temp 0.7 / top-k 50 / top-p 0.9, Chatterbox temp 0.8 / top-k 1000 / top-p 0.95 where temp `0` is greedy and top-k `0` disables the cutoff; `topP` in `(0, 1]`) |
|
|
862
1076
|
| `maxFrames` | number | engine max | Parler + Audio8 generation cap in frames (Parler ~86/s of audio, Audio8 ~21.5/s); `0` = model default, Parler rejects 1–9 |
|
|
863
1077
|
| `greedy` | boolean | `false` | Audio8-only: take the argmax instead of sampling; ignores `temperature` / `topK` / `topP` |
|
|
864
1078
|
| `minNewTokens` | number | GGUF default | Parler-only minimum tokens before EOS (`-1` = model default) |
|
|
865
1079
|
| `normalizeNumbers` | boolean | `true` | Parler-only: expand digits before tokenization (English words; script-native digits on indic) — parler voices raw digits badly |
|
|
866
|
-
| `instruct` | object \| string | — | CosyVoice3-only: instruction controls (`dialect` / `volume` / `style`, resolved by precedence in that order) or a raw instruction string; an unknown key or invalid value throws, and it counts toward the one-instruction rule (see [CosyVoice3 instruct](#cosyvoice3-instruct)) |
|
|
1080
|
+
| `instruct` | object \| string | — | CosyVoice3-only: instruction controls (`dialect` / `volume` / `style`, resolved by precedence in that order) or a raw instruction string; an unknown key or invalid value throws, and it counts toward the one-instruction rule (see [CosyVoice3 instruct](#cosyvoice3-instruct)). Also accepted per call, where it replaces the configured conditioning like a per-call `emotion` / `pace` |
|
|
867
1081
|
| `promptText` | string | — | CosyVoice3-only: verbatim transcript of `referenceAudio` — set for zero-shot cloning, omit for cross-lingual; without a reference it overrides the baked voice's transcript |
|
|
868
|
-
| `streamLeftContextTokens` | number | — | CosyVoice3-only: intended native chunk-streaming left-context tokens. Reserved / not yet effective — the pinned engine accepts but does not read it |
|
|
869
1082
|
| `mecabDictDir` | string | — | Chatterbox MTL Japanese (`ja`): compiled MeCab/IPAdic dictionary directory |
|
|
870
1083
|
| `cangjieTsvPath` | string | — | Chatterbox MTL Chinese (`zh`): `Cangjie5_TC` TSV path |
|
|
871
1084
|
| `backendsDir` | string | `resolveBackendsDir()` | Root dir the addon scans for dynamically-loaded ggml backend `.so` files. Defaults to the package's own `prebuilds/` when present, otherwise the installed platform package. Required on Android when backends ship elsewhere (e.g. inside the APK); ignored on platforms that statically link the backend |
|
|
872
1085
|
| `openclCacheDir` | string | unset | Android-only: directory where the OpenCL backend persists its compiled program-binary cache. Setting it across runs avoids re-JITing the kernels on every fresh process |
|
|
873
1086
|
| `vulkanCacheDir` | string | unset | Supertonic + `useGPU: true` only: writable directory where the Vulkan backend persists its compiled pipeline cache (`GGML_VK_PIPELINE_CACHE_DIR`). Moves the one-time first-dispatch pipeline-compile cost (seconds on Mali) off the first `run()` — paid once per install instead of once per process — and enables a load-time pre-warm. Fully opt-in: unset -> no cross-process cache, no pre-warm, behaviour unchanged |
|
|
874
|
-
| `config.language` | string | `"en"` | Chatterbox MTL accepts `es/fr/de/pt/it/zh/ja/ko/...`; turbo & Supertonic are English |
|
|
875
|
-
| `config.useGPU` | boolean | `false` | Set to `true` to route through Metal / CUDA / Vulkan / OpenCL if available. Honored for Chatterbox/Supertonic on GPU-capable hosts (including Android, per `tts-cpp`'s per-vendor allowlist); Parler is validated on Apple/Metal, linux CUDA, and Android/ARM Mali Vulkan; CosyVoice3 and Audio8 offload on Apple/Metal, desktop linux CUDA/Vulkan, Windows Vulkan, and Android OpenCL/Adreno. Unsupported backends fall back to CPU. See [Backends & GPU acceleration](#backends--gpu-acceleration) |
|
|
876
|
-
| `config.outputSampleRate` | number | — (engine-native) | Resample the output to this rate (8000–192000 Hz). Omit to keep the engine-native rate (Chatterbox 24 kHz, Supertonic / Parler / Audio8 44.1 kHz, CosyVoice3 24 kHz, enhancer 48 kHz). Parler native chunk streaming accepts a non-native rate only with the enhancer active |
|
|
877
|
-
| `opts.stats` | boolean | `false` | Populate `response.stats` with RTF, `backendDevice` (0=CPU, 1=GPU), `backendId` (0=CPU, 1=Metal, 2=CUDA, 3=Vulkan, 4=OpenCL, 99=other), and
|
|
1087
|
+
| `config.language` | string | `"en"` | Chatterbox MTL accepts `es/fr/de/pt/it/zh/ja/ko/...`; turbo & Supertonic are English; MOSS takes it as a prompt hint (`en`, `zh`, ...) |
|
|
1088
|
+
| `config.useGPU` | boolean | `false` | Set to `true` to route through Metal / CUDA / Vulkan / OpenCL if available. Honored for Chatterbox/Supertonic on GPU-capable hosts (including Android, per `tts-cpp`'s per-vendor allowlist); Parler is validated on Apple/Metal, linux CUDA, and Android/ARM Mali Vulkan; CosyVoice3 and Audio8 offload on Apple/Metal, desktop linux CUDA/Vulkan, Windows Vulkan, and Android OpenCL/Adreno; MOSS GPU offload is validated on Apple/Metal. Unsupported backends fall back to CPU. See [Backends & GPU acceleration](#backends--gpu-acceleration) |
|
|
1089
|
+
| `config.outputSampleRate` | number | — (engine-native) | Resample the output to this rate (8000–192000 Hz). Omit to keep the engine-native rate (Chatterbox 24 kHz, Supertonic / Parler / Audio8 44.1 kHz, CosyVoice3 24 kHz, MOSS 24 kHz, enhancer 48 kHz). Parler native chunk streaming accepts a non-native rate only with the enhancer active; MOSS accepts only its native rate |
|
|
1090
|
+
| `opts.stats` | boolean | `false` | Populate `response.stats` with RTF, `backendDevice` (0=CPU, 1=GPU), `backendId` (0=CPU, 1=Metal, 2=CUDA, 3=Vulkan, 4=OpenCL, 99=other), `enhancerBackendDevice` / `enhancerBackendId` and `denoiserBackendDevice` / `denoiserBackendId` (`-1` when that LavaSR stage is off), and the engine's own stage stats: Chatterbox `t3Ms` / `s3genMs` / `t3Tokens`; CosyVoice3 per-stage ms (`lmPrefillMs` … `hiftDecodeMs`, `stageTotalMs`) and work counters (`speechTokens`, `decodeSteps`, …); Audio8 per-stage ms (`prefillMs`, `fastDecodeMs`, `codecSynthMs`, …, `stageTotalMs`) |
|
|
878
1091
|
| `exclusiveRun` | boolean | `false` | **Top-level** option (not under `opts`): serialize overlapping streaming runs |
|
|
879
1092
|
|
|
880
1093
|
### Methods
|
|
@@ -887,6 +1100,8 @@ a one-off encode when a new reference recording is supplied.
|
|
|
887
1100
|
config (`language`, `useGPU`, `outputSampleRate`, …).
|
|
888
1101
|
- `await model.destroy()` — `unload()` + mark this instance dead.
|
|
889
1102
|
- `await model.cancel()` — best-effort cancel of any in-flight run.
|
|
1103
|
+
- `TTSGgml.getVoiceControls()` — static: tts-cpp's emotion / pace vocabulary
|
|
1104
|
+
and each engine's supported subset (see [Emotion & pace](#emotion--pace-cross-engine)).
|
|
890
1105
|
- `model.run({ input, type: 'text' })` → `QvacResponse`.
|
|
891
1106
|
- `model.run({ input, streamOutput: true })` → sentence-chunked
|
|
892
1107
|
synthesis driven by the JS-side sentence splitter (see
|
|
@@ -913,7 +1128,7 @@ All `run*` methods return a `QvacResponse` (from `@qvac/infer-base`):
|
|
|
913
1128
|
```js
|
|
914
1129
|
response.onUpdate(data => {
|
|
915
1130
|
data.outputArray // Int16Array — mono PCM
|
|
916
|
-
data.sampleRate // actual rate: Chatterbox 24000, Supertonic/Parler/Audio8 44100, enhancer 48000
|
|
1131
|
+
data.sampleRate // actual rate: Chatterbox/CosyVoice3/MOSS 24000, Supertonic/Parler/Audio8 44100, enhancer 48000
|
|
917
1132
|
data.chunkIndex // present on sentence-streaming events only
|
|
918
1133
|
data.sentenceChunk // present on sentence-streaming events only
|
|
919
1134
|
})
|
|
@@ -924,8 +1139,10 @@ response.stats.totalTime // seconds
|
|
|
924
1139
|
response.stats.realTimeFactor // synthesis time / audio duration
|
|
925
1140
|
response.stats.audioDurationMs
|
|
926
1141
|
response.stats.totalSamples
|
|
927
|
-
response.stats.tokensPerSecond // Audio8
|
|
928
|
-
response.stats.generatedFrames // Audio8 only: codec frames
|
|
1142
|
+
response.stats.tokensPerSecond // Audio8 and MOSS count codec frames, the others characters
|
|
1143
|
+
response.stats.generatedFrames // Audio8 and MOSS only: codec frames (Audio8 every 46 ms, MOSS every 80 ms)
|
|
1144
|
+
response.stats.codecSidecarLoaded // Audio8 only, macOS / iOS: 1 while the Apple Core ML codec sidecar is attached
|
|
1145
|
+
response.stats.codecOnCoreml // Audio8 only: 1 when this synthesis ran the codec on that sidecar, 0 when on the ggml backend
|
|
929
1146
|
response.stats.backendDevice // 0=CPU, 1=GPU
|
|
930
1147
|
response.stats.backendId // 0=CPU, 1=Metal, 2=CUDA, 3=Vulkan, 4=OpenCL, 99=other
|
|
931
1148
|
// present when a LavaSR enhancer is active:
|
|
@@ -933,6 +1150,11 @@ response.stats.enhancerBackendDevice // -1 none, 0 CPU, 1 GPU
|
|
|
933
1150
|
response.stats.enhancerBackendId
|
|
934
1151
|
```
|
|
935
1152
|
|
|
1153
|
+
For `runStream()` and `runStreaming()`, `codecSidecarLoaded` and
|
|
1154
|
+
`codecOnCoreml` retain the last chunk that reported each field, including zero
|
|
1155
|
+
when a sidecar is retired or synthesis falls back to ggml. They are not summed;
|
|
1156
|
+
if no chunk reports a field, it remains absent.
|
|
1157
|
+
|
|
936
1158
|
### Text helpers
|
|
937
1159
|
|
|
938
1160
|
The sentence splitter is available from
|
|
@@ -966,6 +1188,57 @@ const {
|
|
|
966
1188
|
| 13010 | `FAILED_TO_STOP` |
|
|
967
1189
|
| 13011 | `JOB_ALREADY_RUNNING` |
|
|
968
1190
|
|
|
1191
|
+
## Assessing fit
|
|
1192
|
+
|
|
1193
|
+
`assessFit` projects a load against the memory free right now. It reads GGUF metadata and never weight data, so the registry's weightless copy of each file answers the same as the file itself and the projection can run before anything is downloaded. It is a module export, not an instance method: nothing is loaded to call it.
|
|
1194
|
+
|
|
1195
|
+
A request is a `createInstance` config plus the workload, so it carries the same keys the load takes and the engine's own config builder reads it.
|
|
1196
|
+
|
|
1197
|
+
```js
|
|
1198
|
+
const TTSGgml = require('@qvac/tts-ggml')
|
|
1199
|
+
|
|
1200
|
+
const fit = TTSGgml.assessFit({
|
|
1201
|
+
engineType: 'supertonic',
|
|
1202
|
+
supertonicModelPath: '/models/supertonic.gguf',
|
|
1203
|
+
useGPU: true,
|
|
1204
|
+
textTokens: 256,
|
|
1205
|
+
audioSeconds: 30
|
|
1206
|
+
})
|
|
1207
|
+
|
|
1208
|
+
fit.status // 'fits' | 'does-not-fit' | 'error'
|
|
1209
|
+
fit.reason // the engine's own wording, e.g. 'model-unreadable', 'workload-too-large'
|
|
1210
|
+
fit.modelVariant // which pipeline was projected, e.g. 'chatterbox-t3-turbo'
|
|
1211
|
+
fit.deviceName
|
|
1212
|
+
fit.deviceBytes
|
|
1213
|
+
fit.weightsBytes
|
|
1214
|
+
fit.stateBytes
|
|
1215
|
+
fit.lmComputeBytes // 0 for a pipeline with no language-model stage, such as supertonic
|
|
1216
|
+
fit.codecComputeBytes // 0 for a pipeline with no separate codec stage
|
|
1217
|
+
fit.hostBytes
|
|
1218
|
+
fit.lavasrFileBytes // the LavaSR stages' size on disk, 0 when none was named
|
|
1219
|
+
fit.report
|
|
1220
|
+
```
|
|
1221
|
+
|
|
1222
|
+
`engineType` picks the fitter, the same key a load uses, and is required: a fit request carries none of the file keys a load is inferred from. Each engine takes the load's own file keys, plus a workload the fit adds:
|
|
1223
|
+
|
|
1224
|
+
| `engineType` | Files | Workload |
|
|
1225
|
+
| --- | --- | --- |
|
|
1226
|
+
| `supertonic` | `supertonicModelPath` | `textTokens`, `audioSeconds`, `precision`, `f16Weights` |
|
|
1227
|
+
| `parler` | `parlerModelPath` | `descriptionTokens`, `promptTokens`, `maxFrames` |
|
|
1228
|
+
| `chatterbox` | `t3ModelPath`, `s3genModelPath` | `textTokens`, `predictTokens` |
|
|
1229
|
+
| `audio8` | `audio8LmPath`, `audio8CodecDecoderPath`, `audio8CodecEncoderPath` | `promptTokens`, `maxFrames`, `referenceSeconds` |
|
|
1230
|
+
| `cosyvoice3` | `cosyvoiceLlmModelPath`, `cosyvoiceFlowModelPath`, `cosyvoiceHiftModelPath`, `cosyvoiceVoiceModelPath` | `textTokens`, `speechTokens` |
|
|
1231
|
+
|
|
1232
|
+
Everything else comes from the load config: `nGpuLayers` and `useGPU` carry the offload intent, `nCtx` and `kvCacheType` size the chatterbox cache, `steps` takes the GGUF's own default at 0, and `vulkanDevice` and `backendsDir` place the backend. Supplying `audio8CodecEncoderPath` projects voice cloning, which the decoder alone cannot do. `marginBytes` sets the free memory that must remain for the projection to count as fitting.
|
|
1233
|
+
|
|
1234
|
+
`lavasrEnhancerPath` and `lavasrDenoiserPath` are reported as `lavasrFileBytes`, their size on disk. Those stages have no fitter, so the figure is a size on disk. It is counted in neither `deviceBytes` nor `hostBytes`.
|
|
1235
|
+
|
|
1236
|
+
`deviceSharesHostMemory` reports that the device pool is system RAM, so host bytes compete with device bytes.
|
|
1237
|
+
|
|
1238
|
+
A model the engine cannot read is `status: "error"`, as is a voice with no fitter, which reports `reason: "unsupported-engine"`. MOSS is the one voice in that state. A broken request, or a host with no native binding, throws.
|
|
1239
|
+
|
|
1240
|
+
The supertonic fitter covers the fused graph path: a validated GPU, or a CPU without the Accelerate pointwise kernels. Elsewhere it answers `compute-path-not-supported` and projects nothing.
|
|
1241
|
+
|
|
969
1242
|
## Examples
|
|
970
1243
|
|
|
971
1244
|
Runnable demos under `examples/`:
|
|
@@ -983,12 +1256,15 @@ Runnable demos under `examples/`:
|
|
|
983
1256
|
| `supertonic-mtl-tts.js` | Multilingual Supertonic synthesis |
|
|
984
1257
|
| `supertonic-mtl-sweep-tts.js` | Multilingual Supertonic sweep across languages |
|
|
985
1258
|
| `supertonic-sentence-stream-tts.js` | Supertonic sentence-level streaming |
|
|
1259
|
+
| `supertonic-chunk-stream-tts.js` | Supertonic native chunk streaming via `streamChunkTokens`, with streaming playback |
|
|
986
1260
|
| `supertonic-enhanced.js` | Supertonic + LavaSR 48 kHz enhancement. `bare examples/supertonic-enhanced.js "Hello"` |
|
|
987
1261
|
| `parler-tts.js` | Parler batch synth with voice/emotion templates. `bare examples/parler-tts.js "Hello" Laura happy` |
|
|
988
1262
|
| `parler-enhanced.js` | Parler + LavaSR 48 kHz enhancement. `bare examples/parler-enhanced.js "Hello" Laura happy` |
|
|
989
1263
|
| `cosyvoice-tts.js` | CosyVoice3 batch synth with the cross-engine emotion option (24 kHz; CPU by default, `--gpu` opts into Metal / desktop Vulkan / Android GPU). `bare examples/cosyvoice-tts.js --gpu "Hello" happy` |
|
|
990
1264
|
| `cosyvoice-enhanced.js` | CosyVoice3 + LavaSR 48 kHz enhancement (add `--denoise` for the denoiser). `bare examples/cosyvoice-enhanced.js "Hello"` |
|
|
991
1265
|
| `audio8-tts.js` | Audio8 batch synth, optionally cloning a reference. Set `QVAC_TTS_AUDIO8_GPU=1` for the desktop GPU backend (CUDA or Vulkan on linux, Vulkan on Windows). `bare examples/audio8-tts.js "Hello" voice.wav "What it says."` |
|
|
1266
|
+
| `moss-tts.js` | MOSS batch synth, optionally cloning a reference. Set `QVAC_TTS_MOSS_STREAM_FRAMES=25` for native chunk streaming, `QVAC_TTS_MOSS_DURATION=38` for a target length and `QVAC_TTS_MOSS_GPU=1` for the GPU backend. `bare examples/moss-tts.js "Hello" voice.wav` |
|
|
1267
|
+
| `moss-dialogue-tts.js` | MOSS-TTSD multi-speaker dialogue from one 24 kHz reference per speaker; the text opens with each reference's transcript. Set `QVAC_TTS_MOSS_GPU=1` for the GPU backend. `bare examples/moss-dialogue-tts.js "[S1] What alice.wav says. [S2] What bob.wav says. [S1] Hi. [S2] Hello." alice.wav bob.wav` |
|
|
992
1268
|
|
|
993
1269
|
The two streaming examples feed PCM into a single long-running
|
|
994
1270
|
`sox play` / `ffplay` process so chunks play back-to-back without any
|
|
@@ -1011,8 +1287,7 @@ files are absent. They cover, across the engines:
|
|
|
1011
1287
|
* batch synthesis with full RuntimeStats,
|
|
1012
1288
|
* sentence-level streaming (`runStream` / `run({ streamOutput: true })`
|
|
1013
1289
|
/ `runStreaming` over async iterators),
|
|
1014
|
-
* native sub-sentence chunk streaming
|
|
1015
|
-
`streamChunkTokens`),
|
|
1290
|
+
* native sub-sentence chunk streaming via `streamChunkTokens`,
|
|
1016
1291
|
* sequential-run / fresh-instance / reload-stability behaviour,
|
|
1017
1292
|
* strict GPU-backend assertion via `response.stats.backendDevice` +
|
|
1018
1293
|
`backendId` (set `NO_GPU=true` to skip on CPU-only runners,
|
|
@@ -1023,6 +1298,11 @@ files are absent. They cover, across the engines:
|
|
|
1023
1298
|
for WER against the synthesized audio (whisper-small via the
|
|
1024
1299
|
`@qvac/asr-ggml` development dependency).
|
|
1025
1300
|
|
|
1301
|
+
The MOSS integration test (`moss.test.js`) does not download its GGUFs:
|
|
1302
|
+
set `QVAC_TEST_MOSS_MODEL_DIR` to a directory holding them, otherwise it is
|
|
1303
|
+
skipped. The C++ suite runs its MOSS synthesis cases when
|
|
1304
|
+
`QVAC_TEST_MOSS_BACKBONE_GGUF` and `QVAC_TEST_MOSS_DECODER_GGUF` are set.
|
|
1305
|
+
|
|
1026
1306
|
To stress-test long inputs, set `INPUT_SENTENCES=medium` (or `long`)
|
|
1027
1307
|
and re-run the integration suite — `addon.test.js` reads the env var to
|
|
1028
1308
|
pick its sentence corpus from `test/data/sentences-{medium,long}.js`.
|
|
@@ -1089,3 +1369,7 @@ OpenCL; Xclipse and Mali use Vulkan).
|
|
|
1089
1369
|
## License
|
|
1090
1370
|
|
|
1091
1371
|
Apache-2.0. See [LICENSE](./LICENSE).
|
|
1372
|
+
|
|
1373
|
+
## Pocket TTS
|
|
1374
|
+
|
|
1375
|
+
See [Pocket TTS](docs/pocket-tts.md) for model conversion, addon/SDK usage, supported controls and validation commands.
|