@qvac/tts-ggml 0.9.1 → 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +342 -40
- package/docs/pocket-tts.md +201 -0
- package/examples/moss-dialogue-tts.js +113 -0
- package/examples/moss-tts.js +154 -0
- package/examples/pocket-tts.js +51 -0
- package/examples/supertonic-chunk-stream-tts.js +156 -0
- package/examples/supertonic-sentence-stream-tts.js +4 -4
- package/index.d.ts +352 -29
- package/index.js +698 -52
- package/lib/fit.d.ts +127 -0
- package/lib/fit.js +29 -0
- package/lib/pocketConfig.d.ts +10 -0
- package/lib/pocketConfig.js +145 -0
- package/package.json +14 -11
- package/test/integration/addon.test.js +5 -0
- package/test/integration/audio8.test.js +7 -0
- package/test/integration/chatterbox-mtl.test.js +6 -0
- package/test/integration/chatterbox-speed.test.js +13 -2
- package/test/integration/cosyvoice3-clone.test.js +16 -1
- package/test/integration/cosyvoice3-lavasr.test.js +6 -0
- package/test/integration/cosyvoice3.test.js +19 -0
- package/test/integration/fit.test.js +174 -0
- package/test/integration/gpu-smoke.test.js +13 -0
- package/test/integration/lavasr-enhancer.test.js +104 -3
- package/test/integration/moss.test.js +282 -0
- package/test/integration/multiple-runs.test.js +18 -2
- package/test/integration/output-sample-rate.test.js +6 -0
- package/test/integration/parler-wer.test.js +7 -1
- package/test/integration/parler.test.js +27 -3
- package/test/integration/pocket.integration.test.js +156 -0
- package/test/integration/supertonic-mtl.test.js +2 -0
- package/test/integration/supertonic.test.js +70 -0
- package/test/integration/supertonic3-quant.test.js +2 -0
- package/test/integration/voice-controls.test.js +56 -0
- package/test/mobile/integration.auto.cjs +16 -1
- package/test/mobile/pocket-worklet-host.cpp +92 -0
- package/test/mobile/pocket-worklet.cjs +47 -0
- package/test/mobile/test-groups.json +8 -4
- package/test/utils/runCosyvoiceTTS.js +1 -0
- package/test/utils/runTTS.js +6 -11
- package/tts.d.ts +13 -2
- package/tts.js +3 -2
package/README.md
CHANGED
|
@@ -7,8 +7,10 @@ v1/v2 still loadable), **Parler** (mini/large English + indic 21-language,
|
|
|
7
7
|
description-conditioned with voice/emotion templates), **CosyVoice3**
|
|
8
8
|
(Fun-CosyVoice3-0.5B, instruct-conditioned, 24 kHz, CPU with opt-in
|
|
9
9
|
Metal on Apple, CUDA/Vulkan on Linux, Vulkan on Windows, and OpenCL/Adreno
|
|
10
|
-
GPU offload on Android),
|
|
11
|
-
(DualAR + neural codec, in-process voice cloning, desktop GPU),
|
|
10
|
+
GPU offload on Android), **Audio8**
|
|
11
|
+
(DualAR + neural codec, in-process voice cloning, desktop GPU), and **MOSS**
|
|
12
|
+
(MOSS-TTS Delay, 24 kHz, in-process voice cloning, native chunk streaming,
|
|
13
|
+
desktop), plus optional
|
|
12
14
|
LavaSR neural denoise + 48 kHz bandwidth-extension enhancement. Unsure
|
|
13
15
|
which checkpoint to stage? Start with [Choosing a model](#choosing-a-model).
|
|
14
16
|
|
|
@@ -24,7 +26,8 @@ backend per its own per-vendor allowlist (Adreno → OpenCL,
|
|
|
24
26
|
Xclipse/Mali → Vulkan). Parler supports Apple/Metal, linux CUDA, and the
|
|
25
27
|
validated Android paths, including Vulkan on ARM Mali (see
|
|
26
28
|
[Backends & GPU acceleration](#backends--gpu-acceleration)). Audio8 supports
|
|
27
|
-
CUDA/Vulkan offload on Linux and Vulkan on Windows.
|
|
29
|
+
CUDA/Vulkan offload on Linux and Vulkan on Windows. MOSS GPU offload is
|
|
30
|
+
validated on Apple/Metal.
|
|
28
31
|
|
|
29
32
|
[qvac-tts-cpp]: https://github.com/tetherto/qvac-fabric-speech.cpp/tree/master/engines/tts
|
|
30
33
|
|
|
@@ -44,6 +47,9 @@ CUDA/Vulkan offload on Linux and Vulkan on Windows.
|
|
|
44
47
|
`config.useGPU: true` on GPU-capable hosts — including Android, where
|
|
45
48
|
`tts-cpp` selects the GPU backend per its per-vendor allowlist (see
|
|
46
49
|
[Backends & GPU acceleration](#backends--gpu-acceleration)).
|
|
50
|
+
- **Apple Core ML sidecars** for the Supertonic vocoder and the Audio8 codec
|
|
51
|
+
on macOS / iOS, picked up when staged next to the model (see
|
|
52
|
+
[Core ML sidecars on Apple](#core-ml-sidecars-on-apple)).
|
|
47
53
|
- **Dynamic backend loading on Android** — per-arch CPU + Vulkan +
|
|
48
54
|
OpenCL `.so` files ship under `prebuilds/<bare-target>/qvac__tts-ggml/`
|
|
49
55
|
and are picked up at runtime via the new `backendsDir` option (see
|
|
@@ -66,6 +72,8 @@ obvious for each job.
|
|
|
66
72
|
| English + voice cloning + low first-audio latency | `chatterbox-t3-turbo.gguf` + `chatterbox-s3gen.gguf` | Reference-wav / voice-profile cloning; native `streamChunkTokens` chunk streaming. Native 24 kHz. |
|
|
67
73
|
| Multilingual + voice cloning (EU / CJK) | `chatterbox-t3-mtl.gguf` + `chatterbox-s3gen-mtl.gguf` | en/es/fr/de/pt/it/zh/ja/ko/…; same cloning + streaming surface as Turbo. |
|
|
68
74
|
| Voice cloning at 44.1 kHz with nothing pre-baked | `audio8-lm-q8_0.gguf` + `audio8-codec-decoder-q8_0.gguf` (+ `audio8-codec-encoder-q8_0.gguf` to clone) | Clones from a reference wav **and its transcript**, encoded in-process — no enrolment step, no voice profile. Whole-utterance only (no native chunk streaming). |
|
|
75
|
+
| Multilingual speech with native chunk streaming and cloning on a desktop | `moss-tts-delay-f16.gguf` + `moss-codec-decoder-f16.gguf` (+ `moss-codec-encoder-f16.gguf` to clone) | 8B backbone, desktop only. Clones from a reference wav encoded in-process (no transcript needed); native `streamChunkTokens` chunk streaming. Native 24 kHz. |
|
|
76
|
+
| Multi-speaker dialogue that continues each speaker's reference voice | `moss-ttsd-f16.gguf` + `moss-codec-decoder-f16.gguf` + `moss-codec-encoder-f16.gguf` | 8B backbone, desktop only. One 24 kHz reference per speaker; the text opens with their transcripts. Native 24 kHz. |
|
|
69
77
|
| Indic languages | `parler-indic-q8_0.gguf` | 21 Indic languages; voice / emotion templates. Emotion is officially tested on 10 languages — see [Parler descriptions & emotions](#parler-descriptions--emotions) (or the upstream model card). Native 44.1 kHz. |
|
|
70
78
|
| Chinese dialects (Cantonese, Sichuan, Shanghai, …) | CosyVoice3 dir (`cosyvoice3-llm-*.gguf` + flow / hift / `voice.gguf`) | Instruct-conditioned; 17 dialects via `instruct: { dialect: '…' }`. CPU, with opt-in GPU offload (Metal on Apple, Vulkan on Linux/Windows, OpenCL/Adreno on Android); native 24 kHz. |
|
|
71
79
|
| Zero-shot / cross-lingual cloning (multilingual) | CosyVoice3 dir + `cosyvoice3-s3tok-*.gguf` + `cosyvoice3-campplus-*.gguf` | Clones from a reference wav; its transcript (`promptText`) selects zero-shot, omitting it selects cross-lingual (timbre only, any target language). Composes with `instruct`. |
|
|
@@ -88,6 +96,8 @@ here, so this guide does not go stale when backends change.
|
|
|
88
96
|
| `parler-mini-v1-q8_0` | English | ~1.2 GB | 44.1 kHz | Description / templates | Sentence streaming |
|
|
89
97
|
| CosyVoice3 (`cosyvoice3/`) | Instruct-led (strong on Chinese + dialects) | ~2.3 GB dir (+ ~300 MB to clone) | 24 kHz | Reference wav (zero-shot / cross-lingual) + `instruct` (dialect / emotion / speed / volume / style) | Native chunk opts |
|
|
90
98
|
| `audio8-lm-q8_0` + codec halves | Text-led (no `language` option) | ~0.7 GB (+ ~120 MB to clone) | 44.1 kHz | Reference wav + transcript, per call too | Sentence streaming |
|
|
99
|
+
| `moss-tts-delay-f16` + codec halves | Multilingual (`language` hint) | ~18.8 GB (+ ~1.8 GB to clone) | 24 kHz | Reference wav, fixed per instance | Sentence + native chunk |
|
|
100
|
+
| `moss-ttsd-f16` + codec halves | Multilingual (`language` hint) | ~20.5 GB | 24 kHz | One reference wav per speaker, fixed per instance | Native chunk |
|
|
91
101
|
|
|
92
102
|
### Legacy Supertonic v1 / v2
|
|
93
103
|
|
|
@@ -119,8 +129,9 @@ npm install @qvac/tts-ggml
|
|
|
119
129
|
Requires [Bare](https://github.com/holepunchto/bare) `>=1.19.0`.
|
|
120
130
|
|
|
121
131
|
`@qvac/tts-ggml` is a meta package that ships the JavaScript wrapper only.
|
|
122
|
-
The native prebuild for each host lives in a version-locked platform
|
|
123
|
-
selected at install time through `os`/`cpu` filtered
|
|
132
|
+
The native prebuild for each desktop host lives in a version-locked platform
|
|
133
|
+
package selected at install time through `os`/`cpu` filtered
|
|
134
|
+
`optionalDependencies`:
|
|
124
135
|
|
|
125
136
|
| Host | Package |
|
|
126
137
|
| --- | --- |
|
|
@@ -129,12 +140,10 @@ selected at install time through `os`/`cpu` filtered `optionalDependencies`:
|
|
|
129
140
|
| darwin-arm64 | `@qvac/tts-ggml-darwin-arm64` |
|
|
130
141
|
| darwin-x64 | `@qvac/tts-ggml-darwin-x64` |
|
|
131
142
|
| win32-x64 | `@qvac/tts-ggml-win32-x64` |
|
|
132
|
-
| android-arm64 | `@qvac/tts-ggml-android-arm64` |
|
|
133
|
-
| ios (device + simulators) | `@qvac/tts-ggml-ios` |
|
|
134
143
|
|
|
135
|
-
Do not depend on platform packages directly. Supported installers are
|
|
136
|
-
pnpm, bun, and Yarn Berry. Yarn v1 and `--omit=optional` installs skip
|
|
137
|
-
platform package and fail at require time with an error naming the missing
|
|
144
|
+
Do not depend on desktop platform packages directly. Supported installers are
|
|
145
|
+
npm 7+, pnpm, bun, and Yarn Berry. Yarn v1 and `--omit=optional` installs skip
|
|
146
|
+
the platform package and fail at require time with an error naming the missing
|
|
138
147
|
package; a locally built `prebuilds/` directory in the package root always
|
|
139
148
|
takes precedence. Use `require('@qvac/tts-ggml').resolveBackendsDir()` to
|
|
140
149
|
locate the directory holding the host's prebuilt binaries and dynamically
|
|
@@ -142,10 +151,29 @@ loaded ggml backends. Unsupported targets must
|
|
|
142
151
|
[build from source](#build-from-source); installation does not automatically
|
|
143
152
|
compile a local addon.
|
|
144
153
|
|
|
154
|
+
Mobile targets are cross-built, so no install host ever matches their `os`,
|
|
155
|
+
and `optionalDependencies` filtering can never select them. Mobile
|
|
156
|
+
applications must declare the target's platform package as a direct
|
|
157
|
+
dependency, pinned to the exact `@qvac/tts-ggml` version:
|
|
158
|
+
|
|
159
|
+
| Target | Package |
|
|
160
|
+
| --- | --- |
|
|
161
|
+
| android-arm64 | `@qvac/tts-ggml-android-arm64` |
|
|
162
|
+
| ios (device + simulators) | `@qvac/tts-ggml-ios` |
|
|
163
|
+
|
|
164
|
+
```json
|
|
165
|
+
{
|
|
166
|
+
"dependencies": {
|
|
167
|
+
"@qvac/tts-ggml": "x.y.z",
|
|
168
|
+
"@qvac/tts-ggml-android-arm64": "x.y.z"
|
|
169
|
+
}
|
|
170
|
+
}
|
|
171
|
+
```
|
|
172
|
+
|
|
145
173
|
## Model files
|
|
146
174
|
|
|
147
|
-
|
|
148
|
-
CosyVoice3, Audio8), each with its own GGUF layout under `models/`:
|
|
175
|
+
Six engine families are wrapped (Chatterbox, Supertonic, Parler,
|
|
176
|
+
CosyVoice3, Audio8, MOSS), each with its own GGUF layout under `models/`:
|
|
149
177
|
|
|
150
178
|
```
|
|
151
179
|
# Chatterbox turbo (English)
|
|
@@ -159,6 +187,9 @@ chatterbox-s3gen-mtl.gguf (~1.0 GB)
|
|
|
159
187
|
# Supertonic 3 (Supertone/supertonic-3; 31 languages) — preferred Supertonic
|
|
160
188
|
# checkpoint; published per quant tier (auto-detected from modelDir)
|
|
161
189
|
supertonic3-q4_0.gguf (~80 MB; also -q8_0 ~126 MB / -f16 ~191 MB / -f32)
|
|
190
|
+
supertonic3-vocoder.mlmodelc/ (optional, macOS / iOS: Apple Core ML sidecar for
|
|
191
|
+
the vocoder, used by the 8-bit and wider tiers; see
|
|
192
|
+
Core ML sidecars on Apple)
|
|
162
193
|
|
|
163
194
|
# Legacy Supertonic (not recommended for new integrations — see Choosing a model)
|
|
164
195
|
supertonic.gguf (~263 MB) — v1 English only
|
|
@@ -177,8 +208,9 @@ parler-indic-q8_0.gguf (~1.3 GB; also -f16 / -f32; 21 Indic languages)
|
|
|
177
208
|
# from the cosyvoice3-llm-*.gguf file
|
|
178
209
|
cosyvoice3/
|
|
179
210
|
cosyvoice3-llm-*.gguf (~973 MB q8_0 — Qwen2.5 speech LM)
|
|
180
|
-
cosyvoice3-flow-*.gguf (~1.3 GB f32 — DiT conditional-flow-matching
|
|
181
|
-
|
|
211
|
+
cosyvoice3-flow-*.gguf (~1.3 GB f32 — DiT conditional-flow-matching; also
|
|
212
|
+
-f16 / -bf16 / -q8_0 / -q4_0)
|
|
213
|
+
cosyvoice3-hift-*.gguf (~83 MB f32 — CausalHiFT vocoder; also -f16)
|
|
182
214
|
voice.gguf (baked default voice: timbre + prompt tensors)
|
|
183
215
|
vocab.json merges.txt (Qwen2 BPE tokenizer)
|
|
184
216
|
cosyvoice3-s3tok-*.gguf (~275 MB q8_0 / ~497 MB f16 — speech tokenizer;
|
|
@@ -190,6 +222,17 @@ cosyvoice3/
|
|
|
190
222
|
audio8-lm-q8_0.gguf (~0.6 GB; also -f16 / -q4_0 / -f32)
|
|
191
223
|
audio8-codec-decoder-q8_0.gguf (~110 MB; also -f16 / -f32)
|
|
192
224
|
audio8-codec-encoder-q8_0.gguf (~120 MB; also -f16 / -f32; cloning only)
|
|
225
|
+
audio8-codec-decoder.mlmodelc/ (optional, macOS / iOS: Apple Core ML sidecar for
|
|
226
|
+
the codec synthesis stack, picked up when it sits
|
|
227
|
+
next to the decoder GGUF; one export serves every
|
|
228
|
+
quant tier)
|
|
229
|
+
|
|
230
|
+
# MOSS (OpenMOSS MOSS-TTS v1.5 Delay; 24 kHz, 8B backbone + RVQ codec); the
|
|
231
|
+
# encoder is only needed to clone a voice or run a dialogue
|
|
232
|
+
moss-tts-delay-f16.gguf (~17 GB)
|
|
233
|
+
moss-ttsd-f16.gguf (~17 GB; MOSS-TTSD dialogue backbone, optional)
|
|
234
|
+
moss-codec-decoder-f16.gguf (~1.8 GB)
|
|
235
|
+
moss-codec-encoder-f16.gguf (~1.8 GB; cloning and dialogue only)
|
|
193
236
|
```
|
|
194
237
|
|
|
195
238
|
Download the registry-published Chatterbox, Supertonic, and Parler models into
|
|
@@ -201,10 +244,15 @@ npm run download-models:registry -- --group chatterbox,supertonic3
|
|
|
201
244
|
npm run download-models:registry -- --output /path/to/models
|
|
202
245
|
```
|
|
203
246
|
|
|
204
|
-
CosyVoice3 and
|
|
205
|
-
Stage their layouts shown above from local converted artifacts.
|
|
206
|
-
|
|
207
|
-
|
|
247
|
+
CosyVoice3, Audio8 and MOSS are not currently included in that registry command.
|
|
248
|
+
Stage their layouts shown above from local converted artifacts. CosyVoice3
|
|
249
|
+
resolves each component by filename prefix and does not rank quantizations, so
|
|
250
|
+
stage one file per component or name it explicitly with
|
|
251
|
+
`files.cosyvoiceFlowModelPath` / `cosyvoiceHiftModelPath`. Which flow tier is
|
|
252
|
+
fastest depends on the backend — `q8_0` on Vulkan / CUDA, `f16` on Metal,
|
|
253
|
+
`bf16` on AVX512-BF16 CPUs and `f16` on other CPUs — and the `f16` HiFT is the
|
|
254
|
+
recommendation everywhere. The package converts upstream Chatterbox,
|
|
255
|
+
Supertonic, and Parler checkpoints via a Python venv pipeline:
|
|
208
256
|
|
|
209
257
|
```bash
|
|
210
258
|
npm run setup-models # creates ./venv, installs requirements.txt, runs convert-models.sh
|
|
@@ -220,7 +268,9 @@ npm run convert-models
|
|
|
220
268
|
The Audio8 GGUFs are produced by the converters in
|
|
221
269
|
qvac-fabric-speech.cpp (`engines/tts/scripts/convert-audio8-lm-to-gguf.py`
|
|
222
270
|
and `convert-audio8-codec-to-gguf.py`) until they are published to the model
|
|
223
|
-
registry alongside the other engines.
|
|
271
|
+
registry alongside the other engines. The MOSS GGUFs come from
|
|
272
|
+
`convert-moss-delay-to-gguf.py` and `convert-moss-codec-to-gguf.py` in the same
|
|
273
|
+
directory.
|
|
224
274
|
|
|
225
275
|
Point the addon at a custom location via `files.modelDir` (engine
|
|
226
276
|
auto-detected from the gguf filenames present), or pass explicit
|
|
@@ -229,7 +279,9 @@ auto-detected from the gguf filenames present), or pass explicit
|
|
|
229
279
|
`files.cosyvoiceModelDir` (CosyVoice3 — a directory, see
|
|
230
280
|
[CosyVoice3 instruct](#cosyvoice3-instruct)) /
|
|
231
281
|
`files.audio8Lm` + `files.audio8CodecDecoder` (+ `files.audio8CodecEncoder`
|
|
232
|
-
to clone) (Audio8)
|
|
282
|
+
to clone) (Audio8) /
|
|
283
|
+
`files.mossBackbone` + `files.mossCodecDecoder` (+ `files.mossCodecEncoder`
|
|
284
|
+
to clone) (MOSS).
|
|
233
285
|
|
|
234
286
|
## Quick start
|
|
235
287
|
|
|
@@ -302,7 +354,8 @@ Full runnable demo (with streaming playback):
|
|
|
302
354
|
|
|
303
355
|
Use when you want the fastest possible first-audio-out **within a
|
|
304
356
|
single utterance**. The C++ engine splits each synthesis into chunks
|
|
305
|
-
of `streamChunkTokens` speech tokens (25 ≈ 1 s of audio
|
|
357
|
+
of `streamChunkTokens` speech tokens (25 ≈ 1 s of audio; on MOSS the unit
|
|
358
|
+
is a codec frame, so 25 ≈ 2 s) and emits
|
|
306
359
|
audio per chunk, keeping HiFT's source cache phase-continuous across
|
|
307
360
|
seams so the joins are inaudible.
|
|
308
361
|
|
|
@@ -327,6 +380,28 @@ await response.onUpdate(data => {
|
|
|
327
380
|
Full runnable demo (with gapless playback via `sox` or `ffplay`):
|
|
328
381
|
`bare examples/chatterbox-chunk-stream-tts.js`
|
|
329
382
|
|
|
383
|
+
Long utterances re-synthesize the whole emitted prefix for every chunk by
|
|
384
|
+
default; `streamLeftContextTokens` (typically 25-50) keeps only that many
|
|
385
|
+
tokens of left context per chunk, which bounds the per-chunk cost.
|
|
386
|
+
|
|
387
|
+
Supertonic streams the same way, but its chunks are measured in text tokens
|
|
388
|
+
(Unicode code points; about 50 for English, 25-30 for CJK). The engine snaps
|
|
389
|
+
each boundary to a sentence end, clause or space within
|
|
390
|
+
`streamChunkTolerancePct` of the target and never emits a chunk shorter than
|
|
391
|
+
`streamMinChunkTokens`. Supertonic native streaming cannot be combined with
|
|
392
|
+
the LavaSR enhancer or denoiser; use sentence-level streaming for enhanced
|
|
393
|
+
output.
|
|
394
|
+
|
|
395
|
+
```js
|
|
396
|
+
const model = new TTSGgml({
|
|
397
|
+
files: { supertonicModel: './models/supertonic3-q8_0.gguf' },
|
|
398
|
+
streamChunkTokens: 50, // text tokens per chunk
|
|
399
|
+
streamFirstChunkTokens: 20 // smaller first chunk
|
|
400
|
+
})
|
|
401
|
+
```
|
|
402
|
+
|
|
403
|
+
Full runnable demo (with streaming playback): `bare examples/supertonic-chunk-stream-tts.js`
|
|
404
|
+
|
|
330
405
|
## Voice cloning
|
|
331
406
|
|
|
332
407
|
### Chatterbox
|
|
@@ -397,6 +472,14 @@ across calls skips the encoder. Per-call fields also ride on the
|
|
|
397
472
|
`runStream` / `runStreaming` options, pinned for the whole response so the
|
|
398
473
|
cache stays hot across chunks.
|
|
399
474
|
|
|
475
|
+
### MOSS
|
|
476
|
+
|
|
477
|
+
MOSS clones from `referenceAudio` alone, with no transcript. The recording
|
|
478
|
+
must be sampled at 24 kHz (there is no resampling; multichannel input is
|
|
479
|
+
downmixed), needs `files.mossCodecEncoder`, and is fixed for the instance.
|
|
480
|
+
For a multi-speaker dialogue, pass one recording per speaker in
|
|
481
|
+
`dialogueReferences` instead. See [MOSS](#moss).
|
|
482
|
+
|
|
400
483
|
### CosyVoice3
|
|
401
484
|
|
|
402
485
|
CosyVoice3 clones **zero-shot or cross-lingual**, selected by whether the
|
|
@@ -561,6 +644,10 @@ Notes:
|
|
|
561
644
|
- Denoiser + native chunk streaming (`streamChunkTokens > 0`) is rejected up
|
|
562
645
|
front for every engine — a stateful streaming denoiser is the follow-up. Use
|
|
563
646
|
batch synthesis, or drop the denoiser for streaming.
|
|
647
|
+
- The denoiser follows the engine's resolved device, like the enhancer: a GPU
|
|
648
|
+
engine runs its ggml graph on the GPU (falling back to the ggml CPU backend
|
|
649
|
+
when no GPU backend initialises), a CPU engine keeps the scalar core.
|
|
650
|
+
`denoiserBackendDevice` / `denoiserBackendId` in the stats report where it ran.
|
|
564
651
|
- The tts-cpp UL-UNAS forward is implemented in
|
|
565
652
|
[qvac-fabric-speech.cpp#78](https://github.com/tetherto/qvac-fabric-speech.cpp/pull/78)
|
|
566
653
|
(scalar CPU port, validated bit-close to the ONNX reference); it requires a
|
|
@@ -587,6 +674,46 @@ On hosts where more than one backend is usable, `TTS_CPP_GPU_BACKEND`
|
|
|
587
674
|
and fails loudly when that backend cannot be resolved; unset (or empty)
|
|
588
675
|
keeps the automatic preference above.
|
|
589
676
|
|
|
677
|
+
### Core ML sidecars on Apple
|
|
678
|
+
|
|
679
|
+
The macOS / iOS prebuilds carry the Apple Core ML (Neural Engine) sidecars
|
|
680
|
+
for the Supertonic vocoder and the Audio8 codec. They are presence-driven:
|
|
681
|
+
each stage runs on a compiled `.mlmodelc` found next to its model file
|
|
682
|
+
(`supertonic3-q8_0.gguf` -> `supertonic3-vocoder.mlmodelc`) and falls back to
|
|
683
|
+
the ggml graph when it is absent, so a model directory without sidecars
|
|
684
|
+
behaves exactly as before. The sidecar name drops the quantization suffix, so
|
|
685
|
+
one sidecar serves every tier of a model. Sidecars are not part of the
|
|
686
|
+
published model set yet; supply your own to opt in.
|
|
687
|
+
|
|
688
|
+
| Model | Sidecar next to the GGUF | Stage on Core ML | Runs on ggml instead | Force ggml |
|
|
689
|
+
| --- | --- | --- | --- | --- |
|
|
690
|
+
| Supertonic 1 / 2 / 3 | `<model>-vocoder.mlmodelc` | vocoder, in 64-latent-frame windows | GGUFs whose vocoder weights are stored below 8 bits (`q4_0`) | `SUPERTONIC_COREML_DISABLE=1` |
|
|
691
|
+
| Audio8 | `audio8-codec-decoder.mlmodelc`, beside the codec decoder GGUF | codec synthesis stack (upsampling + DAC decoder), in 64-post-frame windows; the language model stays on the ggml backend | a call that fails on the sidecar, which also retires it for every later call on that instance | `AUDIO8_COREML_DISABLE=1` |
|
|
692
|
+
| Chatterbox, Parler, CosyVoice3, MOSS, LavaSR | none | — | always | — |
|
|
693
|
+
|
|
694
|
+
Set the force-ggml variables in the process environment before `load()`.
|
|
695
|
+
Audio8 reports its codec path in `response.stats`: `codecSidecarLoaded` is 1
|
|
696
|
+
while the sidecar is attached and `codecOnCoreml` is 1 when that synthesis
|
|
697
|
+
ran its codec on it (see [Response shape](#response-shape)). The Supertonic
|
|
698
|
+
vocoder path is not reported in the stats.
|
|
699
|
+
|
|
700
|
+
Worth it where the GPU is consumer-class: on an Apple M4 the Supertonic
|
|
701
|
+
vocoder runs 1.6-2.9x faster on the Neural Engine than on Metal (1.06-1.13x
|
|
702
|
+
end to end). On workstation parts the GPU wins — an M3 Ultra is 0.6-0.9x —
|
|
703
|
+
so do not stage a sidecar there. `q4_0` models ignore the vocoder sidecar:
|
|
704
|
+
it carries full-precision weights and would substitute a different vocoder
|
|
705
|
+
rather than accelerate the quantized one.
|
|
706
|
+
|
|
707
|
+
Export the sidecars from the model GGUFs with
|
|
708
|
+
`engines/tts/scripts/export-supertonic-coreml.py` and
|
|
709
|
+
`engines/tts/scripts/export-audio8-codec-coreml.py` from the
|
|
710
|
+
[`qvac-fabric-speech.cpp`](https://github.com/tetherto/qvac-fabric-speech.cpp)
|
|
711
|
+
tree at the ref `speech-cpp` pins. Its
|
|
712
|
+
[Supertonic](https://github.com/tetherto/qvac-fabric-speech.cpp/blob/master/engines/tts/docs/supertonic.md#core-ml-vocoder-sidecar)
|
|
713
|
+
and
|
|
714
|
+
[Audio8](https://github.com/tetherto/qvac-fabric-speech.cpp/blob/master/engines/tts/docs/audio8.md#core-ml-codec-sidecar)
|
|
715
|
+
guides cover export, placement, and measurements.
|
|
716
|
+
|
|
590
717
|
When the addon is built with `ENABLE_CUDA` — on in the published linux-x64
|
|
591
718
|
prebuilds, opt-in on linux-arm64 and win32-x64 (`npm run build:cuda` or
|
|
592
719
|
`bare-make generate -D ENABLE_CUDA=ON`) — the CUDA backend ships as a
|
|
@@ -658,7 +785,9 @@ and they are set the same way: the constructor and `reload()` everywhere, plus
|
|
|
658
785
|
per call on the engines that can change them per call. The vocabulary is owned
|
|
659
786
|
by tts-cpp (`include/tts-cpp/voice_controls.h`) and mirrored here; each engine
|
|
660
787
|
declares the subset it supports, and an unsupported value throws naming that
|
|
661
|
-
engine's set.
|
|
788
|
+
engine's set. `TTSGgml.getVoiceControls()` returns tts-cpp's vocabulary and
|
|
789
|
+
each engine's subset straight from the native library, without loading a model
|
|
790
|
+
(`{ emotions, paces, engines: { parler: { emotions, paces }, … } }`).
|
|
662
791
|
|
|
663
792
|
| engine | `emotion` | `pace` | per call | exact rate knob |
|
|
664
793
|
|---|---|---|---|---|
|
|
@@ -667,6 +796,7 @@ engine's set.
|
|
|
667
796
|
| Supertonic | not supported | slow / moderate / fast | no | `speed` |
|
|
668
797
|
| Chatterbox | not supported | not supported | — | `speed` |
|
|
669
798
|
| Audio8 | not supported | not supported | — | — |
|
|
799
|
+
| MOSS | not supported | not supported | — | — |
|
|
670
800
|
|
|
671
801
|
The 12 canonical emotions (case-insensitive): `command`, `anger`, `narration`,
|
|
672
802
|
`conversation`, `disgust`, `fear`, `happy`, `neutral`, `proper noun`, `news`,
|
|
@@ -801,6 +931,89 @@ available, the engine falls back to CPU and sets
|
|
|
801
931
|
`response.stats.gpuUnsupported`. Voice cloning uses the same backend and adds
|
|
802
932
|
a one-off encode when a new reference recording is supplied.
|
|
803
933
|
|
|
934
|
+
## MOSS
|
|
935
|
+
|
|
936
|
+
MOSS Delay is an autoregressive model that predicts 32 RVQ codebooks per 80 ms
|
|
937
|
+
frame on a delay pattern, and a transformer codec turns those codes back into
|
|
938
|
+
24 kHz audio. The backbone and the codec's synthesis half run on every
|
|
939
|
+
synthesis, the analysis half only to clone a voice or run a dialogue, so a
|
|
940
|
+
text-only deployment can omit `files.mossCodecEncoder`. The backbones have 8B
|
|
941
|
+
parameters, so MOSS targets desktop hosts; mobile is not supported.
|
|
942
|
+
|
|
943
|
+
```js
|
|
944
|
+
const model = new TTSGgml({
|
|
945
|
+
engine: TTSGgml.ENGINE_MOSS,
|
|
946
|
+
files: { modelDir: './models' },
|
|
947
|
+
referenceAudio: './voice.wav', // optional; 24 kHz, needs moss-codec-encoder-*.gguf
|
|
948
|
+
streamChunkTokens: 25, // optional; codec frames per chunk (2 s)
|
|
949
|
+
config: { language: 'en' },
|
|
950
|
+
opts: { stats: true }
|
|
951
|
+
})
|
|
952
|
+
await model.load()
|
|
953
|
+
await model.run({ input: 'Hello from a fully on-device pipeline.' })
|
|
954
|
+
```
|
|
955
|
+
|
|
956
|
+
`config.language` is passed to the model as a prompt hint. The reference
|
|
957
|
+
recording is encoded once at load and stays fixed for the instance, so
|
|
958
|
+
changing voices means a new instance. `streamChunkTokens > 0` delivers audio in
|
|
959
|
+
chunks of that many codec frames (12.5 per second) while the backbone is still
|
|
960
|
+
generating; `streamFirstChunkTokens` is not supported. MOSS emits 24 kHz only,
|
|
961
|
+
so `config.outputSampleRate` is rejected unless it equals `24000`, and the
|
|
962
|
+
LavaSR enhancer / denoiser are not supported.
|
|
963
|
+
|
|
964
|
+
The speech is directable from the text itself: `[pause 2.0s]` markers insert
|
|
965
|
+
a silence of roughly that length, and inline Pinyin (`ni3 hao3`) or IPA
|
|
966
|
+
(`/həloʊ/`) steers pronunciation. `durationTokens` asks for a target length
|
|
967
|
+
in codec frames (12.5 per second, so `38` is about 3 s; `0` keeps the length
|
|
968
|
+
free). It is reloadable, and the engine rejects a target that does not fit in
|
|
969
|
+
its generation budget.
|
|
970
|
+
|
|
971
|
+
```js
|
|
972
|
+
const model = new TTSGgml({
|
|
973
|
+
engine: TTSGgml.ENGINE_MOSS,
|
|
974
|
+
files: { modelDir: './models' },
|
|
975
|
+
durationTokens: 38,
|
|
976
|
+
config: { language: 'en' }
|
|
977
|
+
})
|
|
978
|
+
await model.load()
|
|
979
|
+
await model.run({ input: 'Hold on [pause 1.0s] here it comes.' })
|
|
980
|
+
```
|
|
981
|
+
|
|
982
|
+
Dialogue needs the MOSS-TTSD backbone: with `dialogueReferences` set, a
|
|
983
|
+
`modelDir` must hold `moss-ttsd-*.gguf` (or name the file with
|
|
984
|
+
`files.mossBackbone`). A `modelDir` holding only the TTSD backbone also serves
|
|
985
|
+
plain synthesis. `dialogueReferences` takes one 24 kHz recording per
|
|
986
|
+
speaker, in the order the text tags them with `[S1]`, `[S2]`, and so on. The
|
|
987
|
+
model continues the references, so the text must open with what each
|
|
988
|
+
recording says, under its speaker tag, followed by the lines to generate; only
|
|
989
|
+
the new lines come out as audio. For the same reason a dialogue cannot be split
|
|
990
|
+
into sentences: `runStream()`, `runStreaming()` and `run({ streamOutput: true })`
|
|
991
|
+
are rejected; use `run()`, with `streamChunkTokens` for chunked audio. The references need
|
|
992
|
+
`files.mossCodecEncoder`, exclude `referenceAudio`, and are fixed for the
|
|
993
|
+
instance. They stay in the codec as causal history, so the first generated
|
|
994
|
+
words continue the reference voices without a seam.
|
|
995
|
+
|
|
996
|
+
```js
|
|
997
|
+
const dialogue = new TTSGgml({
|
|
998
|
+
engine: TTSGgml.ENGINE_MOSS,
|
|
999
|
+
files: { modelDir: './models' },
|
|
1000
|
+
dialogueReferences: ['./alice.wav', './bob.wav'],
|
|
1001
|
+
config: { language: 'en' }
|
|
1002
|
+
})
|
|
1003
|
+
await dialogue.load()
|
|
1004
|
+
await dialogue.run({
|
|
1005
|
+
input:
|
|
1006
|
+
'[S1] What alice.wav says. [S2] What bob.wav says. ' +
|
|
1007
|
+
'[S1] Did the build finish? [S2] Yes, every test passed.'
|
|
1008
|
+
})
|
|
1009
|
+
```
|
|
1010
|
+
|
|
1011
|
+
MOSS runs on CPU by default. `config.useGPU: true` (or a non-zero
|
|
1012
|
+
`nGpuLayers`) asks the engine for a GPU backend; when none is available it
|
|
1013
|
+
falls back to CPU and sets `response.stats.gpuUnsupported`. `backendsDir`
|
|
1014
|
+
reaches MOSS like every other engine, so builds that load ggml backends at
|
|
1015
|
+
runtime find them.
|
|
1016
|
+
|
|
804
1017
|
## API overview
|
|
805
1018
|
|
|
806
1019
|
### Constructor — `new TTSGgml(options)`
|
|
@@ -816,47 +1029,65 @@ a one-off encode when a new reference recording is supplied.
|
|
|
816
1029
|
| `files.cosyvoiceS3tokModel` | string | — | CosyVoice3 speech_tokenizer_v3 GGUF; needed only with `referenceAudio` (auto-discovered in the model dir as `cosyvoice3-s3tok-*.gguf`) |
|
|
817
1030
|
| `files.cosyvoiceCampplusModel` | string | — | CosyVoice3 CAM++ speaker-encoder GGUF; needed only with `referenceAudio` (auto-discovered as `cosyvoice3-campplus-*.gguf`) |
|
|
818
1031
|
| `files.cosyvoiceLlmModelPath` / `cosyvoiceFlowModelPath` / `cosyvoiceHiftModelPath` | string | — | Per-component overrides for the CosyVoice3 model dir |
|
|
1032
|
+
| `files.cosyvoiceVocab` / `cosyvoiceMerges` / `cosyvoiceVoiceModel` | string | — | CosyVoice3 text frontend (`vocab.json`, `merges.txt`) and baked voice (`voice.gguf`) overrides; each resolves under the model dir when unset. `cosyvoiceVoiceModel` selects a different baked voice without a separate model dir (`*Path` aliases accepted) |
|
|
819
1033
|
| `files.audio8Lm` | string | — | Audio8 DualAR language model GGUF (overrides `modelDir`) |
|
|
820
1034
|
| `files.audio8CodecDecoder`| string | — | Audio8 codec synthesis half — codes to wav (overrides `modelDir`) |
|
|
821
1035
|
| `files.audio8CodecEncoder`| string | — | Audio8 codec analysis half — wav to codes; only needed to clone a voice |
|
|
1036
|
+
| `files.mossBackbone` | string | — | MOSS Delay backbone GGUF: MOSS-TTS or the MOSS-TTSD dialogue checkpoint (overrides `modelDir`) |
|
|
1037
|
+
| `files.mossCodecDecoder` | string | — | MOSS codec synthesis half — codes to wav (overrides `modelDir`) |
|
|
1038
|
+
| `files.mossCodecEncoder` | string | — | MOSS codec analysis half — wav to codes; only needed to clone a voice or for `dialogueReferences` |
|
|
822
1039
|
| `files.lavasrEnhancer` | string | — | LavaSR enhancer GGUF — supplying it turns on 48 kHz enhancement |
|
|
823
1040
|
| `files.lavasrDenoiser` | string | — | LavaSR denoiser GGUF — supplying it turns on denoising (batch only) |
|
|
824
|
-
| `engine` | string | auto | Force `'chatterbox'`, `'supertonic'`, `'cosyvoice3'`, `'parler'` or `'
|
|
825
|
-
| `referenceAudio` | string | — | Wav to clone (Chatterbox: mono, ≥ 5 s; CosyVoice3: 0.5-30 s, multichannel downmixed to mono, needs the s3tok + campplus GGUFs; Audio8: also needs `referenceText`). Audio8 accepts it per call too |
|
|
1041
|
+
| `engine` | string | auto | Force `'chatterbox'`, `'supertonic'`, `'cosyvoice3'`, `'parler'`, `'audio8'` or `'moss'` (`TTSGgml.ENGINE_CHATTERBOX` / `ENGINE_SUPERTONIC` / `ENGINE_COSYVOICE3` / `ENGINE_PARLER` / `ENGINE_AUDIO8` / `ENGINE_MOSS`); auto-detected from the GGUFs present otherwise |
|
|
1042
|
+
| `referenceAudio` | string | — | Wav to clone (Chatterbox: mono, ≥ 5 s; CosyVoice3: 0.5-30 s, multichannel downmixed to mono, needs the s3tok + campplus GGUFs; Audio8: also needs `referenceText`; MOSS: needs `files.mossCodecEncoder`). Audio8 accepts it per call too |
|
|
826
1043
|
| `referenceText` | string | — | Audio8-only: what `referenceAudio` says, verbatim. Required whenever a reference is set; accepted per call |
|
|
1044
|
+
| `dialogueReferences` | string[] | — | MOSS-only: one 24 kHz recording per speaker (`[S1]`, `[S2]`, ...) for dialogue synthesis; the text must open with each recording's transcript under its tag; needs `files.mossCodecEncoder`, excludes `referenceAudio`, fixed per instance |
|
|
1045
|
+
| `durationTokens` | number | `0` | MOSS-only: target length in codec frames (12.5 per second); `0` keeps the length free; reloadable |
|
|
827
1046
|
| `voiceDir` | string | — | Pre-baked voice profile |
|
|
828
|
-
| `seed` | number | 42 | RNG seed (CFM noise + sampling) |
|
|
1047
|
+
| `seed` | number | 42 | RNG seed (CFM noise + sampling); MOSS defaults to its engine's 1234 |
|
|
829
1048
|
| `nGpuLayers` | number | 0 | Layers offloaded to GPU (mirrors `useGPU`; pass `99` to offload all) |
|
|
830
1049
|
| `nCtx` | number | 4096 | Chatterbox T3 context limit for the prompt plus generated speech tokens (25 tokens ≈ 1 s of audio). The KV cache is allocated up front at this length, so the 4096-token default directly bounds memory. Pass `0` to use the GGUF metadata value |
|
|
831
1050
|
| `kvCacheType` | string | `f16` | T3 KV-cache dtype: `f32` \| `f16` \| `q8_0`. `f16` (~50% of f32) is the safe cross-backend default. `q8_0` stores the cache at ~27% of f32 and decodes 20-30% faster on Metal, but only works on backends with a q8_0 CONT op (CPU, CUDA) — it hard-aborts the multilingual model on Metal, so it is opt-in. Turbo greedy decoding is byte-identical across all three (upstream-validated). Pass `f32` for bit-exact pre-quantisation behaviour |
|
|
832
|
-
| `threads` | number | hw.concurrency capped at 4 | |
|
|
833
|
-
| `streamChunkTokens` | number | 0 | **>0 enables native chunk streaming** |
|
|
1051
|
+
| `threads` | number | hw.concurrency capped at 4 | MOSS: engine default of 4; `0` keeps it |
|
|
1052
|
+
| `streamChunkTokens` | number | 0 | **>0 enables native chunk streaming** (speech tokens per chunk; Supertonic: text tokens; MOSS: codec frames per chunk, 12.5 per second) |
|
|
834
1053
|
| `streamFirstChunkTokens` | number | = streamChunkTokens | Smaller first chunk for low first-audio-out |
|
|
835
|
-
| `
|
|
1054
|
+
| `streamChunkTolerancePct` | number | 20 | Supertonic-only: boundary-snap window for clause / space chunk boundaries, in percent of the chunk target |
|
|
1055
|
+
| `streamMinChunkTokens` | number | 30 | Supertonic-only: floor on every streamed chunk's size in text tokens; shorter trailing chunks merge into the previous one |
|
|
1056
|
+
| `streamLeftContextTokens` | number | 0 | Chatterbox: left-context speech tokens S3Gen keeps per streamed chunk (typically 25-50), bounding per-chunk cost on long utterances; `0` = the whole emitted prefix. CosyVoice3: reserved — the pinned engine accepts but does not read it |
|
|
1057
|
+
| `cfmSteps` | number | 2 | Chatterbox native-streaming CFM steps: 1 = faster (halved CFM cost) |
|
|
1058
|
+
| `batchCfmSteps` | number | engine default | Chatterbox-only batch CFM steps (`0` = 2-step meanflow on Turbo, the model's full schedule on multilingual). The multilingual engine does not floor it, so values well below its schedule under-integrate |
|
|
1059
|
+
| `nPredict` | number | 1000 | Chatterbox-only cap on speech tokens per synthesis (25 ≈ 1 s, so ~40 s by default). Longer single calls also need `nCtx` to fit, or use `maxSentenceChars` |
|
|
1060
|
+
| `maxSentenceChars` | number | 0 | Chatterbox-only sentence auto-split: T3 + S3Gen run per segment of at most this many bytes (the reference CLI uses 180), so `nPredict` applies per segment |
|
|
1061
|
+
| `crossfadeMs` | number | 30 | Chatterbox-only raised-cosine crossfade between auto-split batch segments; native streaming stays gapless |
|
|
1062
|
+
| `repeatPenalty` | number | 1.2 | Chatterbox-only T3 repetition penalty (`1` = off) |
|
|
1063
|
+
| `exaggeration` / `cfgWeight` / `minP` | number | 0.5 / 0.5 / 0 | Chatterbox multilingual only (Turbo ignores them): prosody intensity (> 0 more expressive, < 0 flatter), T3 classifier-free guidance scale (`0` = off; `cfgRate` is the separate S3Gen rate), and the min-p sampling threshold |
|
|
836
1064
|
| `speed` | number | 1.0 | Speaking-rate multiplier, bounded `[0.25, 4.0]` (`< 1` slower, `> 1` faster). Both engines |
|
|
837
1065
|
| `voice` / `voiceName` | string | — | Supertonic voice id (e.g. `'F1'`, `'M1'`); Parler template speaker name (e.g. `'Laura'`, `'Rohit'`) |
|
|
1066
|
+
| `voiceJsonPath` | string | — | Supertonic-only: external voice JSON (`{ style_ttl, style_dp }`, e.g. a cloned voice) that overrides `voice`; the engine checks the tensor sizes against the model |
|
|
1067
|
+
| `prewarmText` | string | — | Supertonic-only: text synthesized once at load so GPU pipelines compile, and a Core ML vocoder sidecar specializes, before the first `run()` (skipped on a plain CPU run); use a representative length. Wins over the `vulkanCacheDir` default pre-warm |
|
|
1068
|
+
| `vulkanDevice` | number | 0 | Supertonic / CosyVoice3 Vulkan adapter, also used by their LavaSR enhancer: `0` = first, `N` = the Nth, `-1` = auto-pick by free VRAM preferring a discrete GPU |
|
|
1069
|
+
| `flowCutPrompt` | boolean | `false` | CosyVoice3-only: attention-only prompt frames in the flow — faster, with output that deviates slightly from the reference |
|
|
838
1070
|
| `steps` / `numInferenceSteps` | number | GGUF default | Supertonic vector-estimator CFM steps (`0` = GGUF default) |
|
|
839
1071
|
| `noiseNpyPath` | string | — | Supertonic: optional fixed CFM noise `.npy` for reproducibility |
|
|
840
1072
|
| `description` / `voiceDescription` | string | fallback caption | Parler-only: full free-text voice description (mutually exclusive with the template fields) |
|
|
841
1073
|
| `emotion`, `pace` | string | — | Cross-engine conditioning (see [Emotion & pace](#emotion--pace-cross-engine)); an unsupported value errors listing what that engine supports |
|
|
842
1074
|
| `pitch`, `expressivity`, `noise`, `reverb`, `quality` | string | — | Parler-only template fields (see [Parler descriptions](#parler-descriptions)); invalid values error listing the valid set |
|
|
843
|
-
| `temperature` / `topK` / `topP` | number | engine defaults | Parler
|
|
1075
|
+
| `temperature` / `topK` / `topP` | number | engine defaults | Parler, Audio8 and Chatterbox sampling knobs (omit for the engine's own defaults — Parler temp 1.0 / top-k 50, Audio8 temp 0.7 / top-k 50 / top-p 0.9, Chatterbox temp 0.8 / top-k 1000 / top-p 0.95 where temp `0` is greedy and top-k `0` disables the cutoff; `topP` in `(0, 1]`) |
|
|
844
1076
|
| `maxFrames` | number | engine max | Parler + Audio8 generation cap in frames (Parler ~86/s of audio, Audio8 ~21.5/s); `0` = model default, Parler rejects 1–9 |
|
|
845
1077
|
| `greedy` | boolean | `false` | Audio8-only: take the argmax instead of sampling; ignores `temperature` / `topK` / `topP` |
|
|
846
1078
|
| `minNewTokens` | number | GGUF default | Parler-only minimum tokens before EOS (`-1` = model default) |
|
|
847
1079
|
| `normalizeNumbers` | boolean | `true` | Parler-only: expand digits before tokenization (English words; script-native digits on indic) — parler voices raw digits badly |
|
|
848
|
-
| `instruct` | object \| string | — | CosyVoice3-only: instruction controls (`dialect` / `volume` / `style`, resolved by precedence in that order) or a raw instruction string; an unknown key or invalid value throws, and it counts toward the one-instruction rule (see [CosyVoice3 instruct](#cosyvoice3-instruct)) |
|
|
1080
|
+
| `instruct` | object \| string | — | CosyVoice3-only: instruction controls (`dialect` / `volume` / `style`, resolved by precedence in that order) or a raw instruction string; an unknown key or invalid value throws, and it counts toward the one-instruction rule (see [CosyVoice3 instruct](#cosyvoice3-instruct)). Also accepted per call, where it replaces the configured conditioning like a per-call `emotion` / `pace` |
|
|
849
1081
|
| `promptText` | string | — | CosyVoice3-only: verbatim transcript of `referenceAudio` — set for zero-shot cloning, omit for cross-lingual; without a reference it overrides the baked voice's transcript |
|
|
850
|
-
| `streamLeftContextTokens` | number | — | CosyVoice3-only: intended native chunk-streaming left-context tokens. Reserved / not yet effective — the pinned engine accepts but does not read it |
|
|
851
1082
|
| `mecabDictDir` | string | — | Chatterbox MTL Japanese (`ja`): compiled MeCab/IPAdic dictionary directory |
|
|
852
1083
|
| `cangjieTsvPath` | string | — | Chatterbox MTL Chinese (`zh`): `Cangjie5_TC` TSV path |
|
|
853
1084
|
| `backendsDir` | string | `resolveBackendsDir()` | Root dir the addon scans for dynamically-loaded ggml backend `.so` files. Defaults to the package's own `prebuilds/` when present, otherwise the installed platform package. Required on Android when backends ship elsewhere (e.g. inside the APK); ignored on platforms that statically link the backend |
|
|
854
1085
|
| `openclCacheDir` | string | unset | Android-only: directory where the OpenCL backend persists its compiled program-binary cache. Setting it across runs avoids re-JITing the kernels on every fresh process |
|
|
855
1086
|
| `vulkanCacheDir` | string | unset | Supertonic + `useGPU: true` only: writable directory where the Vulkan backend persists its compiled pipeline cache (`GGML_VK_PIPELINE_CACHE_DIR`). Moves the one-time first-dispatch pipeline-compile cost (seconds on Mali) off the first `run()` — paid once per install instead of once per process — and enables a load-time pre-warm. Fully opt-in: unset -> no cross-process cache, no pre-warm, behaviour unchanged |
|
|
856
|
-
| `config.language` | string | `"en"` | Chatterbox MTL accepts `es/fr/de/pt/it/zh/ja/ko/...`; turbo & Supertonic are English |
|
|
857
|
-
| `config.useGPU` | boolean | `false` | Set to `true` to route through Metal / CUDA / Vulkan / OpenCL if available. Honored for Chatterbox/Supertonic on GPU-capable hosts (including Android, per `tts-cpp`'s per-vendor allowlist); Parler is validated on Apple/Metal, linux CUDA, and Android/ARM Mali Vulkan; CosyVoice3 and Audio8 offload on Apple/Metal, desktop linux CUDA/Vulkan, Windows Vulkan, and Android OpenCL/Adreno. Unsupported backends fall back to CPU. See [Backends & GPU acceleration](#backends--gpu-acceleration) |
|
|
858
|
-
| `config.outputSampleRate` | number | — (engine-native) | Resample the output to this rate (8000–192000 Hz). Omit to keep the engine-native rate (Chatterbox 24 kHz, Supertonic / Parler / Audio8 44.1 kHz, CosyVoice3 24 kHz, enhancer 48 kHz). Parler native chunk streaming accepts a non-native rate only with the enhancer active |
|
|
859
|
-
| `opts.stats` | boolean | `false` | Populate `response.stats` with RTF, `backendDevice` (0=CPU, 1=GPU), `backendId` (0=CPU, 1=Metal, 2=CUDA, 3=Vulkan, 4=OpenCL, 99=other), and
|
|
1087
|
+
| `config.language` | string | `"en"` | Chatterbox MTL accepts `es/fr/de/pt/it/zh/ja/ko/...`; turbo & Supertonic are English; MOSS takes it as a prompt hint (`en`, `zh`, ...) |
|
|
1088
|
+
| `config.useGPU` | boolean | `false` | Set to `true` to route through Metal / CUDA / Vulkan / OpenCL if available. Honored for Chatterbox/Supertonic on GPU-capable hosts (including Android, per `tts-cpp`'s per-vendor allowlist); Parler is validated on Apple/Metal, linux CUDA, and Android/ARM Mali Vulkan; CosyVoice3 and Audio8 offload on Apple/Metal, desktop linux CUDA/Vulkan, Windows Vulkan, and Android OpenCL/Adreno; MOSS GPU offload is validated on Apple/Metal. Unsupported backends fall back to CPU. See [Backends & GPU acceleration](#backends--gpu-acceleration) |
|
|
1089
|
+
| `config.outputSampleRate` | number | — (engine-native) | Resample the output to this rate (8000–192000 Hz). Omit to keep the engine-native rate (Chatterbox 24 kHz, Supertonic / Parler / Audio8 44.1 kHz, CosyVoice3 24 kHz, MOSS 24 kHz, enhancer 48 kHz). Parler native chunk streaming accepts a non-native rate only with the enhancer active; MOSS accepts only its native rate |
|
|
1090
|
+
| `opts.stats` | boolean | `false` | Populate `response.stats` with RTF, `backendDevice` (0=CPU, 1=GPU), `backendId` (0=CPU, 1=Metal, 2=CUDA, 3=Vulkan, 4=OpenCL, 99=other), `enhancerBackendDevice` / `enhancerBackendId` and `denoiserBackendDevice` / `denoiserBackendId` (`-1` when that LavaSR stage is off), and the engine's own stage stats: Chatterbox `t3Ms` / `s3genMs` / `t3Tokens`; CosyVoice3 per-stage ms (`lmPrefillMs` … `hiftDecodeMs`, `stageTotalMs`) and work counters (`speechTokens`, `decodeSteps`, …); Audio8 per-stage ms (`prefillMs`, `fastDecodeMs`, `codecSynthMs`, …, `stageTotalMs`) |
|
|
860
1091
|
| `exclusiveRun` | boolean | `false` | **Top-level** option (not under `opts`): serialize overlapping streaming runs |
|
|
861
1092
|
|
|
862
1093
|
### Methods
|
|
@@ -869,6 +1100,8 @@ a one-off encode when a new reference recording is supplied.
|
|
|
869
1100
|
config (`language`, `useGPU`, `outputSampleRate`, …).
|
|
870
1101
|
- `await model.destroy()` — `unload()` + mark this instance dead.
|
|
871
1102
|
- `await model.cancel()` — best-effort cancel of any in-flight run.
|
|
1103
|
+
- `TTSGgml.getVoiceControls()` — static: tts-cpp's emotion / pace vocabulary
|
|
1104
|
+
and each engine's supported subset (see [Emotion & pace](#emotion--pace-cross-engine)).
|
|
872
1105
|
- `model.run({ input, type: 'text' })` → `QvacResponse`.
|
|
873
1106
|
- `model.run({ input, streamOutput: true })` → sentence-chunked
|
|
874
1107
|
synthesis driven by the JS-side sentence splitter (see
|
|
@@ -895,7 +1128,7 @@ All `run*` methods return a `QvacResponse` (from `@qvac/infer-base`):
|
|
|
895
1128
|
```js
|
|
896
1129
|
response.onUpdate(data => {
|
|
897
1130
|
data.outputArray // Int16Array — mono PCM
|
|
898
|
-
data.sampleRate // actual rate: Chatterbox 24000, Supertonic/Parler/Audio8 44100, enhancer 48000
|
|
1131
|
+
data.sampleRate // actual rate: Chatterbox/CosyVoice3/MOSS 24000, Supertonic/Parler/Audio8 44100, enhancer 48000
|
|
899
1132
|
data.chunkIndex // present on sentence-streaming events only
|
|
900
1133
|
data.sentenceChunk // present on sentence-streaming events only
|
|
901
1134
|
})
|
|
@@ -906,8 +1139,10 @@ response.stats.totalTime // seconds
|
|
|
906
1139
|
response.stats.realTimeFactor // synthesis time / audio duration
|
|
907
1140
|
response.stats.audioDurationMs
|
|
908
1141
|
response.stats.totalSamples
|
|
909
|
-
response.stats.tokensPerSecond // Audio8
|
|
910
|
-
response.stats.generatedFrames // Audio8 only: codec frames
|
|
1142
|
+
response.stats.tokensPerSecond // Audio8 and MOSS count codec frames, the others characters
|
|
1143
|
+
response.stats.generatedFrames // Audio8 and MOSS only: codec frames (Audio8 every 46 ms, MOSS every 80 ms)
|
|
1144
|
+
response.stats.codecSidecarLoaded // Audio8 only, macOS / iOS: 1 while the Apple Core ML codec sidecar is attached
|
|
1145
|
+
response.stats.codecOnCoreml // Audio8 only: 1 when this synthesis ran the codec on that sidecar, 0 when on the ggml backend
|
|
911
1146
|
response.stats.backendDevice // 0=CPU, 1=GPU
|
|
912
1147
|
response.stats.backendId // 0=CPU, 1=Metal, 2=CUDA, 3=Vulkan, 4=OpenCL, 99=other
|
|
913
1148
|
// present when a LavaSR enhancer is active:
|
|
@@ -915,6 +1150,11 @@ response.stats.enhancerBackendDevice // -1 none, 0 CPU, 1 GPU
|
|
|
915
1150
|
response.stats.enhancerBackendId
|
|
916
1151
|
```
|
|
917
1152
|
|
|
1153
|
+
For `runStream()` and `runStreaming()`, `codecSidecarLoaded` and
|
|
1154
|
+
`codecOnCoreml` retain the last chunk that reported each field, including zero
|
|
1155
|
+
when a sidecar is retired or synthesis falls back to ggml. They are not summed;
|
|
1156
|
+
if no chunk reports a field, it remains absent.
|
|
1157
|
+
|
|
918
1158
|
### Text helpers
|
|
919
1159
|
|
|
920
1160
|
The sentence splitter is available from
|
|
@@ -948,6 +1188,57 @@ const {
|
|
|
948
1188
|
| 13010 | `FAILED_TO_STOP` |
|
|
949
1189
|
| 13011 | `JOB_ALREADY_RUNNING` |
|
|
950
1190
|
|
|
1191
|
+
## Assessing fit
|
|
1192
|
+
|
|
1193
|
+
`assessFit` projects a load against the memory free right now. It reads GGUF metadata and never weight data, so the registry's weightless copy of each file answers the same as the file itself and the projection can run before anything is downloaded. It is a module export, not an instance method: nothing is loaded to call it.
|
|
1194
|
+
|
|
1195
|
+
A request is a `createInstance` config plus the workload, so it carries the same keys the load takes and the engine's own config builder reads it.
|
|
1196
|
+
|
|
1197
|
+
```js
|
|
1198
|
+
const TTSGgml = require('@qvac/tts-ggml')
|
|
1199
|
+
|
|
1200
|
+
const fit = TTSGgml.assessFit({
|
|
1201
|
+
engineType: 'supertonic',
|
|
1202
|
+
supertonicModelPath: '/models/supertonic.gguf',
|
|
1203
|
+
useGPU: true,
|
|
1204
|
+
textTokens: 256,
|
|
1205
|
+
audioSeconds: 30
|
|
1206
|
+
})
|
|
1207
|
+
|
|
1208
|
+
fit.status // 'fits' | 'does-not-fit' | 'error'
|
|
1209
|
+
fit.reason // the engine's own wording, e.g. 'model-unreadable', 'workload-too-large'
|
|
1210
|
+
fit.modelVariant // which pipeline was projected, e.g. 'chatterbox-t3-turbo'
|
|
1211
|
+
fit.deviceName
|
|
1212
|
+
fit.deviceBytes
|
|
1213
|
+
fit.weightsBytes
|
|
1214
|
+
fit.stateBytes
|
|
1215
|
+
fit.lmComputeBytes // 0 for a pipeline with no language-model stage, such as supertonic
|
|
1216
|
+
fit.codecComputeBytes // 0 for a pipeline with no separate codec stage
|
|
1217
|
+
fit.hostBytes
|
|
1218
|
+
fit.lavasrFileBytes // the LavaSR stages' size on disk, 0 when none was named
|
|
1219
|
+
fit.report
|
|
1220
|
+
```
|
|
1221
|
+
|
|
1222
|
+
`engineType` picks the fitter, the same key a load uses, and is required: a fit request carries none of the file keys a load is inferred from. Each engine takes the load's own file keys, plus a workload the fit adds:
|
|
1223
|
+
|
|
1224
|
+
| `engineType` | Files | Workload |
|
|
1225
|
+
| --- | --- | --- |
|
|
1226
|
+
| `supertonic` | `supertonicModelPath` | `textTokens`, `audioSeconds`, `precision`, `f16Weights` |
|
|
1227
|
+
| `parler` | `parlerModelPath` | `descriptionTokens`, `promptTokens`, `maxFrames` |
|
|
1228
|
+
| `chatterbox` | `t3ModelPath`, `s3genModelPath` | `textTokens`, `predictTokens` |
|
|
1229
|
+
| `audio8` | `audio8LmPath`, `audio8CodecDecoderPath`, `audio8CodecEncoderPath` | `promptTokens`, `maxFrames`, `referenceSeconds` |
|
|
1230
|
+
| `cosyvoice3` | `cosyvoiceLlmModelPath`, `cosyvoiceFlowModelPath`, `cosyvoiceHiftModelPath`, `cosyvoiceVoiceModelPath` | `textTokens`, `speechTokens` |
|
|
1231
|
+
|
|
1232
|
+
Everything else comes from the load config: `nGpuLayers` and `useGPU` carry the offload intent, `nCtx` and `kvCacheType` size the chatterbox cache, `steps` takes the GGUF's own default at 0, and `vulkanDevice` and `backendsDir` place the backend. Supplying `audio8CodecEncoderPath` projects voice cloning, which the decoder alone cannot do. `marginBytes` sets the free memory that must remain for the projection to count as fitting.
|
|
1233
|
+
|
|
1234
|
+
`lavasrEnhancerPath` and `lavasrDenoiserPath` are reported as `lavasrFileBytes`, their size on disk. Those stages have no fitter, so the figure is a size on disk. It is counted in neither `deviceBytes` nor `hostBytes`.
|
|
1235
|
+
|
|
1236
|
+
`deviceSharesHostMemory` reports that the device pool is system RAM, so host bytes compete with device bytes.
|
|
1237
|
+
|
|
1238
|
+
A model the engine cannot read is `status: "error"`, as is a voice with no fitter, which reports `reason: "unsupported-engine"`. MOSS is the one voice in that state. A broken request, or a host with no native binding, throws.
|
|
1239
|
+
|
|
1240
|
+
The supertonic fitter covers the fused graph path: a validated GPU, or a CPU without the Accelerate pointwise kernels. Elsewhere it answers `compute-path-not-supported` and projects nothing.
|
|
1241
|
+
|
|
951
1242
|
## Examples
|
|
952
1243
|
|
|
953
1244
|
Runnable demos under `examples/`:
|
|
@@ -965,12 +1256,15 @@ Runnable demos under `examples/`:
|
|
|
965
1256
|
| `supertonic-mtl-tts.js` | Multilingual Supertonic synthesis |
|
|
966
1257
|
| `supertonic-mtl-sweep-tts.js` | Multilingual Supertonic sweep across languages |
|
|
967
1258
|
| `supertonic-sentence-stream-tts.js` | Supertonic sentence-level streaming |
|
|
1259
|
+
| `supertonic-chunk-stream-tts.js` | Supertonic native chunk streaming via `streamChunkTokens`, with streaming playback |
|
|
968
1260
|
| `supertonic-enhanced.js` | Supertonic + LavaSR 48 kHz enhancement. `bare examples/supertonic-enhanced.js "Hello"` |
|
|
969
1261
|
| `parler-tts.js` | Parler batch synth with voice/emotion templates. `bare examples/parler-tts.js "Hello" Laura happy` |
|
|
970
1262
|
| `parler-enhanced.js` | Parler + LavaSR 48 kHz enhancement. `bare examples/parler-enhanced.js "Hello" Laura happy` |
|
|
971
1263
|
| `cosyvoice-tts.js` | CosyVoice3 batch synth with the cross-engine emotion option (24 kHz; CPU by default, `--gpu` opts into Metal / desktop Vulkan / Android GPU). `bare examples/cosyvoice-tts.js --gpu "Hello" happy` |
|
|
972
1264
|
| `cosyvoice-enhanced.js` | CosyVoice3 + LavaSR 48 kHz enhancement (add `--denoise` for the denoiser). `bare examples/cosyvoice-enhanced.js "Hello"` |
|
|
973
1265
|
| `audio8-tts.js` | Audio8 batch synth, optionally cloning a reference. Set `QVAC_TTS_AUDIO8_GPU=1` for the desktop GPU backend (CUDA or Vulkan on linux, Vulkan on Windows). `bare examples/audio8-tts.js "Hello" voice.wav "What it says."` |
|
|
1266
|
+
| `moss-tts.js` | MOSS batch synth, optionally cloning a reference. Set `QVAC_TTS_MOSS_STREAM_FRAMES=25` for native chunk streaming, `QVAC_TTS_MOSS_DURATION=38` for a target length and `QVAC_TTS_MOSS_GPU=1` for the GPU backend. `bare examples/moss-tts.js "Hello" voice.wav` |
|
|
1267
|
+
| `moss-dialogue-tts.js` | MOSS-TTSD multi-speaker dialogue from one 24 kHz reference per speaker; the text opens with each reference's transcript. Set `QVAC_TTS_MOSS_GPU=1` for the GPU backend. `bare examples/moss-dialogue-tts.js "[S1] What alice.wav says. [S2] What bob.wav says. [S1] Hi. [S2] Hello." alice.wav bob.wav` |
|
|
974
1268
|
|
|
975
1269
|
The two streaming examples feed PCM into a single long-running
|
|
976
1270
|
`sox play` / `ffplay` process so chunks play back-to-back without any
|
|
@@ -993,8 +1287,7 @@ files are absent. They cover, across the engines:
|
|
|
993
1287
|
* batch synthesis with full RuntimeStats,
|
|
994
1288
|
* sentence-level streaming (`runStream` / `run({ streamOutput: true })`
|
|
995
1289
|
/ `runStreaming` over async iterators),
|
|
996
|
-
* native sub-sentence chunk streaming
|
|
997
|
-
`streamChunkTokens`),
|
|
1290
|
+
* native sub-sentence chunk streaming via `streamChunkTokens`,
|
|
998
1291
|
* sequential-run / fresh-instance / reload-stability behaviour,
|
|
999
1292
|
* strict GPU-backend assertion via `response.stats.backendDevice` +
|
|
1000
1293
|
`backendId` (set `NO_GPU=true` to skip on CPU-only runners,
|
|
@@ -1005,6 +1298,11 @@ files are absent. They cover, across the engines:
|
|
|
1005
1298
|
for WER against the synthesized audio (whisper-small via the
|
|
1006
1299
|
`@qvac/asr-ggml` development dependency).
|
|
1007
1300
|
|
|
1301
|
+
The MOSS integration test (`moss.test.js`) does not download its GGUFs:
|
|
1302
|
+
set `QVAC_TEST_MOSS_MODEL_DIR` to a directory holding them, otherwise it is
|
|
1303
|
+
skipped. The C++ suite runs its MOSS synthesis cases when
|
|
1304
|
+
`QVAC_TEST_MOSS_BACKBONE_GGUF` and `QVAC_TEST_MOSS_DECODER_GGUF` are set.
|
|
1305
|
+
|
|
1008
1306
|
To stress-test long inputs, set `INPUT_SENTENCES=medium` (or `long`)
|
|
1009
1307
|
and re-run the integration suite — `addon.test.js` reads the env var to
|
|
1010
1308
|
pick its sentence corpus from `test/data/sentences-{medium,long}.js`.
|
|
@@ -1071,3 +1369,7 @@ OpenCL; Xclipse and Mali use Vulkan).
|
|
|
1071
1369
|
## License
|
|
1072
1370
|
|
|
1073
1371
|
Apache-2.0. See [LICENSE](./LICENSE).
|
|
1372
|
+
|
|
1373
|
+
## Pocket TTS
|
|
1374
|
+
|
|
1375
|
+
See [Pocket TTS](docs/pocket-tts.md) for model conversion, addon/SDK usage, supported controls and validation commands.
|