@qvac/tts-ggml 0.13.0 → 0.14.1

package/README.md

@@ -17,6 +17,22 @@ LavaSR neural denoise + 48 kHz bandwidth-extension enhancement.  Unsure
 which checkpoint to stage? Start with [Choosing a model](#choosing-a-model).
 
 
+### CosyVoice3 Core ML sidecars
+
+Apple builds support optional CosyVoice3 Core ML sidecars through the native
+`speech-cpp` 2026-10-19 dependency. Export and placement details are in the
+[CosyVoice3 Core ML guide](https://github.com/tetherto/qvac-fabric-speech.cpp/blob/ea128498e23d20cd122729275e92361b11a361df/engines/tts/docs/cosyvoice3.md#core-ml-sidecars).
+Re-export existing CosyVoice3 LM sidecars with the updated exporter to include
+the cache feedback correction for consecutive predictions.
+
+The LM, flow frontend/decoder, F0 predictor and HiFT vocoder use resident
+sidecars when present and fall back to ggml when unavailable. HiFT exports
+use FP32 for waveform parity. Core ML chooses operation placement, so a
+sidecar does not guarantee ANE execution or a speed improvement.
+When a sidecar is present, `assessFit` returns an error with reason
+`coreml-memory-estimate-unavailable`; its memory usage is not projected.
+
+
 Runs in-process with a persistent native engine — the GGUFs, the S3Gen
 preload, the ggml backend, and any voice-conditioning tensors are
 loaded once and reused across every synthesis call.  GPU acceleration
@@ -38,7 +54,7 @@ and registry prerequisites described under [MOSS](#moss).
 
 - Batch synthesis (`run({ input })` → single PCM buffer).
 - **Sentence-granularity streaming** — `runStreaming(asyncIterable)`:
-  yields one audio chunk per input sentence.
+  yields audio for each input sentence; Audio8 emits PCM as codec windows finish.
 - **Native per-chunk streaming** — set `streamChunkTokens` and audio
   flows out of the C++ engine chunk-by-chunk as T3 tokens produce
   S3Gen+HiFT output; sub-second first-audio-out inside a single
@@ -675,6 +691,8 @@ Notes:
   engine runs its ggml graph on the GPU (falling back to the ggml CPU backend
   when no GPU backend initialises), a CPU engine keeps the scalar core.
   `denoiserBackendDevice` / `denoiserBackendId` in the stats report where it ran.
+- Android ARM Mali devices can use Vulkan for denoising with `config.useGPU: true`.
+  This path is validated on Pixel 9 Pro XL (Mali-G715).
 - The tts-cpp UL-UNAS forward is implemented in
   [qvac-fabric-speech.cpp#78](https://github.com/tetherto/qvac-fabric-speech.cpp/pull/78)
   (scalar CPU port, validated bit-close to the ONNX reference); it requires a
@@ -964,6 +982,19 @@ await model.load()
 await model.run({ input: 'Hello from a fully on-device pipeline.' })
 ```
 
+`runStream(text)`, `runStreaming(asyncIterable)` and
+`run({ input: text, streamOutput: true })` emit PCM as codec windows finish,
+before the current text chunk's complete waveform is ready. Ordinary `run()`
+retains batch synthesis. Native end markers have an empty `outputArray`. Use `response.await()`
+for completion and aggregate statistics, including for asynchronous text input. PCM window sizes are fixed by the codec, so
+`streamChunkTokens` and `streamFirstChunkTokens` remain unsupported.
+
+The native streaming API is included in the pinned `speech-cpp` dependency.
+Rebuild the addon to use it; an older published prebuild does not gain native
+streaming from a JavaScript update alone.
+Use [audio8-streaming.js](examples/audio8-streaming.js) to measure first PCM
+and completion through the wrapper, excluding model loading.
+
 Sampling is **repetition-aware**: a semantic token that repeats one from the
 recent window is re-drawn under a narrower nucleus at a higher temperature,
 which is what keeps the model out of the babble attractor that pure top-p
@@ -987,6 +1018,15 @@ synthesis, the analysis half only to clone a voice or run a dialogue, so a
 text-only deployment can omit `files.mossCodecEncoder`.  The backbones have 8B
 parameters, so MOSS targets desktop hosts; mobile is not supported.
 
+On Metal, the backbone projects only the text-control logits needed during
+audio generation and omits text projection during the forced drain. Embedding
+tables stay on the GPU, and the model weight type is preserved.
+
+When `dialogueReferences` contains one recording, its encoded codes also serve
+as continuation, avoiding a duplicate encoder pass during engine creation.
+Multiple references retain their joined-audio continuation encoding. Prepared
+reference codes stay cached per instance for batch and streaming synthesis.
+
 ```js
 const model = new TTSGgml({
   engine: TTSGgml.ENGINE_MOSS,
@@ -1072,20 +1112,16 @@ The combined desktop CUDA/Vulkan speech changes in
 [Vulkan #308](https://github.com/tetherto/qvac-fabric-speech.cpp/pull/308),
 including [CUDA #321](https://github.com/tetherto/qvac-fabric-speech.cpp/pull/321),
 merged to native master at `84b02694001cfea863167776dbc185474a816656`.
-This checkout consumes the merged [registry #423](https://github.com/tetherto/qvac-registry-vcpkg/pull/423)
-at `e31be272ecdf9f20970302ebf8a2943d47803001`: `speech-cpp` 2026-10-12#1
-pins that merged revision, and `ggml-speech` 2026-10-07#1 pins
-`06f1b66806626a635904f31db6914ad2d0e71197`. The earlier native `f4d4c257`
+The exact native dependency versions are declared in [vcpkg.json](vcpkg.json),
+and the registry configuration is declared in
+[vcpkg-configuration.json](vcpkg-configuration.json). The earlier native `f4d4c257`
 passed ten smaller-model CUDA/Vulkan cases; those results do not establish GPU
 validation of the final native revision or the packaged addon. Rebuild the addon
 before running its GPU tests. Native desktop results also do not establish
 Android support or speech-quality signoff.
 
-The registry `baseline` is the main commit containing #423. The shared exact
-`speech-cpp` overrides and backend implementation are already on qvac main.
-This baseline update completes the TTS registry configuration; packaged-addon
-CUDA/Vulkan inference still requires validation on provisioned GPU hosts before
-claiming a validated release.
+Packaged-addon CUDA/Vulkan inference still requires validation on provisioned
+GPU hosts before claiming a validated release.
 
 From `packages/tts-ggml`, resolve the checked-in manifest without building or
 downloading models (with `VCPKG_ROOT` set):
@@ -1128,7 +1164,29 @@ case completes. Run both files together without test filters: a successful full
 run has 12 passing tests, including the two completion checks, and no skips.
 Preflight only checks asset presence; actual model loading and inference must
 also pass. These commands cover TTS, TTSD and Speech, not MOSS-SoundEffect (SFX).
-No addon CUDA/Vulkan hardware run has been completed for this test change yet.
+To validate the fresh packaged addons on the provisioned Linux GPU runner,
+dispatch the existing integration workflow on the branch being tested:
+
+```sh
+gh workflow run integration-test-nx.yml --repo tetherto/qvac --ref <branch> -f moss_packaged_gpu=true
+```
+
+This opt-in mode retains the `release` environment gate and builds both addons
+from the dispatch SHA with build concurrency capped at two. It stages the fresh
+platform packages through `link:platform`, verifies that both CUDA and Vulkan
+plugins are present, then runs CUDA and Vulkan sequentially. Each backend must
+pass the 12 TTS/TTSD/Speech tests above, two SFX tests (including exact backend
+and no-fallback assertions for both GPU clips), and the focused ASR Spanish
+transcription test with default and hotword prompts. Fixture skips fail the run.
+CPU control cases remain part of the TTS/SFX suites.
+
+The runner stages q8_0 TTS/TTSD/Speech/SFX backbones and q5_0 Transcribe, with
+the required codec precision preserved. TTS/TTSD q8_0 are generated on the
+runner from the source checkpoints using the pinned native quantizer. The
+artifacts retain source pins, model and prebuild hashes, the fresh prebuilds,
+and per-backend logs/results; model weights are removed and never uploaded.
+This mode does not publish. A dispatched or building run is not GPU validation:
+only passing case logs from both backends establish packaged-addon coverage.
 
 Without strict mode, developer behavior is unchanged: an unpinned run accepts
 the normal GPU cascade, missing model directories skip tests, and missing
@@ -1146,6 +1204,15 @@ to 30 seconds.  It is not a speech model, and it ships as one GGUF.  Select it
 with `engine: TTSGgml.ENGINE_MOSS_SFX` or by staging `moss-sfx-*.gguf` in the
 `modelDir`.
 
+Native generation reuses fixed cross-attention K/V across diffusion steps and
+uses Metal DiT FlashAttention when the GPU supports both attention shapes.
+Memory preflight includes the request cache (360 MiB for guided generation
+with the released models) alongside the scheduler's retained peak across text
+encoding, diffusion and VAE decoding, including repeated requests.
+FlashAttention can slightly change waveform samples. See the
+[native SoundEffect guide](https://github.com/tetherto/qvac-fabric-speech.cpp/blob/33174cc8693b1c8cdb44b5ff57b0f120bc89e454/engines/tts/docs/moss.md#moss-soundeffect)
+for precision details and the M1 Ultra measurements.
+
 ```js
 const sfx = new TTSGgml({
   engine: TTSGgml.ENGINE_MOSS_SFX,
@@ -1549,6 +1616,7 @@ Runnable demos under `examples/`:
 | `parler-enhanced.js` | Parler + LavaSR 48 kHz enhancement. `bare examples/parler-enhanced.js "Hello" Laura happy` |
 | `cosyvoice-tts.js` | CosyVoice3 batch synth with the cross-engine emotion option (24 kHz; CPU by default, `--gpu` opts into Metal / desktop Vulkan / Android GPU). `bare examples/cosyvoice-tts.js --gpu "Hello" happy` |
 | `cosyvoice-enhanced.js` | CosyVoice3 + LavaSR 48 kHz enhancement (add `--denoise` for the denoiser). `bare examples/cosyvoice-enhanced.js "Hello"` |
+| `audio8-streaming.js` | Measure Audio8 wrapper batch, streaming, or duplex latency: `bare examples/audio8-streaming.js ./models stream "Hello"`. Modes: `batch`, `stream`, `duplex`. |
 | `audio8-tts.js` | Audio8 batch synth, optionally cloning a reference. Set `QVAC_TTS_AUDIO8_GPU=1` for the desktop GPU backend (CUDA or Vulkan on linux, Vulkan on Windows). `bare examples/audio8-tts.js "Hello" voice.wav "What it says."` |
 | `moss-tts.js` | MOSS batch synth, optionally cloning a reference. Set `QVAC_TTS_MOSS_STREAM_FRAMES=25` for native chunk streaming, `QVAC_TTS_MOSS_DURATION=38` for a target length and `QVAC_TTS_MOSS_GPU=1` for the GPU backend. `bare examples/moss-tts.js "Hello" voice.wav` |
 | `moss-sound-effect.js` | MOSS-SoundEffect text-to-sound-effects at 48 kHz. Set `QVAC_TTS_MOSS_SFX_GPU=1` for the GPU backend and `QVAC_TTS_MOSS_SFX_STEPS` to trade quality for time. `bare examples/moss-sound-effect.js "Rain on a tin roof." 5` |
@@ -1720,4 +1788,9 @@ Apache-2.0.  See [LICENSE](./LICENSE).
 
 ## Pocket TTS
 
+The native `speech-cpp` dependency includes Pocket Metal, CUDA, Vulkan and
+OpenCL support in the native library. The JavaScript Pocket API continues
+to require `config.useGPU: false`; updating the native dependency does not
+remove that addon validation.
+
 See [Pocket TTS](docs/pocket-tts.md) for model conversion, addon/SDK usage, supported controls and validation commands.

package/docs/pocket-tts.md

@@ -1,7 +1,9 @@
 # Pocket TTS
 
-Pocket is an English, CPU-only TTS engine implemented in the Fabric speech
-library. It streams PCM16 audio through the `@qvac/tts-ggml` addon. Python
+Pocket is an English TTS engine implemented in the Fabric speech library.
+The native library supports CPU, Metal, CUDA, Vulkan and OpenCL; the
+`@qvac/tts-ggml` JavaScript API currently accepts CPU inference only.
+It streams PCM16 audio through the addon. Python
 is used only to convert weights and benchmark the upstream reference; it is
 not needed for native synthesis. Inference plugin and public SDK integration
 will follow in a separate change after the addon release.
@@ -139,22 +141,22 @@ audio and broad subjective quality evaluations remain outside this coverage.
 
 ## Native dependencies and measured performance
 
-The manifest requires speech-cpp and ggml-speech `2026-09-15` without changing
-`vcpkg-configuration.json`. These registered versions select the merged source
-commits `a976c4d63195601fcff8bfc0c2cdf3ef36fba539` (speech-cpp) and
-`59a0ca2c2bcd7de36c27d930f79e0c94cc0c2ff3` (ggml-speech).
-[Registry #367](https://github.com/tetherto/qvac-registry-vcpkg/pull/367)
-contains these packages. Pocket resolves CPU memory planning through the
+The exact native dependency versions are declared in [vcpkg.json](../vcpkg.json).
+The Pocket GPU implementation comes from the merged
+[Fabric #338](https://github.com/tetherto/qvac-fabric-speech.cpp/pull/338)
+change, registered in
+[registry #433](https://github.com/tetherto/qvac-registry-vcpkg/pull/433).
+Pocket resolves CPU memory planning through the
 backend registry, fixing the Linux/Android unresolved `ggml_graph_plan` import.
 The native fixes are merged in [ggml #92](https://github.com/tetherto/qvac-ext-ggml/pull/92)
 and [speech #251](https://github.com/tetherto/qvac-fabric-speech.cpp/pull/251).
 
-Before the addon-only review revision, these exact native port trees passed
+Before the addon-only review revision, the original CPU-only native dependency versions passed
 [the full prebuild and integration run](https://github.com/tetherto/qvac/actions/runs/34975293366)
 at Fabric `49d864d2e1a2070a4cb9872cd0f62319fdb9e839`: nine platform prebuilds
 and seven desktop integration lanes covering CPU, Metal, Vulkan and CUDA.
 That run validates the native dependencies; it does not validate subsequent
-JavaScript review changes. The same native pins also passed all 10 Pocket
+JavaScript review changes. Those original native pins also passed all 10 Pocket
 CTests with upstream numerical fixtures, static/dynamic CPU planner tests,
 and 116 selected Supertonic Metal operations against CPU. Six paired
 old-versus-merged native audio cases produced bit-identical PCM; this is

package/examples/audio8-streaming.js

@@ -0,0 +1,74 @@
+'use strict'
+
+const TTSGgml = require('..')
+const process = require('bare-process')
+
+const DEFAULT_TEXT = 'The first audio should arrive while the rest of this response is generated.'
+const MODES = ['batch', 'stream', 'duplex']
+
+async function* textSource(text) {
+  yield text
+}
+
+function start(model, mode, text) {
+  if (mode === 'batch') return model.run({ input: text })
+  if (mode === 'stream') return model.runStream(text, { maxChunkScalars: Array.from(text).length })
+  return model.runStreaming(textSource(text), { accumulateSentences: false })
+}
+
+async function measure(model, mode, text) {
+  const started = Date.now()
+  let firstAudioMs = null
+  let samples = 0
+  let sampleRate = 0
+  let chunks = 0
+  const response = await start(model, mode, text)
+  response.onUpdate((data) => {
+    if (!data.outputArray?.length) return
+    firstAudioMs ??= Date.now() - started
+    samples += data.outputArray.length
+    sampleRate = data.sampleRate
+    chunks++
+  })
+  await response.await()
+  if (firstAudioMs === null || sampleRate <= 0) throw new Error('Synthesis returned no audio')
+  console.log(
+    JSON.stringify({
+      mode,
+      firstAudioMs,
+      totalMs: Date.now() - started,
+      audioSeconds: samples / sampleRate,
+      chunks,
+      stats: response.stats
+    })
+  )
+}
+
+async function main() {
+  const [, , modelDir, mode = 'stream', text = DEFAULT_TEXT] = process.argv
+  if (!modelDir || !MODES.includes(mode)) {
+    throw new Error(
+      'Usage: bare examples/audio8-streaming.js MODEL_DIR [batch|stream|duplex] [TEXT]'
+    )
+  }
+  const model = new TTSGgml({
+    engine: TTSGgml.ENGINE_AUDIO8,
+    files: { modelDir },
+    greedy: true,
+    seed: 42,
+    threads: 4,
+    config: { useGPU: process.env.QVAC_TTS_AUDIO8_GPU === '1' },
+    opts: { stats: true }
+  })
+  try {
+    await model.load()
+    await measure(model, mode, text)
+  } finally {
+    await model.unload()
+  }
+}
+
+main().catch((error) => {
+  console.error(error)
+  process.exitCode = 1
+})