@qvac/audiogen-ggml 0.10.0 → 0.11.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -7,6 +7,69 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
7
7
 
8
8
  ## [Unreleased]
9
9
 
10
+ ## [0.11.0] - 2026-11-18
11
+
12
+ ### Changed
13
+
14
+ - `@qvac/fabric` dependency bumped `^0.30.1` -> `^0.31.0`, which carries
15
+ `qvac-fabric` `11739.3.0#1` -> `11739.4.0`
16
+ ([#5831](https://github.com/tetherto/qvac/pull/5831)): opt-in WebGPU builds,
17
+ mtmd-helper decode limits, Core ML FFN prefill for Qwen3.5 0.8B and 2B,
18
+ OpenCL fixes for device-side uploads, GROUP_NORM and long flash-attention
19
+ chunks, the XDNA fused decode and host-decode fallback, and TurboVec IVF
20
+ lists stored contiguously. The mobile dev and peer dependencies
21
+ `@qvac/fabric-android-arm64` / `@qvac/fabric-ios` move to `^0.31.0` with it.
22
+ - `@qvac/fabric` 0.31 adds fields to `common_params_speculative_draft` and the
23
+ `mtmd_helper_*` structs, so it is not ABI-compatible with 0.30.x. The RPC
24
+ protocol stays at 117. Every addon loaded in one process must resolve the
25
+ same `@qvac/fabric` minor. Mobile apps must move `@qvac/fabric` and
26
+ `@qvac/fabric-android-arm64` or `@qvac/fabric-ios` to `0.31.0` together.
27
+ - This package needed no code change for the new fabric.
28
+ - `qvac-lib-inference-addon-cpp` dependency floor raised `1.5.0` -> `1.7.0`
29
+ ([#5812](https://github.com/tetherto/qvac/pull/5812)). 1.6.0 lets runtime
30
+ stat values be strings (`RuntimeStatValue`), and 1.7.0 adds
31
+ `JsAsyncTask::runOn` for running async work on an addon-owned executor.
32
+ Compile-time only; no API or runtime behaviour change for this package.
33
+
34
+
35
+ - Require `inference-addon-cpp` 1.6.0 for the shared runtime-stat value type
36
+ ([#5812](https://github.com/tetherto/qvac/pull/5812)).
37
+
38
+ - Share the `@qvac/fabric` runtime and backend loader, retaining addon-local
39
+ speech engines built against its headers
40
+ ([#5808](https://github.com/tetherto/qvac/pull/5808)).
41
+
42
+ - Build the bundled speech engine from the source published as
43
+ `speech-cpp` `2026-11-18` (`5e1e4d34`).
44
+
45
+ ### Fixed
46
+
47
+ - `encode()` / `encodePcm()` release every bare-ffmpeg object they create for
48
+ compressed formats, including when an encode fails. Before this, each encode
49
+ kept the input PCM, the encoded output and the native FFmpeg state in memory
50
+ for the life of the process
51
+ ([#5610](https://github.com/tetherto/qvac/pull/5610)).
52
+
53
+ ## [0.10.1] - 2026-11-10
54
+
55
+ ### Added
56
+
57
+ - ACE-Step `backend`, `lmBackend`, and `dumpStagesDir` controls, with matching
58
+ backend selection in `assessFit()` and opt-in WebGPU source builds.
59
+ - Apple Core ML VAE/conditioner support and MiniMax `coremlConditionerPath`
60
+ for generation and preflight; MiniMax Core ML memory projections remain `unknown`.
61
+
62
+ ### Fixed
63
+
64
+ - ACE-Step `normalizeLoudness: false` returns the engine's raw output. The
65
+ addon rescaled every non-edit result to a 0.9 peak whatever the option said,
66
+ so the flag only switched off the engine's percentile pass. Samples the VAE
67
+ leaves outside `[-1, 1]` now saturate at int16 full scale on that path.
68
+ - ACE-Step lego stems keep their mix gain relative to the source; the same
69
+ rescale raised or lowered each stem to a 0.9 peak.
70
+ - Include independent LM device pools in `assessFit()` when the primary
71
+ pipeline runs on CPU, counting the host peak once.
72
+
10
73
  ## [0.10.0] - 2026-11-05
11
74
 
12
75
  ### Breaking
package/README.md CHANGED
@@ -8,14 +8,11 @@ on desktop CPUs and GPUs and returns stereo 44.1 kHz audio.
8
8
 
9
9
  ## How it works
10
10
 
11
- The C++ CI carve-out builds the native addon on the Ubuntu 22.04 CPU runner,
12
- reusing vcpkg and compiler caches with two build workers. Addon-level C++ tests
13
- remain a stub. Only trusted default-branch builds save shared caches; pushes
14
- affecting AudioGen or its cache configuration, a schedule every 2 days, and
15
- manual dispatches warm them.
16
- Non-PR runs build the triggering commit and reject inputs selecting another
17
- repository or revision. PR events retain caller-selected PR-head checkouts.
18
- See [C++ CI configuration](../../docs/ci/nx-ci-consolidation.md#optionsci-cheat-sheet).
11
+ The C++ tests run through the CI contract's `cpp:build` and `cpp:tests` targets
12
+ on the Ubuntu 22.04 CPU runner, with two build workers. Addon-level C++ tests
13
+ remain a stub. Only trusted default-branch runs save the shared vcpkg cache;
14
+ `cache-warm.yml` warms it on pushes that touch cache inputs, every 2 days, and
15
+ on dispatch. See [the package CI contract](../../docs/ci/workflow-standard.md).
19
16
 
20
17
  Under the hood the model runs a small pipeline, and the addon just drives it and
21
18
  hands you the audio:
@@ -92,27 +89,18 @@ dependency, pinned to the exact `@qvac/audiogen-ggml` version:
92
89
  }
93
90
  ```
94
91
 
95
- The published linux-x64 prebuild bundles the CUDA backend next to Vulkan; the
96
- linux-arm64 and Windows prebuilds ship Vulkan, and there CUDA is opt-in at
97
- build time via `npm run build:cuda` (or `bare-make generate -D ENABLE_CUDA=ON`;
98
- needs `nvcc` on the build host). When
99
- CUDA is compiled in, ggml runs in hybrid dynamically-loaded backend mode: the
100
- CPU-variant, Vulkan, and CUDA backends ship as runtime-loaded modules (`.so` on
101
- Linux, `.dll` on Windows) beside the addon, and only the CUDA module depends on
102
- the CUDA runtime. Engaging CUDA needs the NVIDIA driver plus the CUDA 13 runtime
103
- libraries (cudart and cuBLAS) resolvable at load time; hosts that cannot resolve
104
- them skip the module and fall back to Vulkan or CPU. The engine prefers CUDA
105
- when both GPU backends are usable.
106
-
107
- On x64 a CUDA build's module targets **compute capability 7.5 and newer**, with
108
- native code for Turing (7.5 — RTX 20xx, GTX 16xx, T4), Ampere (8.0, 8.6), Ada
109
- (8.9), Hopper (9.0), and Blackwell (12.0, 12.1). Anything newer JIT-compiles
110
- from the bundled 8.0 PTX on first use, a one-off compile the driver caches. On
111
- linux-arm64 the native set is Jetson Orin (8.7), Grace-Hopper (9.0), and
112
- GB10 / DGX Spark (12.1), with discrete Ampere+ cards and newer parts covered
113
- through the bundled 8.0 PTX. Volta and Pascal fall outside CUDA 13's support
114
- entirely, so they have no code path here: the backend skips such devices at
115
- registration and the addon falls back to Vulkan or CPU.
92
+ The ggml runtime is the shared `@qvac/fabric` prebuild, a dependency of this
93
+ package: the addon links it dynamically, ships no ggml of its own, and shares
94
+ one runtime in memory with every other fabric addon in the process. The
95
+ runtime ships Metal inside itself on macOS and iOS, and per-platform
96
+ runtime-loaded modules elsewhere (CPU variants, Vulkan, CUDA on linux-x64,
97
+ linux-arm64 and win32-x64, OpenCL on Android), so no backend is chosen at this
98
+ package's build time. Engaging CUDA needs the NVIDIA driver plus the CUDA 13
99
+ runtime libraries (cudart and cuBLAS) resolvable at load time; hosts that
100
+ cannot resolve them skip the module and fall back to Vulkan or CPU. The engine
101
+ prefers CUDA when both GPU backends are usable. The CUDA module's device
102
+ coverage (compute capability 7.5 and newer, Volta and Pascal excluded) is the
103
+ runtime's; see the `@qvac/fabric` README.
116
104
 
117
105
  To build the native addon from source in a repository checkout:
118
106
 
@@ -165,9 +153,54 @@ the engine time into `arMs`, `conditionMs`, `flowMs` and `vocoderMs`, and
165
153
  the song ended on its own. See
166
154
  [`examples/generate-music-minimax.js`](examples/generate-music-minimax.js).
167
155
 
156
+ ### ACE-Step backend selection and diagnostics
157
+
158
+ `config.backend` selects the primary native backend by name or device name,
159
+ for example `cpu`, `MTL0`, `CUDA0`, or `WebGPU`. `config.lmBackend` selects
160
+ an independent LM backend. Empty or `auto` keeps native automatic placement
161
+ and the existing `useGPU`/`nGpuLayers` behavior. An explicit unavailable
162
+ backend fails instead of silently selecting another one. Use the same
163
+ `backend` and `lmBackend` values in `assessFit()` to project that placement.
164
+ The returned device rows include the independent LM pool even when the
165
+ primary backend is CPU; the host CPU remains the last row and appears once.
166
+
167
+ `config.dumpStagesDir` enables native intermediate tensor dumps in an existing
168
+ directory. It is disabled when omitted or empty. The directory is not created
169
+ by the addon. See
170
+ [`examples/generate-with-backends.js`](examples/generate-with-backends.js).
171
+
172
+ The addon has no GPU build options of its own: GPU backends come with the
173
+ `@qvac/fabric` runtime. To enable WebGPU, build Fabric with
174
+ `-D VCPKG_MANIFEST_FEATURES=webgpu` and link its platform package before
175
+ building this addon. The default published runtime excludes WebGPU.
176
+ Once a runtime that carries WebGPU is linked, `backendId: 5` maps to `webgpu`;
177
+ native WebGPU support is unmeasured and its automatic placement keeps the LM
178
+ and detokenizer on CPU. Independent backend selection does not override
179
+ native operation or model compatibility checks. JavaScript packages do not
180
+ include Hexagon.
181
+
182
+ ### Apple Core ML sidecars
183
+
184
+ Apple builds enable the native Core ML sidecar support. An ACE-Step VAE
185
+ sidecar is discovered next to its GGUF: `vae-BF16.gguf` uses
186
+ `vae-decoder.mlmodelc` (the native resolver removes the quantization suffix). Absent
187
+ sidecars retain the native VAE route. Its existing native failure/fallback
188
+ policy is unchanged. ACE-Step preflight sizes the native ggml VAE route;
189
+ it does not include the additional Core ML sidecar allocations.
190
+
191
+ MiniMax can instead receive `config.coremlConditionerPath`, the absolute path
192
+ to a compiled conditioner `.mlmodelc` on the inference host. Only its
193
+ conditioner convolution uses Core ML with CPU and ANE allowed; the native
194
+ layer mix and resampling remain on CPU, while the other stages retain
195
+ `device`/`useGPU` placement. This option alone does not enable the full Core ML
196
+ profile. An explicit invalid or unsupported conditioner fails to load.
197
+ Forward the same `coremlConditionerPath` to `assessFit()`; it returns
198
+ `{ status: 'unknown', devices: [], stats: {} }` because Core ML resident
199
+ memory cannot be projected.
200
+
168
201
  ### MiniMax full Core ML on macOS
169
202
 
170
- On macOS 15+, `speech-cpp[audiogen]` includes the full Core ML runtime. Use
203
+ On macOS 15+, the addon-local audiogen engine includes the full Core ML runtime. Use
171
204
  the original F16 GGUF pair and its matching compiled profile:
172
205
 
173
206
  ```js
@@ -192,7 +225,9 @@ must accommodate the prompt and generation; no request is truncated to fit it.
192
225
  The programs stay resident between generations and are released on unload.
193
226
  Core ML returns `backendId: 99` (`other`); native-only `backendDevice` and
194
227
  `gpuFallbackReason` are omitted. Memory preflight returns
195
- `coreml-memory-estimate-unavailable` for this route.
228
+ `{ status: 'unknown', devices: [], stats: {} }` for this route. The full
229
+ Core ML route remains experimental; exposing its controls does not establish
230
+ audio parity or a performance improvement for a profile.
196
231
 
197
232
  CPU/ANE initial-logit parity against the native reference is still under
198
233
  validation; the measured device policies are not a claim of identical sampled
@@ -663,8 +698,11 @@ runnable end-to-end script (`npm run example`).
663
698
  | `inferenceSteps` / `shift` | Advanced; leave unset to auto-tune per DiT. `run()` can override both per call. |
664
699
  | `cfgScale` | Default MiniMax flow guidance scale; `0` uses the model default. |
665
700
  | `nGpuLayers` | GPU layers to offload when `useGPU` is set (99 = all). |
701
+ | `backend`, `lmBackend` | ACE-Step primary and independent LM backend names; empty/auto preserves native selection. |
702
+ | `dumpStagesDir` | ACE-Step existing directory for intermediate tensor dumps; omitted/empty disables dumps. |
703
+ | `coremlConditionerPath` | MiniMax compiled conditioner `.mlmodelc`; leaves other stages on the selected native device. |
666
704
  | `threads` | CPU thread count (0 / unset = hardware default). |
667
- | `backendsDir` | Advanced; directory scanned, as given, for dlopen'd ggml backend modules. Unset scans the addon's own, next to its native module, which is where the platform package and `bare-link` put them. |
705
+ | `backendsDir` | Advanced; directory scanned, as given, for dlopen'd ggml backend modules. Unset uses the directory the `@qvac/fabric` runtime loads its own backends from (its platform package, or next to the runtime in a `bare-link`ed app). |
668
706
 
669
707
  `logger` — an optional object implementing `error`/`warn`/`info`/`debug`,
670
708
  wrapped by a level-gated `QvacLogger`.
@@ -689,7 +727,7 @@ wrapped by a level-gated `QvacLogger`.
689
727
  | `lmPhase1` | Allow the LM to infer missing metadata before generating semantic codes. |
690
728
  | `simpleMode` | Expand the caption query into a full request (caption, lyrics, unset metadata) before synthesis. |
691
729
  | `rewriteQuery` | LM FORMAT pass: rewrite the caption into a detailed description, preserving the lyric content. Requires real `lyrics` and `taskType: 'text2music'`; mutually exclusive with `simpleMode`. |
692
- | `normalizeLoudness` | Percentile loudness normalization of generated audio (default `true`); edits are never normalized. |
730
+ | `normalizeLoudness` | Percentile loudness normalization of generated audio, then a 0.9 peak (default `true`); `false` returns the raw engine output. Edits and lego stems are never normalized. |
693
731
  | `generateLrc` | Synchronized lyric timestamps: `stats.lrc` (LRC text) + `stats.lyricsScore`. Requires lyrics and `taskType: 'text2music'`. |
694
732
  | `computeQualityScore` | Teacher-forced LM quality score of the generated codes; `stats.qualityScore` in `[0, 1]`. Requires `taskType: 'text2music'`. |
695
733
  | `dcwEnabled` / `dcwScaler` / `dcwHighScaler` | Haar DCW correction controls. |
@@ -799,6 +837,8 @@ and `vaePath` name them individually and win over it.
799
837
  | `withSourceAudio` | Projects the extra VAE-encoder phase a cover or reference request loads. |
800
838
  | `keepStages` | `-1` mirrors the engine, `0` forces the staged projection, `1` all-resident. |
801
839
  | `gpuLayers` | Greater than 0 requests the GPU stack, with the fallbacks a real load applies. |
840
+ | `backend`, `lmBackend` | ACE-Step placement selectors, as the generation config takes them. |
841
+ | `coremlConditionerPath` | MiniMax conditioner path; projects as `unknown` with no device rows. |
802
842
  | `threads`, `backendsDir` | As the load takes them. |
803
843
  | `marginBytes` | Free memory that must remain for the projection to count as fitting. Defaults to 256 MiB. |
804
844
 
@@ -875,14 +915,12 @@ dispatches to the corresponding `audiogen-cpp` engine.
875
915
  - `addon/src/addon/AddonJs.hpp` — `createInstance` / `activate` / `runJob`.
876
916
  - `addon/src/model-interface/acestep/` — `AcestepModel`, wrapping the engine.
877
917
  - `addon/src/model-interface/minimax/` — desktop-only `MinimaxModel`.
878
- - Built with `cmake-bare` + `cmake-vcpkg`; `vcpkg.json` depends on the
879
- `speech-cpp[audiogen]` port.
880
- - Native dependency versions are pinned with vcpkg `overrides` in
881
- [vcpkg.json](vcpkg.json), and `package.json` records the `ggml-speech`
882
- build as `qvac["ggml-speech"]`. The speech addons ship their
883
- ggml backends under the same file names, so an app that bundles several of
884
- them keeps one copy of each: use releases that record the same
885
- `ggml-speech`.
918
+ - Speech engines are compiled into each addon from the checksum-pinned source in [`speech-source.json`](../../cmake/qvac-addon/speech-source.json), using only the headers shipped by `@qvac/fabric`. No speech or ggml vcpkg port is built.
919
+ - The ggml runtime is the shared `@qvac/fabric` prebuild: the addon links it
920
+ dynamically (`DT_NEEDED qvac__fabric-<host>@0.bare`) and ships no ggml of
921
+ its own. The speech revision and checksum live in
922
+ [speech-source.json](../../cmake/qvac-addon/speech-source.json). See
923
+ [`packages/fabric/INTEGRATION.md`](../fabric/INTEGRATION.md).
886
924
 
887
925
  ## Fuzzing
888
926
 
@@ -950,3 +988,7 @@ version bump.
950
988
 
951
989
  Apache-2.0. ACE-Step model weights belong to ACE Studio and StepFun.
952
990
  MiniMax-Music3 weights are governed by the MiniMax-Music3 Community License.
991
+
992
+ Shared ggml logging belongs to Fabric. Speech initialization preserves its
993
+ callback; Whisper and Parakeet retain their engine-local log forwarding.
994
+ Explicit `backendsDir` overrides remain supported; an unset value uses Fabric.
package/audiogen.d.ts CHANGED
@@ -11,6 +11,10 @@ export interface AudioGenConfigurationParams {
11
11
  useGPU?: boolean;
12
12
  /** MiniMax compute device ('cpu' | 'gpu' | 'auto'); overrides useGPU. */
13
13
  device?: string;
14
+ backend?: string;
15
+ lmBackend?: string;
16
+ dumpStagesDir?: string;
17
+ coremlConditionerPath?: string;
14
18
  coremlProfileDir?: string;
15
19
  coremlComputeUnits?: string;
16
20
  nGpuLayers?: number;
@@ -153,40 +153,34 @@ The matrix runner additionally honours `MATRIX_JSON` (the sweep) and
153
153
 
154
154
  ## Running in CI
155
155
 
156
- There are three entry points.
157
-
158
- `.github/workflows/benchmark-performance-audiogen-ggml.yml` is the manual one
159
- (`workflow_dispatch`). It builds prebuilds (or takes a published
160
- `prebuild_package`), fans out to both lanes, and aggregates.
161
-
162
- On a pull request, add the **`run-benchmarks`** label alongside
163
- `run-desktop-addon-tests` and `run-mobile-addon-tests`. The lanes then run with
164
- `run_rtf_benchmarks: true` and `summarize-benchmarks` renders the findings table
165
- onto the run summary. Without the label a PR pays nothing, which is the default:
166
- a full sweep renders for hours and downloads every DiT variant.
167
-
168
- Both paths share `reusable-summarize-audiogen-ggml-benchmarks.yml`, so the table
169
- on a PR is the table a manual sweep produces.
170
-
171
- `.github/workflows/perf-report.yml` is the weekly cross-addon report. It measures
172
- nothing itself: it re-aggregates the artifacts of the last six sweeps into
173
- `reports/audiogen-ggml-performance.md`, which lands in that run's summary
156
+ audiogen is on the package CI contract, so the benchmarks run through
157
+ `benchmarks.yml`, dispatched with `-f packages=audiogen-ggml`.
158
+
159
+ **Desktop** is the `benchmark` target, one job per configuration in
160
+ `project.json`. Each configuration names a set in `ci/benchmark-matrix.json`
161
+ with `--matrix`: every runner sweeps the three
162
+ variants on CPU, and GPU-capable runners sweep them again on the platform backend
163
+ (Vulkan on linux/win32, Metal on darwin). Artifacts:
164
+ `audiogen-ggml-benchmark-<configuration>`. A job fails when the matrix runner
165
+ fails or writes no result.
166
+
167
+ **Mobile** runs when the dispatch sets `run_mobile`, on the chosen platform and
168
+ devices. The dispatch's `args` input picks the one matrix row the run measures,
169
+ as `key=value` words (`dit_variant`, `use_gpu`, `warmup`, `runs`, `duration`), for
170
+ example `args=dit_variant=sft use_gpu=true`; `ci/prepare-mobile.mjs` turns it into
171
+ the device environment and runs only the perf tests. The numbers leave the device
172
+ inside `[PERF_REPORT_START]` log markers, which the shared extractor scrapes.
173
+ Artifacts: `audiogen-ggml-perf-mobile-<platform>-<run number>`.
174
+
175
+ The `benchmark` target's `report` configuration then runs
176
+ `ci/benchmark-report.sh` over every result of the run and renders the findings
177
+ table onto the run summary.
178
+
179
+ `benchmarks.yml`'s `perf-report` job is the weekly cross-addon report. It measures
180
+ nothing itself: it re-aggregates the artifacts of the last six `Benchmarks` runs
181
+ into `reports/audiogen-ggml-performance.md`, which lands in that run's summary
174
182
  alongside the other addons. ACE-Step is therefore only as fresh as the last
175
- sweep — nothing dispatches one on a schedule.
176
-
177
- **Desktop** — `integration-test-audiogen-ggml.yml` with
178
- `run_rtf_benchmarks: true`. Every runner sweeps the three variants on CPU;
179
- GPU-capable runners sweep them again on the platform backend (Vulkan on
180
- linux/win32, Metal on darwin). Artifacts: `rtf-results-audiogen-ggml-*`.
181
-
182
- **Mobile** — `integration-mobile-test-audiogen-ggml.yml` with
183
- `run_rtf_benchmarks: true`. One AWS Device Farm row per variant/provider on
184
- Android and iOS, since each row downloads its variant's GGUFs on-device. The
185
- numbers leave the device inside `[PERF_REPORT_START]` log markers, which the
186
- shared extractor scrapes. Artifacts: `perf-report-audiogen-ggml-*`.
187
-
188
- Both lanes are `continue-on-error`, so a configuration that OOMs or overruns
189
- shows up as a missing row rather than a failed sweep.
183
+ sweep; nothing dispatches one on a schedule.
190
184
 
191
185
  ## How the findings table is built
192
186
 
package/binding.js CHANGED
@@ -1 +1,3 @@
1
+ require('@qvac/fabric')
2
+
1
3
  module.exports = require('#host-addon')
@@ -0,0 +1,58 @@
1
+ 'use strict'
2
+
3
+ const fs = require('bare-fs')
4
+ const proc = require('bare-process')
5
+ const { AudioGen, assessFit } = require('@qvac/audiogen-ggml')
6
+
7
+ async function collectAudio(response) {
8
+ const chunks = []
9
+ let sampleRate = 0
10
+ let channels = 0
11
+ for await (const chunk of response.iterate()) {
12
+ if (!chunk.outputArray) continue
13
+ chunks.push(
14
+ Buffer.from(
15
+ chunk.outputArray.buffer,
16
+ chunk.outputArray.byteOffset,
17
+ chunk.outputArray.byteLength
18
+ )
19
+ )
20
+ sampleRate = chunk.sampleRate
21
+ channels = chunk.channels
22
+ }
23
+ return { pcm: Buffer.concat(chunks), sampleRate, channels, stats: await response.await() }
24
+ }
25
+
26
+ async function main() {
27
+ const [modelDir, backend, lmBackend, dumpStagesDir, outputPath] = proc.argv.slice(2)
28
+ if (!modelDir || !backend || !lmBackend || !dumpStagesDir || !outputPath) {
29
+ throw new Error(
30
+ 'Usage: bare examples/generate-with-backends.js MODEL_DIR BACKEND LM_BACKEND EXISTING_DUMP_DIR OUTPUT.wav'
31
+ )
32
+ }
33
+
34
+ console.log(await assessFit({ modelsDir: modelDir, backend, lmBackend }))
35
+ const generator = new AudioGen({
36
+ files: { modelDir },
37
+ config: { backend, lmBackend, dumpStagesDir }
38
+ })
39
+ try {
40
+ await generator.load()
41
+ const response = await generator.run('A warm acoustic instrumental with gentle guitar', {
42
+ lyrics: '[Instrumental]',
43
+ duration: 6,
44
+ seed: 7
45
+ })
46
+ const { pcm, sampleRate, channels, stats } = await collectAudio(response)
47
+ const encoded = AudioGen.encode(pcm, 'wav', { sampleRate, channels })
48
+ fs.writeFileSync(outputPath, encoded.data)
49
+ console.log(stats)
50
+ } finally {
51
+ await generator.destroy()
52
+ }
53
+ }
54
+
55
+ main().catch((error) => {
56
+ console.error(error)
57
+ proc.exitCode = 1
58
+ })
package/index.d.ts CHANGED
@@ -53,6 +53,14 @@ export interface AudioGenRuntimeConfig {
53
53
  coremlProfileDir?: string;
54
54
  /** MiniMax only: explicit Core ML device policy; requires coremlProfileDir. */
55
55
  coremlComputeUnits?: AudiogenMinimaxCoreMLComputeUnits;
56
+ /** MiniMax only: compiled conditioner .mlmodelc; the remaining stages use device/useGPU. */
57
+ coremlConditionerPath?: string;
58
+ /** ACE-Step primary backend name or device name; empty/auto preserves native selection. */
59
+ backend?: string;
60
+ /** ACE-Step LM backend, selected independently from the primary backend. */
61
+ lmBackend?: string;
62
+ /** ACE-Step diagnostics: existing directory receiving native intermediate tensor dumps. */
63
+ dumpStagesDir?: string;
56
64
  /** ACE-Step only: GPU layers to offload when `useGPU` is set (99 = all). */
57
65
  nGpuLayers?: number;
58
66
  /** 0 = engine auto-picks. */
@@ -136,8 +144,11 @@ export interface GenerateOptions {
136
144
  /**
137
145
  * Percentile loudness normalization on the generated audio (default true):
138
146
  * the 99.999th-percentile sample scales to full scale and the tiny tail
139
- * above it clips, matching the reference loudness. Set false for the raw
140
- * engine output. Audio edits are never normalized.
147
+ * above it clips, matching the reference loudness; the result is then
148
+ * scaled to a 0.9 peak for int16 headroom. Set false for the raw engine
149
+ * output at its own gain: samples the VAE leaves outside [-1, 1] saturate
150
+ * at int16 full scale. Audio edits and lego stems are never normalized, so
151
+ * a stem keeps its mix gain relative to the source.
141
152
  */
142
153
  normalizeLoudness?: boolean;
143
154
  /**
@@ -445,7 +456,7 @@ export interface UnderstandOptions {
445
456
  lmTopK?: number;
446
457
  }
447
458
  /** Name of a backend `AudiogenStats.backendId` can resolve to. */
448
- export type AudiogenBackendName = 'cpu' | 'metal' | 'cuda' | 'vulkan' | 'opencl' | 'other';
459
+ export type AudiogenBackendName = 'cpu' | 'metal' | 'cuda' | 'vulkan' | 'opencl' | 'webgpu' | 'other';
449
460
  /** `AudiogenStats.backendId` codes, named. Codes match @qvac/tts-ggml. */
450
461
  export declare const AUDIOGEN_BACKEND_NAMES: Readonly<Record<number, AudiogenBackendName>>;
451
462
  /** `undefined` for an unset or unrecognised id, never a guessed name. */
package/index.js CHANGED
@@ -48,6 +48,7 @@ exports.AUDIOGEN_BACKEND_NAMES = {
48
48
  2: 'cuda',
49
49
  3: 'vulkan',
50
50
  4: 'opencl',
51
+ 5: 'webgpu',
51
52
  99: 'other'
52
53
  };
53
54
  /** `undefined` for an unset or unrecognised id, never a guessed name. */
@@ -365,8 +366,19 @@ function validateAcestepOptions(files, config) {
365
366
  if (config.device !== undefined) {
366
367
  throw invalidInput('ACE-Step does not accept device (use useGPU and nGpuLayers)');
367
368
  }
368
- if (config.coremlProfileDir !== undefined || config.coremlComputeUnits !== undefined) {
369
- throw invalidInput('coremlProfileDir and coremlComputeUnits are MiniMax-only options');
369
+ validateRuntimeStrings(config);
370
+ if (config.coremlConditionerPath !== undefined ||
371
+ config.coremlProfileDir !== undefined ||
372
+ config.coremlComputeUnits !== undefined) {
373
+ throw invalidInput('coremlConditionerPath, coremlProfileDir and coremlComputeUnits are MiniMax-only options');
374
+ }
375
+ }
376
+ function validateRuntimeStrings(config) {
377
+ for (const key of ['backend', 'lmBackend', 'dumpStagesDir', 'coremlConditionerPath']) {
378
+ const value = config[key];
379
+ if (value !== undefined && (typeof value !== 'string' || value.includes('\0'))) {
380
+ throw invalidInput(`${key} must be a string without NUL characters`);
381
+ }
370
382
  }
371
383
  }
372
384
  function validateMinimaxCoreMLConfig(config) {
@@ -384,6 +396,12 @@ function validateMinimaxCoreMLConfig(config) {
384
396
  }
385
397
  }
386
398
  function validateMinimaxConfig(config) {
399
+ validateRuntimeStrings(config);
400
+ if (config.backend !== undefined ||
401
+ config.lmBackend !== undefined ||
402
+ config.dumpStagesDir !== undefined) {
403
+ throw invalidInput('MiniMax does not accept backend, lmBackend or dumpStagesDir');
404
+ }
387
405
  validateMinimaxCoreMLConfig(config);
388
406
  if (config.useGPU !== undefined && typeof config.useGPU !== 'boolean') {
389
407
  throw invalidInput('useGPU must be a boolean');
@@ -610,6 +628,7 @@ class AudioGen {
610
628
  threads,
611
629
  useGPU: config.useGPU ?? false,
612
630
  device: config.device,
631
+ coremlConditionerPath: config.coremlConditionerPath,
613
632
  coremlProfileDir: config.coremlProfileDir,
614
633
  coremlComputeUnits: config.coremlComputeUnits,
615
634
  backendsDir
@@ -634,6 +653,9 @@ class AudioGen {
634
653
  inferenceSteps: this._defaultInferenceSteps,
635
654
  shift: requireAcestepShift(config.shift ?? 0),
636
655
  useGPU: config.useGPU ?? false,
656
+ backend: config.backend,
657
+ lmBackend: config.lmBackend,
658
+ dumpStagesDir: config.dumpStagesDir,
637
659
  nGpuLayers: requireFiniteNumber(config.nGpuLayers ?? 99, 'nGpuLayers', true),
638
660
  threads,
639
661
  backendsDir
@@ -230,7 +230,16 @@ function createWriteIO(ffmpeg) {
230
230
  }
231
231
  })
232
232
 
233
- return { io, getData: () => Buffer.from(buf.subarray(0, size)) }
233
+ // OutputFormatContext takes the handle via transfer() before it can throw; track the new owner.
234
+ let owner = io
235
+ const transfer = io.transfer
236
+ io.transfer = () => (owner = transfer.call(io))
237
+
238
+ return {
239
+ io,
240
+ getData: () => Buffer.from(buf.subarray(0, size)),
241
+ destroy: () => owner.destroy()
242
+ }
234
243
  }
235
244
 
236
245
  // A read-only in-memory source for FFmpeg (feeds the decoder our WAV bytes).
@@ -307,109 +316,145 @@ function encodeWithFfmpeg(pcm, format, { sampleRate = 48000, channels = 2 } = {}
307
316
  const outRate = spec.rate || sampleRate
308
317
  const sampleFormat = C.sampleFormats[spec.sampleFormat]
309
318
 
310
- // --- decode side: wrap PCM as WAV so ffmpeg produces clean input frames ---
311
- const wav = pcmToWav(pcm, { sampleRate, channels })
312
- const inFmt = new ffmpeg.InputFormatContext(createReadIO(ffmpeg, wav))
313
- const inStream = inFmt.streams[0]
314
- const decoder = inStream.decoder()
315
- decoder.open()
319
+ // bare-ffmpeg handles have no GC finalizer: until destroyed, the IOContexts hold
320
+ // strong refs to their callbacks, which pin the input WAV and the output buffer.
321
+ let inFmt = null
322
+ let decoder = null
323
+ let cc = null
324
+ let sink = null
325
+ let ofc = null
326
+ let resampler = null
327
+ let fifo = null
328
+ let packet = null
329
+ let inPacket = null
330
+ let raw = null
331
+
332
+ try {
333
+ // --- decode side: wrap PCM as WAV so ffmpeg produces clean input frames ---
334
+ const wav = pcmToWav(pcm, { sampleRate, channels })
335
+ inFmt = new ffmpeg.InputFormatContext(createReadIO(ffmpeg, wav))
336
+ const inStream = inFmt.streams[0]
337
+ decoder = inStream.decoder()
338
+ decoder.open()
316
339
 
317
- // --- encoder ---
318
- const encoder = new ffmpeg.Encoder(spec.encoder)
319
- const cc = new ffmpeg.CodecContext(encoder)
320
- cc.sampleRate = outRate
321
- cc.sampleFormat = sampleFormat
322
- cc.channelLayout = ffmpeg.ChannelLayout.from(layout)
323
- cc.timeBase = new ffmpeg.Rational(1, outRate)
324
- if (spec.experimental) {
325
- try {
326
- cc.setOption('strict', 'experimental')
327
- } catch {
328
- /* older bindings: option may already default to permissive */
340
+ // --- encoder ---
341
+ const encoder = new ffmpeg.Encoder(spec.encoder)
342
+ cc = new ffmpeg.CodecContext(encoder)
343
+ cc.sampleRate = outRate
344
+ cc.sampleFormat = sampleFormat
345
+ cc.channelLayout = ffmpeg.ChannelLayout.from(layout)
346
+ cc.timeBase = new ffmpeg.Rational(1, outRate)
347
+ if (spec.experimental) {
348
+ try {
349
+ cc.setOption('strict', 'experimental')
350
+ } catch {
351
+ /* older bindings: option may already default to permissive */
352
+ }
329
353
  }
330
- }
331
- cc.open()
332
- const frameSize = cc.frameSize > 0 ? cc.frameSize : 4096
354
+ cc.open()
355
+ const frameSize = cc.frameSize > 0 ? cc.frameSize : 4096
333
356
 
334
- // --- muxer (seekable in-memory) ---
335
- const sink = createWriteIO(ffmpeg)
336
- const ofc = new ffmpeg.OutputFormatContext(spec.container, sink.io)
337
- const ostream = ofc.createStream()
338
- ostream.codecParameters.fromContext(cc)
339
- ofc.writeHeader()
357
+ // --- muxer (seekable in-memory) ---
358
+ sink = createWriteIO(ffmpeg)
359
+ ofc = new ffmpeg.OutputFormatContext(spec.container, sink.io)
360
+ const ostream = ofc.createStream()
361
+ ostream.codecParameters.fromContext(cc)
362
+ ofc.writeHeader()
340
363
 
341
- // --- resample (input format/rate -> encoder format/rate) + FIFO framing ---
342
- const resampler = new ffmpeg.Resampler(
343
- inStream.codecParameters.sampleRate,
344
- inStream.codecParameters.channelLayout,
345
- inStream.codecParameters.format,
346
- outRate,
347
- layout,
348
- sampleFormat
349
- )
350
- const fifo = new ffmpeg.AudioFIFO(sampleFormat, channels, frameSize)
351
- const packet = new ffmpeg.Packet()
352
- let pts = 0
364
+ // --- resample (input format/rate -> encoder format/rate) + FIFO framing ---
365
+ resampler = new ffmpeg.Resampler(
366
+ inStream.codecParameters.sampleRate,
367
+ inStream.codecParameters.channelLayout,
368
+ inStream.codecParameters.format,
369
+ outRate,
370
+ layout,
371
+ sampleFormat
372
+ )
373
+ fifo = new ffmpeg.AudioFIFO(sampleFormat, channels, frameSize)
374
+ packet = new ffmpeg.Packet()
375
+ let pts = 0
353
376
 
354
- const emit = (frame) => {
355
- cc.sendFrame(frame)
356
- while (cc.receivePacket(packet)) {
357
- packet.streamIndex = ostream.index
358
- packet.rescaleTimestamps(cc.timeBase, ostream.timeBase)
359
- ofc.writeFrame(packet)
360
- packet.unref()
377
+ const emit = (frame) => {
378
+ cc.sendFrame(frame)
379
+ while (cc.receivePacket(packet)) {
380
+ packet.streamIndex = ostream.index
381
+ packet.rescaleTimestamps(cc.timeBase, ostream.timeBase)
382
+ ofc.writeFrame(packet)
383
+ packet.unref()
384
+ }
361
385
  }
362
- }
363
386
 
364
- const drain = (flush) => {
365
- while (fifo.size >= frameSize || (flush && fifo.size > 0)) {
366
- const n = Math.min(frameSize, fifo.size)
367
- const frame = new ffmpeg.Frame()
368
- frame.format = sampleFormat
369
- frame.channelLayout = ffmpeg.ChannelLayout.from(layout)
370
- frame.sampleRate = outRate
371
- frame.nbSamples = n
372
- frame.alloc()
373
- fifo.read(frame, n)
374
- frame.pts = pts
375
- pts += n
376
- emit(frame)
387
+ const drain = (flush) => {
388
+ while (fifo.size >= frameSize || (flush && fifo.size > 0)) {
389
+ const n = Math.min(frameSize, fifo.size)
390
+ const frame = new ffmpeg.Frame()
391
+ try {
392
+ frame.format = sampleFormat
393
+ frame.channelLayout = ffmpeg.ChannelLayout.from(layout)
394
+ frame.sampleRate = outRate
395
+ frame.nbSamples = n
396
+ frame.alloc()
397
+ fifo.read(frame, n)
398
+ frame.pts = pts
399
+ pts += n
400
+ emit(frame)
401
+ } finally {
402
+ frame.destroy()
403
+ }
404
+ }
377
405
  }
378
- }
379
406
 
380
- const pushResampled = (rawFrame) => {
381
- const out = new ffmpeg.Frame()
382
- out.channelLayout = ffmpeg.ChannelLayout.from(layout)
383
- out.format = sampleFormat
384
- out.sampleRate = outRate
385
- const inSamples = rawFrame ? rawFrame.nbSamples : 1024
386
- out.nbSamples = Math.ceil((inSamples * outRate) / inStream.codecParameters.sampleRate) + 1024
387
- const samples = new ffmpeg.Samples()
388
- samples.fill(out)
389
- const count = rawFrame ? resampler.convert(rawFrame, out) : resampler.flush(out)
390
- if (count > 0) {
391
- out.nbSamples = count
392
- fifo.write(out)
407
+ const pushResampled = (rawFrame) => {
408
+ const out = new ffmpeg.Frame()
409
+ try {
410
+ out.channelLayout = ffmpeg.ChannelLayout.from(layout)
411
+ out.format = sampleFormat
412
+ out.sampleRate = outRate
413
+ const inSamples = rawFrame ? rawFrame.nbSamples : 1024
414
+ out.nbSamples =
415
+ Math.ceil((inSamples * outRate) / inStream.codecParameters.sampleRate) + 1024
416
+ const samples = new ffmpeg.Samples()
417
+ samples.fill(out)
418
+ const count = rawFrame ? resampler.convert(rawFrame, out) : resampler.flush(out)
419
+ if (count > 0) {
420
+ out.nbSamples = count
421
+ fifo.write(out)
422
+ }
423
+ return count
424
+ } finally {
425
+ out.destroy()
426
+ }
393
427
  }
394
- return count
395
- }
396
428
 
397
- const inPacket = new ffmpeg.Packet()
398
- const raw = new ffmpeg.Frame()
399
- while (inFmt.readFrame(inPacket)) {
400
- decoder.sendPacket(inPacket)
401
- while (decoder.receiveFrame(raw)) pushResampled(raw)
402
- inPacket.unref()
403
- drain(false)
404
- }
405
- while (pushResampled(null) > 0) {
406
- /* flush the resampler's internal buffer */
407
- }
408
- drain(true)
409
- emit(null) // flush the encoder
410
- ofc.writeTrailer()
429
+ inPacket = new ffmpeg.Packet()
430
+ raw = new ffmpeg.Frame()
431
+ while (inFmt.readFrame(inPacket)) {
432
+ decoder.sendPacket(inPacket)
433
+ while (decoder.receiveFrame(raw)) pushResampled(raw)
434
+ inPacket.unref()
435
+ drain(false)
436
+ }
437
+ while (pushResampled(null) > 0) {
438
+ /* flush the resampler's internal buffer */
439
+ }
440
+ drain(true)
441
+ emit(null) // flush the encoder
442
+ ofc.writeTrailer()
411
443
 
412
- return sink.getData()
444
+ return sink.getData()
445
+ } finally {
446
+ raw?.destroy()
447
+ inPacket?.destroy()
448
+ packet?.destroy()
449
+ fifo?.destroy()
450
+ resampler?.destroy()
451
+ cc?.destroy()
452
+ decoder?.destroy()
453
+ ofc?.destroy()
454
+ // No-op once ofc owns the IO; releases it if OutputFormatContext was never created.
455
+ sink?.destroy()
456
+ inFmt?.destroy()
457
+ }
413
458
  }
414
459
 
415
460
  // Encode a single format. `pcm`/`wav` stay dependency-free; the rest go through
package/lib/fit.d.ts CHANGED
@@ -19,6 +19,10 @@ export interface AudiogenFitRequest {
19
19
  /** MiniMax synthesis GGUF; pairs with `lmPath`. */
20
20
  synthPath?: string
21
21
 
22
+ /** ACE-Step primary backend name/device; empty/auto preserves native selection. */
23
+ backend?: string
24
+ /** ACE-Step LM backend, independent of the primary backend. */
25
+ lmBackend?: string
22
26
  /** > 0 requests the GPU stack, with the fallbacks a real load applies. */
23
27
  gpuLayers?: number
24
28
  /**
@@ -26,7 +30,9 @@ export interface AudiogenFitRequest {
26
30
  * > 0 projects `auto` (a GPU when one is usable) and anything else `cpu`.
27
31
  */
28
32
  device?: 'cpu' | 'gpu' | 'auto'
29
- /** MiniMax full Core ML profile. Projection returns coreml-memory-estimate-unavailable. */
33
+ /** MiniMax compiled conditioner. Core ML projection returns status unknown with no device rows. */
34
+ coremlConditionerPath?: string
35
+ /** MiniMax full Core ML profile. Projection returns status unknown with no device rows. */
30
36
  coremlProfileDir?: string
31
37
  coremlComputeUnits?: 'CPU_ONLY' | 'CPU_AND_GPU' | 'CPU_AND_NE' | 'ALL'
32
38
  threads?: number
package/package.json CHANGED
@@ -1,10 +1,7 @@
1
1
  {
2
2
  "name": "@qvac/audiogen-ggml",
3
- "version": "0.10.0",
3
+ "version": "0.11.0",
4
4
  "description": "Audio generation (music) addon for qvac",
5
- "qvac": {
6
- "ggml-speech": "2026-10-27#1"
7
- },
8
5
  "engines": {
9
6
  "bare": ">=1.19.0"
10
7
  },
@@ -35,9 +32,7 @@
35
32
  "scripts": {
36
33
  "build": "npm run build:ts && npm run build:native",
37
34
  "build:native": "bare-make generate -D BUILD_FUZZING=OFF && bare-make build && bare-make install && npm run link:platform",
38
- "build:native:cuda": "bare-make generate -D ENABLE_CUDA=ON -D BUILD_FUZZING=OFF && bare-make build && bare-make install && npm run link:platform",
39
35
  "link:platform": "node ../../scripts/ci/slice-platform-packages.mjs --link-local --workdir .",
40
- "build:cuda": "npm run build:ts && npm run build:native:cuda",
41
36
  "build:ts": "tsc -p tsconfig.build.json",
42
37
  "build:pack": "npm run build:ts && mkdir -p dist && npm pack --pack-destination dist",
43
38
  "example": "bare examples/generate-music.js \"Upbeat pop rock with driving electric guitars and a catchy hook\"",
@@ -113,7 +108,8 @@
113
108
  "README.md",
114
109
  "CHANGELOG.md",
115
110
  "NOTICE",
116
- "LICENSE"
111
+ "LICENSE",
112
+ "examples/generate-with-backends.js"
117
113
  ],
118
114
  "license": "Apache-2.0",
119
115
  "repository": {
@@ -145,7 +141,17 @@
145
141
  "prettier-config-holepunch": "^2.0.0",
146
142
  "typescript": "^6.0.3",
147
143
  "typescript-eslint": "^8.63.0",
148
- "cmake-npm": "^1.1.0"
144
+ "cmake-npm": "^1.1.0",
145
+ "bare": "^1.36.0",
146
+ "bare-runtime": "^1.36.0",
147
+ "bare-make": "^2.1.0",
148
+ "cmake-toolchains": "^2.1.0",
149
+ "cmake-runtime": "^4.1.1",
150
+ "llvm-runtime": "^23.1.0",
151
+ "nasm-runtime": "^3.2.0",
152
+ "ninja-runtime": "^1.13.1",
153
+ "@qvac/fabric-android-arm64": "^0.31.0",
154
+ "@qvac/fabric-ios": "^0.31.0"
149
155
  },
150
156
  "dependencies": {
151
157
  "@qvac/error": "^0.1.1",
@@ -155,24 +161,33 @@
155
161
  "bare-fs": "^4.5.6",
156
162
  "bare-os": "^3.8.0",
157
163
  "bare-path": "^3.0.0",
158
- "bare-process": "^4.2.2"
164
+ "bare-process": "^4.2.2",
165
+ "@qvac/fabric": "^0.31.0"
159
166
  },
160
167
  "optionalDependencies": {
161
- "@qvac/audiogen-ggml-linux-x64": "0.10.0",
162
- "@qvac/audiogen-ggml-linux-arm64": "0.10.0",
163
- "@qvac/audiogen-ggml-darwin-arm64": "0.10.0",
164
- "@qvac/audiogen-ggml-darwin-x64": "0.10.0",
165
- "@qvac/audiogen-ggml-win32-x64": "0.10.0"
168
+ "@qvac/audiogen-ggml-linux-x64": "0.11.0",
169
+ "@qvac/audiogen-ggml-linux-arm64": "0.11.0",
170
+ "@qvac/audiogen-ggml-darwin-arm64": "0.11.0",
171
+ "@qvac/audiogen-ggml-darwin-x64": "0.11.0",
172
+ "@qvac/audiogen-ggml-win32-x64": "0.11.0"
166
173
  },
167
174
  "overrides": {
168
175
  "bare-module-lexer": "1.6.3"
169
176
  },
170
177
  "types": "index.d.ts",
171
178
  "peerDependencies": {
172
- "@qvac/audiogen-ggml-android-arm64": "0.10.0",
173
- "@qvac/audiogen-ggml-ios": "0.10.0"
179
+ "@qvac/fabric-android-arm64": "^0.31.0",
180
+ "@qvac/fabric-ios": "^0.31.0",
181
+ "@qvac/audiogen-ggml-android-arm64": "0.11.0",
182
+ "@qvac/audiogen-ggml-ios": "0.11.0"
174
183
  },
175
184
  "peerDependenciesMeta": {
185
+ "@qvac/fabric-android-arm64": {
186
+ "optional": true
187
+ },
188
+ "@qvac/fabric-ios": {
189
+ "optional": true
190
+ },
176
191
  "@qvac/audiogen-ggml-android-arm64": {
177
192
  "optional": true
178
193
  },