@qvac/audiogen-ggml 0.10.1 → 0.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +43 -0
- package/README.md +35 -45
- package/benchmarks/RTF-BENCHMARKS.md +27 -33
- package/binding.js +2 -0
- package/lib/audio-format.js +137 -92
- package/package.json +29 -15
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,49 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.11.0] - 2026-11-18
|
|
11
|
+
|
|
12
|
+
### Changed
|
|
13
|
+
|
|
14
|
+
- `@qvac/fabric` dependency bumped `^0.30.1` -> `^0.31.0`, which carries
|
|
15
|
+
`qvac-fabric` `11739.3.0#1` -> `11739.4.0`
|
|
16
|
+
([#5831](https://github.com/tetherto/qvac/pull/5831)): opt-in WebGPU builds,
|
|
17
|
+
mtmd-helper decode limits, Core ML FFN prefill for Qwen3.5 0.8B and 2B,
|
|
18
|
+
OpenCL fixes for device-side uploads, GROUP_NORM and long flash-attention
|
|
19
|
+
chunks, the XDNA fused decode and host-decode fallback, and TurboVec IVF
|
|
20
|
+
lists stored contiguously. The mobile dev and peer dependencies
|
|
21
|
+
`@qvac/fabric-android-arm64` / `@qvac/fabric-ios` move to `^0.31.0` with it.
|
|
22
|
+
- `@qvac/fabric` 0.31 adds fields to `common_params_speculative_draft` and the
|
|
23
|
+
`mtmd_helper_*` structs, so it is not ABI-compatible with 0.30.x. The RPC
|
|
24
|
+
protocol stays at 117. Every addon loaded in one process must resolve the
|
|
25
|
+
same `@qvac/fabric` minor. Mobile apps must move `@qvac/fabric` and
|
|
26
|
+
`@qvac/fabric-android-arm64` or `@qvac/fabric-ios` to `0.31.0` together.
|
|
27
|
+
- This package needed no code change for the new fabric.
|
|
28
|
+
- `qvac-lib-inference-addon-cpp` dependency floor raised `1.5.0` -> `1.7.0`
|
|
29
|
+
([#5812](https://github.com/tetherto/qvac/pull/5812)). 1.6.0 lets runtime
|
|
30
|
+
stat values be strings (`RuntimeStatValue`), and 1.7.0 adds
|
|
31
|
+
`JsAsyncTask::runOn` for running async work on an addon-owned executor.
|
|
32
|
+
Compile-time only; no API or runtime behaviour change for this package.
|
|
33
|
+
|
|
34
|
+
|
|
35
|
+
- Require `inference-addon-cpp` 1.6.0 for the shared runtime-stat value type
|
|
36
|
+
([#5812](https://github.com/tetherto/qvac/pull/5812)).
|
|
37
|
+
|
|
38
|
+
- Share the `@qvac/fabric` runtime and backend loader, retaining addon-local
|
|
39
|
+
speech engines built against its headers
|
|
40
|
+
([#5808](https://github.com/tetherto/qvac/pull/5808)).
|
|
41
|
+
|
|
42
|
+
- Build the bundled speech engine from the source published as
|
|
43
|
+
`speech-cpp` `2026-11-18` (`5e1e4d34`).
|
|
44
|
+
|
|
45
|
+
### Fixed
|
|
46
|
+
|
|
47
|
+
- `encode()` / `encodePcm()` release every bare-ffmpeg object they create for
|
|
48
|
+
compressed formats, including when an encode fails. Before this, each encode
|
|
49
|
+
kept the input PCM, the encoded output and the native FFmpeg state in memory
|
|
50
|
+
for the life of the process
|
|
51
|
+
([#5610](https://github.com/tetherto/qvac/pull/5610)).
|
|
52
|
+
|
|
10
53
|
## [0.10.1] - 2026-11-10
|
|
11
54
|
|
|
12
55
|
### Added
|
package/README.md
CHANGED
|
@@ -8,14 +8,11 @@ on desktop CPUs and GPUs and returns stereo 44.1 kHz audio.
|
|
|
8
8
|
|
|
9
9
|
## How it works
|
|
10
10
|
|
|
11
|
-
The C++
|
|
12
|
-
|
|
13
|
-
remain a stub. Only trusted default-branch
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
Non-PR runs build the triggering commit and reject inputs selecting another
|
|
17
|
-
repository or revision. PR events retain caller-selected PR-head checkouts.
|
|
18
|
-
See [C++ CI configuration](../../docs/ci/nx-ci-consolidation.md#optionsci-cheat-sheet).
|
|
11
|
+
The C++ tests run through the CI contract's `cpp:build` and `cpp:tests` targets
|
|
12
|
+
on the Ubuntu 22.04 CPU runner, with two build workers. Addon-level C++ tests
|
|
13
|
+
remain a stub. Only trusted default-branch runs save the shared vcpkg cache;
|
|
14
|
+
`cache-warm.yml` warms it on pushes that touch cache inputs, every 2 days, and
|
|
15
|
+
on dispatch. See [the package CI contract](../../docs/ci/workflow-standard.md).
|
|
19
16
|
|
|
20
17
|
Under the hood the model runs a small pipeline, and the addon just drives it and
|
|
21
18
|
hands you the audio:
|
|
@@ -92,27 +89,18 @@ dependency, pinned to the exact `@qvac/audiogen-ggml` version:
|
|
|
92
89
|
}
|
|
93
90
|
```
|
|
94
91
|
|
|
95
|
-
The
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
On x64 a CUDA build's module targets **compute capability 7.5 and newer**, with
|
|
108
|
-
native code for Turing (7.5 — RTX 20xx, GTX 16xx, T4), Ampere (8.0, 8.6), Ada
|
|
109
|
-
(8.9), Hopper (9.0), and Blackwell (12.0, 12.1). Anything newer JIT-compiles
|
|
110
|
-
from the bundled 8.0 PTX on first use, a one-off compile the driver caches. On
|
|
111
|
-
linux-arm64 the native set is Jetson Orin (8.7), Grace-Hopper (9.0), and
|
|
112
|
-
GB10 / DGX Spark (12.1), with discrete Ampere+ cards and newer parts covered
|
|
113
|
-
through the bundled 8.0 PTX. Volta and Pascal fall outside CUDA 13's support
|
|
114
|
-
entirely, so they have no code path here: the backend skips such devices at
|
|
115
|
-
registration and the addon falls back to Vulkan or CPU.
|
|
92
|
+
The ggml runtime is the shared `@qvac/fabric` prebuild, a dependency of this
|
|
93
|
+
package: the addon links it dynamically, ships no ggml of its own, and shares
|
|
94
|
+
one runtime in memory with every other fabric addon in the process. The
|
|
95
|
+
runtime ships Metal inside itself on macOS and iOS, and per-platform
|
|
96
|
+
runtime-loaded modules elsewhere (CPU variants, Vulkan, CUDA on linux-x64,
|
|
97
|
+
linux-arm64 and win32-x64, OpenCL on Android), so no backend is chosen at this
|
|
98
|
+
package's build time. Engaging CUDA needs the NVIDIA driver plus the CUDA 13
|
|
99
|
+
runtime libraries (cudart and cuBLAS) resolvable at load time; hosts that
|
|
100
|
+
cannot resolve them skip the module and fall back to Vulkan or CPU. The engine
|
|
101
|
+
prefers CUDA when both GPU backends are usable. The CUDA module's device
|
|
102
|
+
coverage (compute capability 7.5 and newer, Volta and Pascal excluded) is the
|
|
103
|
+
runtime's; see the `@qvac/fabric` README.
|
|
116
104
|
|
|
117
105
|
To build the native addon from source in a repository checkout:
|
|
118
106
|
|
|
@@ -181,12 +169,12 @@ directory. It is disabled when omitted or empty. The directory is not created
|
|
|
181
169
|
by the addon. See
|
|
182
170
|
[`examples/generate-with-backends.js`](examples/generate-with-backends.js).
|
|
183
171
|
|
|
184
|
-
|
|
185
|
-
`
|
|
186
|
-
`
|
|
187
|
-
|
|
188
|
-
`backendId: 5` maps to `webgpu
|
|
189
|
-
|
|
172
|
+
The addon has no GPU build options of its own: GPU backends come with the
|
|
173
|
+
`@qvac/fabric` runtime. To enable WebGPU, build Fabric with
|
|
174
|
+
`-D VCPKG_MANIFEST_FEATURES=webgpu` and link its platform package before
|
|
175
|
+
building this addon. The default published runtime excludes WebGPU.
|
|
176
|
+
Once a runtime that carries WebGPU is linked, `backendId: 5` maps to `webgpu`;
|
|
177
|
+
native WebGPU support is unmeasured and its automatic placement keeps the LM
|
|
190
178
|
and detokenizer on CPU. Independent backend selection does not override
|
|
191
179
|
native operation or model compatibility checks. JavaScript packages do not
|
|
192
180
|
include Hexagon.
|
|
@@ -212,7 +200,7 @@ memory cannot be projected.
|
|
|
212
200
|
|
|
213
201
|
### MiniMax full Core ML on macOS
|
|
214
202
|
|
|
215
|
-
On macOS 15+,
|
|
203
|
+
On macOS 15+, the addon-local audiogen engine includes the full Core ML runtime. Use
|
|
216
204
|
the original F16 GGUF pair and its matching compiled profile:
|
|
217
205
|
|
|
218
206
|
```js
|
|
@@ -714,7 +702,7 @@ runnable end-to-end script (`npm run example`).
|
|
|
714
702
|
| `dumpStagesDir` | ACE-Step existing directory for intermediate tensor dumps; omitted/empty disables dumps. |
|
|
715
703
|
| `coremlConditionerPath` | MiniMax compiled conditioner `.mlmodelc`; leaves other stages on the selected native device. |
|
|
716
704
|
| `threads` | CPU thread count (0 / unset = hardware default). |
|
|
717
|
-
| `backendsDir` | Advanced; directory scanned, as given, for dlopen'd ggml backend modules. Unset
|
|
705
|
+
| `backendsDir` | Advanced; directory scanned, as given, for dlopen'd ggml backend modules. Unset uses the directory the `@qvac/fabric` runtime loads its own backends from (its platform package, or next to the runtime in a `bare-link`ed app). |
|
|
718
706
|
|
|
719
707
|
`logger` — an optional object implementing `error`/`warn`/`info`/`debug`,
|
|
720
708
|
wrapped by a level-gated `QvacLogger`.
|
|
@@ -927,14 +915,12 @@ dispatches to the corresponding `audiogen-cpp` engine.
|
|
|
927
915
|
- `addon/src/addon/AddonJs.hpp` — `createInstance` / `activate` / `runJob`.
|
|
928
916
|
- `addon/src/model-interface/acestep/` — `AcestepModel`, wrapping the engine.
|
|
929
917
|
- `addon/src/model-interface/minimax/` — desktop-only `MinimaxModel`.
|
|
930
|
-
-
|
|
931
|
-
|
|
932
|
-
|
|
933
|
-
|
|
934
|
-
|
|
935
|
-
|
|
936
|
-
them keeps one copy of each: use releases that record the same
|
|
937
|
-
`ggml-speech`.
|
|
918
|
+
- Speech engines are compiled into each addon from the checksum-pinned source in [`speech-source.json`](../../cmake/qvac-addon/speech-source.json), using only the headers shipped by `@qvac/fabric`. No speech or ggml vcpkg port is built.
|
|
919
|
+
- The ggml runtime is the shared `@qvac/fabric` prebuild: the addon links it
|
|
920
|
+
dynamically (`DT_NEEDED qvac__fabric-<host>@0.bare`) and ships no ggml of
|
|
921
|
+
its own. The speech revision and checksum live in
|
|
922
|
+
[speech-source.json](../../cmake/qvac-addon/speech-source.json). See
|
|
923
|
+
[`packages/fabric/INTEGRATION.md`](../fabric/INTEGRATION.md).
|
|
938
924
|
|
|
939
925
|
## Fuzzing
|
|
940
926
|
|
|
@@ -1002,3 +988,7 @@ version bump.
|
|
|
1002
988
|
|
|
1003
989
|
Apache-2.0. ACE-Step model weights belong to ACE Studio and StepFun.
|
|
1004
990
|
MiniMax-Music3 weights are governed by the MiniMax-Music3 Community License.
|
|
991
|
+
|
|
992
|
+
Shared ggml logging belongs to Fabric. Speech initialization preserves its
|
|
993
|
+
callback; Whisper and Parakeet retain their engine-local log forwarding.
|
|
994
|
+
Explicit `backendsDir` overrides remain supported; an unset value uses Fabric.
|
|
@@ -153,40 +153,34 @@ The matrix runner additionally honours `MATRIX_JSON` (the sweep) and
|
|
|
153
153
|
|
|
154
154
|
## Running in CI
|
|
155
155
|
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
`
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
`
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
`
|
|
156
|
+
audiogen is on the package CI contract, so the benchmarks run through
|
|
157
|
+
`benchmarks.yml`, dispatched with `-f packages=audiogen-ggml`.
|
|
158
|
+
|
|
159
|
+
**Desktop** is the `benchmark` target, one job per configuration in
|
|
160
|
+
`project.json`. Each configuration names a set in `ci/benchmark-matrix.json`
|
|
161
|
+
with `--matrix`: every runner sweeps the three
|
|
162
|
+
variants on CPU, and GPU-capable runners sweep them again on the platform backend
|
|
163
|
+
(Vulkan on linux/win32, Metal on darwin). Artifacts:
|
|
164
|
+
`audiogen-ggml-benchmark-<configuration>`. A job fails when the matrix runner
|
|
165
|
+
fails or writes no result.
|
|
166
|
+
|
|
167
|
+
**Mobile** runs when the dispatch sets `run_mobile`, on the chosen platform and
|
|
168
|
+
devices. The dispatch's `args` input picks the one matrix row the run measures,
|
|
169
|
+
as `key=value` words (`dit_variant`, `use_gpu`, `warmup`, `runs`, `duration`), for
|
|
170
|
+
example `args=dit_variant=sft use_gpu=true`; `ci/prepare-mobile.mjs` turns it into
|
|
171
|
+
the device environment and runs only the perf tests. The numbers leave the device
|
|
172
|
+
inside `[PERF_REPORT_START]` log markers, which the shared extractor scrapes.
|
|
173
|
+
Artifacts: `audiogen-ggml-perf-mobile-<platform>-<run number>`.
|
|
174
|
+
|
|
175
|
+
The `benchmark` target's `report` configuration then runs
|
|
176
|
+
`ci/benchmark-report.sh` over every result of the run and renders the findings
|
|
177
|
+
table onto the run summary.
|
|
178
|
+
|
|
179
|
+
`benchmarks.yml`'s `perf-report` job is the weekly cross-addon report. It measures
|
|
180
|
+
nothing itself: it re-aggregates the artifacts of the last six `Benchmarks` runs
|
|
181
|
+
into `reports/audiogen-ggml-performance.md`, which lands in that run's summary
|
|
174
182
|
alongside the other addons. ACE-Step is therefore only as fresh as the last
|
|
175
|
-
sweep
|
|
176
|
-
|
|
177
|
-
**Desktop** — `integration-test-audiogen-ggml.yml` with
|
|
178
|
-
`run_rtf_benchmarks: true`. Every runner sweeps the three variants on CPU;
|
|
179
|
-
GPU-capable runners sweep them again on the platform backend (Vulkan on
|
|
180
|
-
linux/win32, Metal on darwin). Artifacts: `rtf-results-audiogen-ggml-*`.
|
|
181
|
-
|
|
182
|
-
**Mobile** — `integration-mobile-test-audiogen-ggml.yml` with
|
|
183
|
-
`run_rtf_benchmarks: true`. One AWS Device Farm row per variant/provider on
|
|
184
|
-
Android and iOS, since each row downloads its variant's GGUFs on-device. The
|
|
185
|
-
numbers leave the device inside `[PERF_REPORT_START]` log markers, which the
|
|
186
|
-
shared extractor scrapes. Artifacts: `perf-report-audiogen-ggml-*`.
|
|
187
|
-
|
|
188
|
-
Both lanes are `continue-on-error`, so a configuration that OOMs or overruns
|
|
189
|
-
shows up as a missing row rather than a failed sweep.
|
|
183
|
+
sweep; nothing dispatches one on a schedule.
|
|
190
184
|
|
|
191
185
|
## How the findings table is built
|
|
192
186
|
|
package/binding.js
CHANGED
package/lib/audio-format.js
CHANGED
|
@@ -230,7 +230,16 @@ function createWriteIO(ffmpeg) {
|
|
|
230
230
|
}
|
|
231
231
|
})
|
|
232
232
|
|
|
233
|
-
|
|
233
|
+
// OutputFormatContext takes the handle via transfer() before it can throw; track the new owner.
|
|
234
|
+
let owner = io
|
|
235
|
+
const transfer = io.transfer
|
|
236
|
+
io.transfer = () => (owner = transfer.call(io))
|
|
237
|
+
|
|
238
|
+
return {
|
|
239
|
+
io,
|
|
240
|
+
getData: () => Buffer.from(buf.subarray(0, size)),
|
|
241
|
+
destroy: () => owner.destroy()
|
|
242
|
+
}
|
|
234
243
|
}
|
|
235
244
|
|
|
236
245
|
// A read-only in-memory source for FFmpeg (feeds the decoder our WAV bytes).
|
|
@@ -307,109 +316,145 @@ function encodeWithFfmpeg(pcm, format, { sampleRate = 48000, channels = 2 } = {}
|
|
|
307
316
|
const outRate = spec.rate || sampleRate
|
|
308
317
|
const sampleFormat = C.sampleFormats[spec.sampleFormat]
|
|
309
318
|
|
|
310
|
-
//
|
|
311
|
-
|
|
312
|
-
|
|
313
|
-
|
|
314
|
-
|
|
315
|
-
|
|
319
|
+
// bare-ffmpeg handles have no GC finalizer: until destroyed, the IOContexts hold
|
|
320
|
+
// strong refs to their callbacks, which pin the input WAV and the output buffer.
|
|
321
|
+
let inFmt = null
|
|
322
|
+
let decoder = null
|
|
323
|
+
let cc = null
|
|
324
|
+
let sink = null
|
|
325
|
+
let ofc = null
|
|
326
|
+
let resampler = null
|
|
327
|
+
let fifo = null
|
|
328
|
+
let packet = null
|
|
329
|
+
let inPacket = null
|
|
330
|
+
let raw = null
|
|
331
|
+
|
|
332
|
+
try {
|
|
333
|
+
// --- decode side: wrap PCM as WAV so ffmpeg produces clean input frames ---
|
|
334
|
+
const wav = pcmToWav(pcm, { sampleRate, channels })
|
|
335
|
+
inFmt = new ffmpeg.InputFormatContext(createReadIO(ffmpeg, wav))
|
|
336
|
+
const inStream = inFmt.streams[0]
|
|
337
|
+
decoder = inStream.decoder()
|
|
338
|
+
decoder.open()
|
|
316
339
|
|
|
317
|
-
|
|
318
|
-
|
|
319
|
-
|
|
320
|
-
|
|
321
|
-
|
|
322
|
-
|
|
323
|
-
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
|
|
328
|
-
|
|
340
|
+
// --- encoder ---
|
|
341
|
+
const encoder = new ffmpeg.Encoder(spec.encoder)
|
|
342
|
+
cc = new ffmpeg.CodecContext(encoder)
|
|
343
|
+
cc.sampleRate = outRate
|
|
344
|
+
cc.sampleFormat = sampleFormat
|
|
345
|
+
cc.channelLayout = ffmpeg.ChannelLayout.from(layout)
|
|
346
|
+
cc.timeBase = new ffmpeg.Rational(1, outRate)
|
|
347
|
+
if (spec.experimental) {
|
|
348
|
+
try {
|
|
349
|
+
cc.setOption('strict', 'experimental')
|
|
350
|
+
} catch {
|
|
351
|
+
/* older bindings: option may already default to permissive */
|
|
352
|
+
}
|
|
329
353
|
}
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
const frameSize = cc.frameSize > 0 ? cc.frameSize : 4096
|
|
354
|
+
cc.open()
|
|
355
|
+
const frameSize = cc.frameSize > 0 ? cc.frameSize : 4096
|
|
333
356
|
|
|
334
|
-
|
|
335
|
-
|
|
336
|
-
|
|
337
|
-
|
|
338
|
-
|
|
339
|
-
|
|
357
|
+
// --- muxer (seekable in-memory) ---
|
|
358
|
+
sink = createWriteIO(ffmpeg)
|
|
359
|
+
ofc = new ffmpeg.OutputFormatContext(spec.container, sink.io)
|
|
360
|
+
const ostream = ofc.createStream()
|
|
361
|
+
ostream.codecParameters.fromContext(cc)
|
|
362
|
+
ofc.writeHeader()
|
|
340
363
|
|
|
341
|
-
|
|
342
|
-
|
|
343
|
-
|
|
344
|
-
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
|
|
348
|
-
|
|
349
|
-
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
|
|
364
|
+
// --- resample (input format/rate -> encoder format/rate) + FIFO framing ---
|
|
365
|
+
resampler = new ffmpeg.Resampler(
|
|
366
|
+
inStream.codecParameters.sampleRate,
|
|
367
|
+
inStream.codecParameters.channelLayout,
|
|
368
|
+
inStream.codecParameters.format,
|
|
369
|
+
outRate,
|
|
370
|
+
layout,
|
|
371
|
+
sampleFormat
|
|
372
|
+
)
|
|
373
|
+
fifo = new ffmpeg.AudioFIFO(sampleFormat, channels, frameSize)
|
|
374
|
+
packet = new ffmpeg.Packet()
|
|
375
|
+
let pts = 0
|
|
353
376
|
|
|
354
|
-
|
|
355
|
-
|
|
356
|
-
|
|
357
|
-
|
|
358
|
-
|
|
359
|
-
|
|
360
|
-
|
|
377
|
+
const emit = (frame) => {
|
|
378
|
+
cc.sendFrame(frame)
|
|
379
|
+
while (cc.receivePacket(packet)) {
|
|
380
|
+
packet.streamIndex = ostream.index
|
|
381
|
+
packet.rescaleTimestamps(cc.timeBase, ostream.timeBase)
|
|
382
|
+
ofc.writeFrame(packet)
|
|
383
|
+
packet.unref()
|
|
384
|
+
}
|
|
361
385
|
}
|
|
362
|
-
}
|
|
363
386
|
|
|
364
|
-
|
|
365
|
-
|
|
366
|
-
|
|
367
|
-
|
|
368
|
-
|
|
369
|
-
|
|
370
|
-
|
|
371
|
-
|
|
372
|
-
|
|
373
|
-
|
|
374
|
-
|
|
375
|
-
|
|
376
|
-
|
|
387
|
+
const drain = (flush) => {
|
|
388
|
+
while (fifo.size >= frameSize || (flush && fifo.size > 0)) {
|
|
389
|
+
const n = Math.min(frameSize, fifo.size)
|
|
390
|
+
const frame = new ffmpeg.Frame()
|
|
391
|
+
try {
|
|
392
|
+
frame.format = sampleFormat
|
|
393
|
+
frame.channelLayout = ffmpeg.ChannelLayout.from(layout)
|
|
394
|
+
frame.sampleRate = outRate
|
|
395
|
+
frame.nbSamples = n
|
|
396
|
+
frame.alloc()
|
|
397
|
+
fifo.read(frame, n)
|
|
398
|
+
frame.pts = pts
|
|
399
|
+
pts += n
|
|
400
|
+
emit(frame)
|
|
401
|
+
} finally {
|
|
402
|
+
frame.destroy()
|
|
403
|
+
}
|
|
404
|
+
}
|
|
377
405
|
}
|
|
378
|
-
}
|
|
379
406
|
|
|
380
|
-
|
|
381
|
-
|
|
382
|
-
|
|
383
|
-
|
|
384
|
-
|
|
385
|
-
|
|
386
|
-
|
|
387
|
-
|
|
388
|
-
|
|
389
|
-
|
|
390
|
-
|
|
391
|
-
|
|
392
|
-
|
|
407
|
+
const pushResampled = (rawFrame) => {
|
|
408
|
+
const out = new ffmpeg.Frame()
|
|
409
|
+
try {
|
|
410
|
+
out.channelLayout = ffmpeg.ChannelLayout.from(layout)
|
|
411
|
+
out.format = sampleFormat
|
|
412
|
+
out.sampleRate = outRate
|
|
413
|
+
const inSamples = rawFrame ? rawFrame.nbSamples : 1024
|
|
414
|
+
out.nbSamples =
|
|
415
|
+
Math.ceil((inSamples * outRate) / inStream.codecParameters.sampleRate) + 1024
|
|
416
|
+
const samples = new ffmpeg.Samples()
|
|
417
|
+
samples.fill(out)
|
|
418
|
+
const count = rawFrame ? resampler.convert(rawFrame, out) : resampler.flush(out)
|
|
419
|
+
if (count > 0) {
|
|
420
|
+
out.nbSamples = count
|
|
421
|
+
fifo.write(out)
|
|
422
|
+
}
|
|
423
|
+
return count
|
|
424
|
+
} finally {
|
|
425
|
+
out.destroy()
|
|
426
|
+
}
|
|
393
427
|
}
|
|
394
|
-
return count
|
|
395
|
-
}
|
|
396
428
|
|
|
397
|
-
|
|
398
|
-
|
|
399
|
-
|
|
400
|
-
|
|
401
|
-
|
|
402
|
-
|
|
403
|
-
|
|
404
|
-
|
|
405
|
-
|
|
406
|
-
|
|
407
|
-
|
|
408
|
-
|
|
409
|
-
|
|
410
|
-
|
|
429
|
+
inPacket = new ffmpeg.Packet()
|
|
430
|
+
raw = new ffmpeg.Frame()
|
|
431
|
+
while (inFmt.readFrame(inPacket)) {
|
|
432
|
+
decoder.sendPacket(inPacket)
|
|
433
|
+
while (decoder.receiveFrame(raw)) pushResampled(raw)
|
|
434
|
+
inPacket.unref()
|
|
435
|
+
drain(false)
|
|
436
|
+
}
|
|
437
|
+
while (pushResampled(null) > 0) {
|
|
438
|
+
/* flush the resampler's internal buffer */
|
|
439
|
+
}
|
|
440
|
+
drain(true)
|
|
441
|
+
emit(null) // flush the encoder
|
|
442
|
+
ofc.writeTrailer()
|
|
411
443
|
|
|
412
|
-
|
|
444
|
+
return sink.getData()
|
|
445
|
+
} finally {
|
|
446
|
+
raw?.destroy()
|
|
447
|
+
inPacket?.destroy()
|
|
448
|
+
packet?.destroy()
|
|
449
|
+
fifo?.destroy()
|
|
450
|
+
resampler?.destroy()
|
|
451
|
+
cc?.destroy()
|
|
452
|
+
decoder?.destroy()
|
|
453
|
+
ofc?.destroy()
|
|
454
|
+
// No-op once ofc owns the IO; releases it if OutputFormatContext was never created.
|
|
455
|
+
sink?.destroy()
|
|
456
|
+
inFmt?.destroy()
|
|
457
|
+
}
|
|
413
458
|
}
|
|
414
459
|
|
|
415
460
|
// Encode a single format. `pcm`/`wav` stay dependency-free; the rest go through
|
package/package.json
CHANGED
|
@@ -1,10 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@qvac/audiogen-ggml",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.11.0",
|
|
4
4
|
"description": "Audio generation (music) addon for qvac",
|
|
5
|
-
"qvac": {
|
|
6
|
-
"ggml-speech": "2026-10-27#1"
|
|
7
|
-
},
|
|
8
5
|
"engines": {
|
|
9
6
|
"bare": ">=1.19.0"
|
|
10
7
|
},
|
|
@@ -35,9 +32,7 @@
|
|
|
35
32
|
"scripts": {
|
|
36
33
|
"build": "npm run build:ts && npm run build:native",
|
|
37
34
|
"build:native": "bare-make generate -D BUILD_FUZZING=OFF && bare-make build && bare-make install && npm run link:platform",
|
|
38
|
-
"build:native:cuda": "bare-make generate -D ENABLE_CUDA=ON -D BUILD_FUZZING=OFF && bare-make build && bare-make install && npm run link:platform",
|
|
39
35
|
"link:platform": "node ../../scripts/ci/slice-platform-packages.mjs --link-local --workdir .",
|
|
40
|
-
"build:cuda": "npm run build:ts && npm run build:native:cuda",
|
|
41
36
|
"build:ts": "tsc -p tsconfig.build.json",
|
|
42
37
|
"build:pack": "npm run build:ts && mkdir -p dist && npm pack --pack-destination dist",
|
|
43
38
|
"example": "bare examples/generate-music.js \"Upbeat pop rock with driving electric guitars and a catchy hook\"",
|
|
@@ -146,7 +141,17 @@
|
|
|
146
141
|
"prettier-config-holepunch": "^2.0.0",
|
|
147
142
|
"typescript": "^6.0.3",
|
|
148
143
|
"typescript-eslint": "^8.63.0",
|
|
149
|
-
"cmake-npm": "^1.1.0"
|
|
144
|
+
"cmake-npm": "^1.1.0",
|
|
145
|
+
"bare": "^1.36.0",
|
|
146
|
+
"bare-runtime": "^1.36.0",
|
|
147
|
+
"bare-make": "^2.1.0",
|
|
148
|
+
"cmake-toolchains": "^2.1.0",
|
|
149
|
+
"cmake-runtime": "^4.1.1",
|
|
150
|
+
"llvm-runtime": "^23.1.0",
|
|
151
|
+
"nasm-runtime": "^3.2.0",
|
|
152
|
+
"ninja-runtime": "^1.13.1",
|
|
153
|
+
"@qvac/fabric-android-arm64": "^0.31.0",
|
|
154
|
+
"@qvac/fabric-ios": "^0.31.0"
|
|
150
155
|
},
|
|
151
156
|
"dependencies": {
|
|
152
157
|
"@qvac/error": "^0.1.1",
|
|
@@ -156,24 +161,33 @@
|
|
|
156
161
|
"bare-fs": "^4.5.6",
|
|
157
162
|
"bare-os": "^3.8.0",
|
|
158
163
|
"bare-path": "^3.0.0",
|
|
159
|
-
"bare-process": "^4.2.2"
|
|
164
|
+
"bare-process": "^4.2.2",
|
|
165
|
+
"@qvac/fabric": "^0.31.0"
|
|
160
166
|
},
|
|
161
167
|
"optionalDependencies": {
|
|
162
|
-
"@qvac/audiogen-ggml-linux-x64": "0.
|
|
163
|
-
"@qvac/audiogen-ggml-linux-arm64": "0.
|
|
164
|
-
"@qvac/audiogen-ggml-darwin-arm64": "0.
|
|
165
|
-
"@qvac/audiogen-ggml-darwin-x64": "0.
|
|
166
|
-
"@qvac/audiogen-ggml-win32-x64": "0.
|
|
168
|
+
"@qvac/audiogen-ggml-linux-x64": "0.11.0",
|
|
169
|
+
"@qvac/audiogen-ggml-linux-arm64": "0.11.0",
|
|
170
|
+
"@qvac/audiogen-ggml-darwin-arm64": "0.11.0",
|
|
171
|
+
"@qvac/audiogen-ggml-darwin-x64": "0.11.0",
|
|
172
|
+
"@qvac/audiogen-ggml-win32-x64": "0.11.0"
|
|
167
173
|
},
|
|
168
174
|
"overrides": {
|
|
169
175
|
"bare-module-lexer": "1.6.3"
|
|
170
176
|
},
|
|
171
177
|
"types": "index.d.ts",
|
|
172
178
|
"peerDependencies": {
|
|
173
|
-
"@qvac/
|
|
174
|
-
"@qvac/
|
|
179
|
+
"@qvac/fabric-android-arm64": "^0.31.0",
|
|
180
|
+
"@qvac/fabric-ios": "^0.31.0",
|
|
181
|
+
"@qvac/audiogen-ggml-android-arm64": "0.11.0",
|
|
182
|
+
"@qvac/audiogen-ggml-ios": "0.11.0"
|
|
175
183
|
},
|
|
176
184
|
"peerDependenciesMeta": {
|
|
185
|
+
"@qvac/fabric-android-arm64": {
|
|
186
|
+
"optional": true
|
|
187
|
+
},
|
|
188
|
+
"@qvac/fabric-ios": {
|
|
189
|
+
"optional": true
|
|
190
|
+
},
|
|
177
191
|
"@qvac/audiogen-ggml-android-arm64": {
|
|
178
192
|
"optional": true
|
|
179
193
|
},
|