@audio/neural-separate 0.3.0 → 0.4.0

package/NOTICE

@@ -56,9 +56,11 @@ The MIT License, for each notice above:
     OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
     DEALINGS IN THE SOFTWARE.
 
-No model weights are included. The export scripts write them to the local cache; their terms are
-their authors' (Open-Unmix umxhq: MIT; SCNet and SCNet-large: MIT, Copyright (c) 2024 starrytong,
-redistributable converted with attribution, github.com/starrytong/SCNet/issues/35; Demucs htdemucs
+No model weights are included. The export scripts write them to the local cache, and the compact
+files of SCNet-large, SCNet, MRX and TIGER are fetched from huggingface.co/audiojs (scnet-large, scnet,
+mrx, tiger-dnr, each with its licence and model card); their terms are their authors' (Open-Unmix
+umxhq: MIT; SCNet and SCNet-large: MIT, Copyright (c) 2024 starrytong, redistributable converted with
+attribution, github.com/starrytong/SCNet/issues/35; Demucs htdemucs
 weights: see the README; MRX: MIT, Copyright 2023 Mitsubishi Electric Research Laboratories (MERL), the
 repository's .reuse/dep5 naming each checkpoint; TIGER's DnR model: Apache 2.0, its model card,
 huggingface.co/JusperLee/TIGER-DnR).

package/README.md

@@ -26,8 +26,8 @@ In the browser `@audio/neural-runtime` runs `onnxruntime-web` instead (`device:
 | `'umxhq'` | Open-Unmix: bi-LSTM on magnitude spectrograms, multichannel Wiener EM | 4 × 35.6 MB (fp16: 4 × 17.8 MB) | MIT | 6.75 · 6.11 · 5.00 · 3.36 | 0.19 (vocals only: 0.11) · 0.12 · 0.29 |
 | `'htdemucs'` | Hybrid Transformer Demucs: waveform and spectrogram U-Nets joined by a cross-domain transformer | 174 MB | research only | 8.86 · 9.55 · 9.30 · 5.69 | 1.4 (vocals only: 1.2) · 0.70 · 0.21 |
 | `'htdemucs_ft'` | four fine-tuned HTDemucs, one per source | 4 × 174 MB | research only | 8.67 · 9.45 · 9.68 · 5.58 | vocals only: 1.1 · not measured · not measured |
-| `'scnet-large'` | SCNet-large: band-split convolutions around dual-path LSTMs, on the complex spectrogram | 169 MB (fp16: 85 MB) | MIT | 11.00 · 10.27 · 8.21 · 6.87 | 2.1 · not measured · not measured |
-| `'scnet'` | SCNet: the same at half the width | 43 MB | MIT | 9.88 · 9.43 · 8.35 · 6.15 | 0.70 · not measured · not measured |
+| `'scnet-large'` | SCNet-large: band-split convolutions around dual-path LSTMs, on the complex spectrogram | 45 MB (export 169 MB) | MIT | 11.00 · 10.27 · 8.21 · 6.87 | 2.1 · 7.8 · 5.0 |
+| `'scnet'` | SCNet: the same at half the width | 13 MB (export 43 MB) | MIT | 9.88 · 9.43 · 8.35 · 6.15 | 0.70 · 1.7 · 0.91 |
 
 SDR as measured below (Verification), identical to the Python originals'. The previews are 7 s excerpts, so these numbers sit apart from full-track results in scale, not in order: the Demucs README reports overall SDR 9.0 for fine-tuned HT Demucs against 5.3 for Open-Unmix on the MUSDB18-HQ test set.
 
@@ -35,10 +35,10 @@ SDR as measured below (Verification), identical to the Python originals'. The pr
 
 A soundtrack's stems, Divide and Remaster's three (the cocktail fork problem: Petermann, Wichern, Wang, Le Roux, ICASSP 2022):
 
-| `model` | Architecture | Files | Weights | Targets | Real-time factor, Node CPU |
+| `model` | Architecture | Files | Weights | Targets | Real-time factor: Node CPU · browser wasm · WebGPU |
 |---|---|---|---|---|---|
-| `'mrx'` | MRX: magnitudes at three STFT resolutions (Hann 1024, 2048, 8192, hop 256) into one hidden layer, a BLSTM per source, their mean, a real mask per source and resolution | 122 MB | MIT | `dialogue`, `music`, `effects` | 0.21 |
-| `'tiger'` | TIGER: three band-split models (57 bands, multi-scale convolutions and frame and frequency attention, 1.4 M parameters each), a complex mask each, one source kept from each | 29 MB | Apache 2.0 | `dialogue`, `music`, `effects` | 7.5–23 |
+| `'mrx'` | MRX: magnitudes at three STFT resolutions (Hann 1024, 2048, 8192, hop 256) into one hidden layer, a BLSTM per source, their mean, a real mask per source and resolution | 31 MB (export 122 MB) | MIT | `dialogue`, `music`, `effects` | 0.21 · 0.31 · 0.75 |
+| `'tiger'` | TIGER: three band-split models (57 bands, multi-scale convolutions and frame and frequency attention, 1.4 M parameters each), a complex mask each, one source kept from each | 7.3 MB (export 29 MB) | Apache 2.0 | `dialogue`, `music`, `effects` | 7.5–23 · 9.2 · 3.1 |
 
 Measured on Divide and Remaster v3's English test set (Watcharasupat, Wu, Orife, 2024; CC BY-SA 4.0, built from sources that allow commercial use), 150 of its 1200 clips of 60 s, every 8th ([`audio`](https://github.com/audiojs/audio)'s `bench/rx/scene.mjs`): each stem against its reference over the whole clip, SNR the median over clips (the measure Bandit v2's paper reports on this set) and SI-SDR the mean (MRX's on v2); remixes, a stem 6 dB up or down as the input plus (g − 1) times it, against the true remix, SNR, the median.
 
@@ -53,11 +53,24 @@ Measured on Divide and Remaster v3's English test set (Watcharasupat, Wu, Orife,
 
 TIGER, at about 50 times MRX's time (RTF 10.6 against 0.21 here), is ahead on 29, 28 and 29 of the 30 clips (dialogue, music, effects), by a median 2.3, 3.7 and 3.4 dB. MRX's own README reports 12.5 · 4.2 · 5.7 dB SI-SDR on DnR v2, the set it trained on, whose music holds singing; v3's music holds none, and its loudness and languages differ (v3's paper). DnR v2's test split comes only inside a 116 GB gzip of the whole set; v3's clips are fetched one by one. The real-time factors are of one run on a 14-core M4 Max shared with other jobs (load averages 25 to 60), so upper bounds: `mrx` a 60 s clip in 13 s; `tiger` three models of 15,000 small operators each, every sample in three 12 s segments.
 
+The browser's, the shipped files (Size, below) on 30 s (a song; a Divide and Remaster clip), model load included:
+onnxruntime-web 1.30 in headless Chromium 153, wasm on its default 4 threads, WebGPU on Metal, the faster of two runs
+on a 14-core M4 Max shared with other jobs (load averages 110 to 300; the other run, at 9 to 140 with another job
+training on the GPU, took 1.5 to 2 times as long), so upper bounds. Peak memory, the browser's processes
+together (0.9 GB of it the browser idle): wasm `scnet` 3.0 GB, `scnet-large` 4.6, `mrx` 3.0, `tiger` 3.5; WebGPU 2.1,
+2.4, 3.4, 1.5. On WebGPU `tiger`'s session takes 13 s to make (38,817 nodes), a 60 s clip about 3 minutes, MRX's 45 s;
+on wasm 9 and 0.3 minutes. Float16 compute does not pay: `tiger`'s on WebGPU is 13 % faster and its music 4.7 dB from
+the float32 graph's.
+
 `targets` picks a subset: `targets: ['vocals']` runs only the vocals graph of `umxhq` (against the residual, see Algorithm: 2.4× faster, vocals SDR 6.50 instead of 6.75) and of `htdemucs_ft`, and skips the other sources' iSTFT for `htdemucs`. Presets resample to their 44.1 kHz and back.
 
 ## Weights
 
-Nothing ships in this package and nothing is fetched by default. Export the weights once from the original checkpoints:
+Nothing ships in this package. `scnet-large`, `scnet`, `mrx` and `tiger` read a compact file each, `<model>.int8.onnx`
+(Size, below), hosted on Hugging Face (`audiojs/scnet-large`, `audiojs/scnet`, `audiojs/mrx`, `audiojs/tiger-dnr`, each
+with its model card and licence): without `weights`, `https://huggingface.co/<repo>/resolve/<revision>/<file>` at the
+revision `REVISIONS` pins, fetched once and checked against its SHA-256 (the browser keeps it in the Cache API, Node in
+the neural cache, below). Every model can also be exported from its original checkpoint:
 
 ```sh
 pip install torch openunmix onnx onnxruntime onnxscript
@@ -74,9 +87,11 @@ python3 node_modules/@audio/neural-separate/scripts/export-mrx.py --verify
 
 pip install safetensors
 python3 node_modules/@audio/neural-separate/scripts/export-tiger.py --verify
+
+python3 node_modules/@audio/neural-separate/scripts/compact.py --model tiger --verify   # tiger.int8.onnx from tiger.onnx
 ```
 
-They land in `$AUDIO_NEURAL_CACHE/<model>/`, default `~/.cache/audiojs/neural/<model>/`, where a preset looks in Node; a missing file throws, naming itself and the script. Elsewhere, serve that directory and pass its URL: `separate(audio, { model: 'umxhq', weights: 'https://…/' })`; `@audio/neural-runtime` fetches each file once and caches it (Cache API in the browser). `--verify` compares the ONNX graphs with the PyTorch modules and writes the reference separation `test.js` checks the JS pipeline against. `export-openunmix.py --fp16` also writes float16 weights with float32 I/O, half the size.
+They land in `$AUDIO_NEURAL_CACHE/<model>/`, default `~/.cache/audiojs/neural/<model>/`, where a preset looks in Node: its compact file when it is the hosted one (its SHA-256), else its export (float32, the numbers the Python original gives), else the hosted file, fetched there once and written whole (offline, a compact file of another SHA-256, `compact.py`'s own build, is read as it is); a missing file throws, naming itself and the script. `weights` a directory: its files as they are, the compact one first. Elsewhere, serve that directory and pass its URL: `separate(audio, { model: 'umxhq', weights: 'https://…/' })`; `@audio/neural-runtime` fetches each file once and caches it (Cache API in the browser). `--verify` compares the ONNX graphs with the PyTorch modules and writes the reference separation `test.js` checks the JS pipeline against. `export-openunmix.py --fp16` also writes float16 weights with float32 I/O, half the size; `export-scnet.py --fp16` float16 weights with float32 compute (`compact.py --as fp16`).
 
 - **`export-openunmix.py`** exports each target's `OpenUnmix` through a wrapper that takes the frame count from the traced tensor (upstream reads it off `x.data.shape`, which export freezes), with the TorchScript exporter (the `torch.export` one bakes the LSTM's sequence length into a reshape). Magnitude `[1, C, F, T]` in and out, T dynamic. `umxhq`'s bandwidth restriction (`max_bin`, 1487 of 2049 bins for `n_fft=4096`@44.1kHz) crops the network's *input* below 16 kHz for efficiency; the final dense layer regresses the **full** bin range from that reduced representation (a learned extrapolation, not a literal zero-fill).
 - **`export-scnet.py`** fetches SCNet's code from ZFTurbo/Music-Source-Separation-Training at a pinned commit and the author's MUSDB18-HQ checkpoint from that repository's releases, and exports the network between its STFT and iSTFT (the same split as demucs.onnx's): `mix_spec [1, 4, 2049, 476]` in, `stems_spec [1, 16, 2049, 476]` out, one 11 s segment. Two changes to the graph, neither to its function: the rFFT and irFFT over time in its separation network become products with cosine and sine matrices (torch 2.14 exports no `fft_rfft`), and each `GroupNorm(1, C)` reduces its mean and variance one axis at a time: exported whole, onnxruntime summed a 10-million-value group in float32 and the stems came out 52 to 55 dB from PyTorch's, against 110 dB now (PyTorch's own float32 against float64: 110 dB). `--fp16` also writes float16 weights.
@@ -84,6 +99,63 @@ They land in `$AUDIO_NEURAL_CACHE/<model>/`, default `~/.cache/audiojs/neural/<m
 - **`export-tiger.py`** fetches TIGER's model code (JusperLee/TIGER, `look2hear/models/tiger_dnr.py` and the two layer modules it reads) at a pinned commit and its Divide and Remaster weights (huggingface.co/JusperLee/TIGER-DnR) at a pinned revision, and exports TIGERDNR's three models between their STFT and iSTFT as one graph, each its kept source (the first's dialogue, the second's effects, the third's music): `mix_spec [1, 2, 1025, 1034]` (one channel's STFT, n 2048, hop 512, Hann, unnormalized) in, `stems_spec [1, 6, 1025, 1034]` (dialogue, music, effects, re and im) out, one 12 s segment. Three changes to the graph, none to its function: `F.adaptive_avg_pool1d`, which the exporter takes only for output sizes dividing the input's, becomes a cumulative sum read at torch's windows; UConvBlock's global sum starts at its first term, not at a zeros tensor exported as a constant the size of the features; each `GroupNorm(1, C)` reduces axis by axis, as SCNet's. ONNX Runtime runs the graph in 12 s segments at about twice real time on a CPU: three models of 15,000 small operators each, every sample in three segments.
 - **`export-htdemucs.py`** follows [sevagh/demucs.onnx](https://github.com/sevagh/demucs.onnx): the STFT and iSTFT, which ONNX export cannot carry, move out of the graph. Rather than a vendored copy of `htdemucs.py`, it runs upstream's own `forward` with its four transform methods swapped on the instance. One graph per 7.8 s segment: `mix [1, 2, 343980]` and `mix_spec [1, 4, 2048, 336]` (complex as channels) in, `stems_spec [1, 4, 4, 2048, 336]` and `stems_wave [1, 4, 2, 343980]` out; a source is the iSTFT of the first plus the second.
 
+## Size
+
+`scnet-large`, `scnet`, `mrx` and `tiger` each read one compact file, `<model>.int8.onnx`, that `scripts/compact.py`
+makes from the export: the weights stored in 8 bits (symmetric, a scale per output channel; an LSTM's per gate row and
+direction), the few whose rounding costs most in 16, computed in float32: onnxruntime folds each weight's Cast and scale
+at load, on every backend, so the session holds float32 weights and runs as fast as the export. A variant is kept when
+its cost is negligible: every stem's median within 0.05 dB of the export's (SDR on the 50 MUSDB18 test previews, SNR on
+Divide and Remaster v3 test clips) and no song or clip worse by 0.5 dB or more; the smallest such is shipped.
+
+| `model` | export (float32) | float16 weights | int8, every weight | **shipped**: int8, the costliest float16 | its gzip · brotli |
+|---|---|---|---|---|---|
+| `'scnet-large'` | 169.2 MB | 86.5 MB | 44.7 MB | **45.1 MB**: 83 of 88 weights int8 | 39.8 · 39.0 MB |
+| `'scnet'` | 42.8 MB | 23.2 MB | 12.8 MB | **12.9 MB**: 77 of 82 | 10.7 · 10.2 MB |
+| `'mrx'` | 122.3 MB | 61.5 MB | 31.3 MB | **31.3 MB**: all 39 | 27.5 · 26.6 MB |
+| `'tiger'` | 28.6 MB (18.8 folded) | 10.9 MB | 7.3 MB | **7.3 MB**: all 444 | 5.2 · 2.6 MB |
+
+Against the export, on the same songs or clips: the median per stem, then the largest loss on one song or clip, dB; and
+the graph's own output against the export's on a calibration excerpt (two MUSDB18 training previews; a Divide and
+Remaster v3 tuning clip), SNR.
+
+| `model`, measure | export | float16 weights | int8, every weight | shipped |
+|---|---|---|---|---|
+| `'scnet-large'`, SDR: vocals · drums · bass · other | 11.00 · 10.27 · 8.21 · 6.87 | 10.99 · 10.27 · 8.21 · 6.87 | | **10.96 · 10.26 · 8.20 · 6.92** |
+| … the song that lost most | | −0.004 | | −0.43 |
+| … output, calibration | | 62.3 dB | 24.0 dB | 43.7 dB |
+| `'scnet'`, SDR | 9.88 · 9.43 · 8.35 · 6.15 | 9.88 · 9.43 · 8.35 · 6.15 | 9.87 · 9.44 · 8.32 · 6.13 | **9.88 · 9.44 · 8.35 · 6.14** |
+| … the song that lost most | | −0.004 | −2.61 | −0.29 |
+| … output, calibration | | 60.6 dB | 23.4 dB | 42.4 dB |
+| `'mrx'`, SNR, 30 clips: dialogue · music · effects | 10.92 · 5.17 · 5.72 | 10.92 · 5.17 · 5.72 | = shipped | **10.92 · 5.17 · 5.70** |
+| … the clip that lost most | | −0.002 | | −0.04 |
+| … output (masked spectra), calibration | | 73.8 dB | 40.4 dB | 40.4 dB |
+| `'tiger'`, SNR, the same 30 clips | 12.68 · 10.24 · 8.06 | | = shipped | **12.66 · 10.23 · 8.03** |
+| … the clip that lost most | | | | −0.10 |
+| … output, calibration | | 73.9 dB | 41.4 dB | 41.4 dB |
+
+Per song or clip the median change is within 0.015 dB everywhere; the largest losses fall where a stem is near silence:
+SCNet's and SCNet-large's on PR - Happy Daze, whose vocals they separate at −2 dB SDR as exported (`scnet` with all 82
+weights in int8: −2.6 dB there).
+The remixes (`rebalance`'s, `scene`'s) move as little: vocals +6 dB 20.21 → 20.22 (`scnet-large`), dialogue +6 dB
+18.29 → 18.30 (`mrx`).
+
+Which weights stay float16: each weight rounded alone to int8 on a calibration excerpt (two MUSDB18 training previews;
+a Divide and Remaster v3 tuning clip), the others float32, its cost the error it adds over the output's power; the
+weights go to int8 in order of that cost per byte saved while the costs' sum stays 40 dB under the output (separate
+weights' costs add: the sum predicted the whole to 0.2 dB). SCNet keeps the five layers ending its decoder in float16,
+1 % of its values (`decoder.2.0`'s convolution, `decoder.1.1`'s three transposed convolutions, `decoder.2.1`'s first:
+each alone at int8 26.8 to 39.0 dB); MRX (40.4 dB) and TIGER (41.4 dB) keep none.
+
+Tried and left out: 4-bit weights (blocks of 32: SCNet's stems 7 dB from float32's; its LSTMs and linear layers alone,
+18 dB); int8 scales per block of 32 values rather than per channel (31.5 dB against 26.3: still the same decoder
+layers); DequantizeLinear for the int8 weights (onnxruntime 1.30 fuses it with the MatMul it feeds into MatMulNBits,
+which quantizes the activations to int8 on a CPU: MRX 38.4 dB from float32 so, 40.4 as stored).
+
+Hosts send these files as they are: Hugging Face's CDN and raw.githubusercontent.com set no Content-Encoding on a binary
+(both answer any origin, `Access-Control-Allow-Origin: *`). gzip would save 12 to 17 % of SCNet's and MRX's, 29 % of
+TIGER's, brotli 14 to 21 % and 65 % (TIGER's graph, 38,817 nodes, is most of its file): not worth a decoder in the page.
+
 ## Algorithm
 
 **`'openunmix'`, `'mask'`** (spectrogram models):
@@ -111,12 +183,16 @@ TIGER's DnR model runs as `'complex'` with its own segments, those of `TIGERDNR.
 | ONNX vs PyTorch module, random input (`--verify`) | umxhq max \|diff\| ≤ 5.7e-6 of max \|y\| (fp16: ≤ 5.3e-3); stems of htdemucs ≤ 1.6e-4, of htdemucs_ft ≤ 2.5e-4 |
 | Pipeline vs `openunmix.Separator` in float64, 9 s reference mix (`test.js`) | 112–134 dB SNR per stem |
 | Pipeline vs `demucs.apply.apply_model`, same mix (`test.js`) | htdemucs 80–84 dB, htdemucs_ft 78–89 dB SNR per stem |
-| ONNX vs `SCNet.forward`, noise and tones (`export-scnet.py --verify`) | scnet-large max \|diff\| ≤ 2.8e-6 of max \|y\| (fp16: ≤ 6.7e-3), scnet ≤ 2.2e-6 |
+| ONNX vs `SCNet.forward`, noise and tones (`export-scnet.py --verify`) | scnet-large max \|diff\| ≤ 2.8e-6 of max \|y\|, scnet ≤ 2.2e-6 |
 | Pipeline vs `SCNet.forward` on `reference.py`'s segments, a 20 s mix (`test.js`) | scnet-large 114–134 dB, scnet 123–133 dB SNR per stem |
 | ONNX vs `MRX.forward`, noise and tones (`export-mrx.py --verify`) | max \|diff\| ≤ 1.2e-6 of max \|y\| |
 | Pipeline vs `separate.separate_soundtrack` (pyloudnorm -27 LUFS in and back), a 20 s mix (`test.js`) | 101–134 dB SNR per stem, in one chunk; in 8 s chunks 12–50 dB |
 | ONNX vs the three `TIGER.forward`, one 12 s segment of tones and noise (`export-tiger.py --verify`) | max \|diff\| ≤ 4.2e-7 of max \|y\| |
 | Pipeline vs `TIGERDNR.wav_chunk_inference`, its segments through the verified graph, a 20 s mix (`test.js`) | 95–140 dB SNR per stem |
+| Compact file vs export, a calibration excerpt (`compact.py --verify`) | scnet-large 43.7 dB, scnet 42.4, mrx 40.4, tiger 41.4 SNR (the worst output); folding alone: bit for bit |
+| Pipeline, compact file vs `reference.py`'s stems, the same 20 s mix (`test.js`) | each stem's error under the mixture: scnet-large 25.7 dB (its other and vocals, which trade the synthetic voice; drums and bass 51), scnet 47.0, mrx 45.6, tiger 58.9 |
+| Hosted files: the pinned revisions' SHA-256 (`test.js`, online); a fetched file of another refused | as `models[name].sha256` |
+| Browser (onnxruntime-web, Chromium) vs Node, the same compact file, 30 s | `rebalance` and `scene` through `audio`, 8 s, the files fetched from Hugging Face: the output 135 and 145 dB from Node's on wasm, 134 and 148 dB on WebGPU |
 | `wienerFilter` vs open-unmix's `wiener()`, float64 fixture (`test.js`) | ratio mask: 3e-12; mixture phase: 1.7e-7, upstream's `atan2` adds a float32 π |
 
 The 50 test tracks of the MUSDB18 7 s previews (`musdb.DB(download=True)`; its terms are educational and non-commercial, so it serves measurement only), SDR by [museval](https://github.com/sigsep/sigsep-mus-eval) (BSSEval v4, 1 s windows, median over windows, then over tracks: the SiSEC 2018 aggregation):
@@ -174,7 +250,7 @@ let vocals = complex.vocals.map(ch => istft(ch, { length: left.length }))
 | `sampleRate` | — | required unless `audio` is `{ channelData, sampleRate }` |
 | `model` | — | required. Preset `'scnet-large' \| 'scnet' \| 'umxhq' \| 'htdemucs' \| 'htdemucs_ft' \| 'mrx' \| 'tiger'` · `url \| bytes` (single target, named `'stem'`) · `{ target: url \| { url, targets }, ... }` (one graph per target, Open-Unmix's own layout; a multi-source graph contributes its own target) · `{ url, targets: [...] }` (one multi-target graph, stacks a target axis) |
 | `targets` | all | the targets to return |
-| `weights` | `$AUDIO_NEURAL_CACHE` or `~/.cache/audiojs/neural` (Node) | where a preset's files are: URL, or a directory in Node |
+| `weights` | `$AUDIO_NEURAL_CACHE` or `~/.cache/audiojs/neural` (Node); a compact preset's hosted file (`REVISIONS`) | where a preset's files are: URL, or a directory in Node |
 | `modelType` | `'openunmix'` | `'openunmix'` (magnitude out) · `'mask'` (`[0,1]` mask out, multiplied by mixture magnitude) · `'hybrid'` (demucs.onnx contract) · `'complex'` (complex spectrogram in and out, SCNet's) · `'multires'` (magnitudes at several resolutions in, masks out, MRX's) · `'waveform'` (Demucs v2-class); presets know theirs |
 | `wiener` | `1` | EM iterations; `0` = raw masks. Ignored for `'hybrid'` and `'waveform'` |
 | `wienerWindow` | `300` | frames per EM window |
@@ -210,7 +286,7 @@ Audit any weight source yourself before shipping it — this table reflects what
 
 ## Reference
 
-Stöter, Uhlich, Liutkus, Mitsufuji, "Open-Unmix - A Reference Implementation for Music Source Separation," *JOSS* 4(41), 2019. · Rouard, Massa, Défossez, "Hybrid Transformers for Music Source Separation," *ICASSP* 2023. · Xu, Li, Chen, Hu, "TIGER: Time-frequency Interleaved Gain Extraction and Reconstruction for Efficient Speech Separation," *ICLR* 2025. · Petermann, Wichern, Wang, Le Roux, "The Cocktail Fork Problem: Three-Stem Audio Separation for Real-World Soundtracks," *ICASSP* 2022. · Watcharasupat, Wu, Orife, "Remastering Divide and Remaster: A Cinematic Audio Source Separation Dataset with Multilingual Support," 2024. · Tong, Zhang, Liu, Li, Yu, "SCNet: Sparse Compression Network for Music Source Separation," *ICASSP* 2024. · Duong, Vincent, Gribonval, "Under-determined reverberant audio source separation using a full-rank spatial covariance model," *IEEE TASLP* 18(7), 2010. · Rafii, Liutkus, Stöter, Mimilakis, Bittner, "The MUSDB18 corpus for music separation," 2017. · [norbert](https://github.com/sigsep/norbert) (Liutkus & Stöter) · [open-unmix-pytorch](https://github.com/sigsep/open-unmix-pytorch) · [Demucs](https://github.com/facebookresearch/demucs) · [demucs.onnx](https://github.com/sevagh/demucs.onnx) · [SCNet](https://github.com/starrytong/SCNet) · [cocktail-fork-separation](https://github.com/merlresearch/cocktail-fork-separation) · [TIGER](https://github.com/JusperLee/TIGER) · [pyloudnorm](https://github.com/csteinmetz1/pyloudnorm) · [Music-Source-Separation-Training](https://github.com/ZFTurbo/Music-Source-Separation-Training) · [museval](https://github.com/sigsep/sigsep-mus-eval) · [Spleeter](https://github.com/deezer/spleeter).
+Stöter, Uhlich, Liutkus, Mitsufuji, "Open-Unmix - A Reference Implementation for Music Source Separation," *JOSS* 4(41), 2019. · Rouard, Massa, Défossez, "Hybrid Transformers for Music Source Separation," *ICASSP* 2023. · Xu, Li, Chen, Hu, "TIGER: Time-frequency Interleaved Gain Extraction and Reconstruction for Efficient Speech Separation," *ICLR* 2025. · Petermann, Wichern, Wang, Le Roux, "The Cocktail Fork Problem: Three-Stem Audio Separation for Real-World Soundtracks," *ICASSP* 2022. · Watcharasupat, Wu, Orife, "Remastering Divide and Remaster: A Cinematic Audio Source Separation Dataset with Multilingual Support," 2024. · Tong, Zhu, Chen, Kang, Jiang, Li, Wu, Meng, "SCNet: Sparse Compression Network for Music Source Separation," *ICASSP* 2024 (arXiv:2401.13276). · Duong, Vincent, Gribonval, "Under-determined reverberant audio source separation using a full-rank spatial covariance model," *IEEE TASLP* 18(7), 2010. · Rafii, Liutkus, Stöter, Mimilakis, Bittner, "The MUSDB18 corpus for music separation," 2017. · [norbert](https://github.com/sigsep/norbert) (Liutkus & Stöter) · [open-unmix-pytorch](https://github.com/sigsep/open-unmix-pytorch) · [Demucs](https://github.com/facebookresearch/demucs) · [demucs.onnx](https://github.com/sevagh/demucs.onnx) · [SCNet](https://github.com/starrytong/SCNet) · [cocktail-fork-separation](https://github.com/merlresearch/cocktail-fork-separation) · [TIGER](https://github.com/JusperLee/TIGER) · [pyloudnorm](https://github.com/csteinmetz1/pyloudnorm) · [Music-Source-Separation-Training](https://github.com/ZFTurbo/Music-Source-Separation-Training) · [museval](https://github.com/sigsep/sigsep-mus-eval) · [Spleeter](https://github.com/deezer/spleeter).
 
 **Use when:** you have (or can license) an ONNX-exported spectrogram-mask, Hybrid-Demucs or waveform separation model and want to run it — with proper multichannel Wiener refinement, not just the raw mask — dependency-free, in Node or the browser; `scnet-large` for the highest SDR here under MIT weights, `scnet` for a quarter of its size and a third of its time, `htdemucs` under research-only terms.<br>
 **Not for:** the classical, model-free case — reach for [`@audio/vocals`](https://github.com/audiojs/vocals) when a center-panned M/S trick is all you need; training a model (this is inference-only); real-time streaming (none of Open-Unmix's bi-LSTM, Hybrid Transformer Demucs or SCNet's dual-path LSTMs is causal: offline and chunked only, as upstream).

package/index.d.ts

@@ -62,11 +62,20 @@ export interface ModelPreset {
 	chunk?: number
 	/** the export script that writes its weights */
 	script?: string
+	/** its compact file (scripts/compact.py: int8 weights, a few float16, float32 compute), read before <name>.onnx */
+	file?: string
+	/** the Hugging Face repository hosting it */
+	repo?: string
+	/** the compact file's SHA-256, checked when it is fetched */
+	sha256?: string
 }
 
 /** Model presets by name */
 export const models: Record<ModelName, ModelPreset>
 
+/** The hosted revision (commit) of each compact preset's repository: https://huggingface.co/<repo>/resolve/<revision>/<file> is read without opts.weights in the browser, and in Node when the cache holds neither the file nor its export. Empty: not hosted. */
+export const REVISIONS: Record<'scnet-large' | 'scnet' | 'mrx' | 'tiger', string>
+
 /** A neural-runtime-shaped session — enough of it to drive separate() with a test double. */
 export interface Session {
 	run(feeds: Record<string, { data: Float32Array; dims: number[]; type: string }>): Promise<Record<string, { data: Float32Array; dims: number[] }>>
@@ -84,7 +93,7 @@ export interface SeparateOptions {
 	modelType?: ModelType
 	/** subset of the model's targets to return; spectral models skip the other graphs, and one target runs Wiener EM against the residual */
 	targets?: string[]
-	/** where a preset's files are: URL, or a directory in Node (default $AUDIO_NEURAL_CACHE or ~/.cache/audiojs/neural) */
+	/** where a preset's files are: URL, or a directory in Node (default $AUDIO_NEURAL_CACHE or ~/.cache/audiojs/neural in Node; a compact preset's hosted file, REVISIONS) */
 	weights?: string
 	/** iterations of multichannel Wiener EM refinement (default 1); 0 = raw masks. Ignored for modelType 'hybrid' and 'waveform'. */
 	wiener?: number

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@audio/neural-separate",
-  "version": "0.3.0",
+  "version": "0.4.0",
   "description": "Source separation (stems): a song's by SCNet (MIT weights), Hybrid Transformer Demucs, Open-Unmix with multichannel Wiener refinement; a soundtrack's dialogue, music and effects by MRX (MIT weights) and TIGER (Apache-2.0); model-agnostic ONNX",
   "type": "module",
   "sideEffects": false,
@@ -22,6 +22,7 @@
     "scripts/export-mrx.py",
     "scripts/export-tiger.py",
     "scripts/reference.py",
+    "scripts/compact.py",
     "NOTICE"
   ],
   "dependencies": {

package/scripts/compact.py

@@ -0,0 +1,351 @@
+#!/usr/bin/env python3
+"""Compact an exported separation graph: its weights stored in fewer bits, its compute left in float32.
+
+    pip install onnx onnxruntime numpy
+    python3 compact.py --model scnet-large --verify             # scnet-large.int8.onnx
+    python3 compact.py --model tiger --as fp16 --verify         # tiger.fp16.onnx
+
+Reads <dir>/<model>.onnx (an export-*.py output; <dir> the neural cache's <model>/) and writes <model>.<as>.onnx beside
+it. The weights are the initializers a Conv, ConvTranspose, MatMul, Gemm, LSTM or GRU takes as its weight, of over 1024
+values (biases, norms and constants stay float32):
+  int8   stored as int8 with a float32 scale per output channel (symmetric, the channel's largest magnitude to 127; an
+         LSTM's or GRU's per gate row of each direction), Cast and multiplied by its scale in the graph; all of them when
+         their rounding together costs --budget dB under the output or less (40; 0: no bound) on the calibration mix,
+         else int8 for the weights cheapest per byte saved (each rounded alone, the rest float32) while their summed
+         cost stays within the budget, float16 for the rest (separate weights' costs add: the sum predicts the whole
+         to a fraction of a dB)
+  fp16   stored as float16, Cast to float32 in the graph
+  fp32   as they are
+onnxruntime folds the Cast and the product at load (both of constants): the session holds float32 weights and computes
+as the float32 graph does, on every execution provider. DequantizeLinear would not: onnxruntime 1.30 fuses it with the
+MatMul it feeds into MatMulNBits, whose default accuracy level quantizes the activations to int8 on a CPU (MRX's int8
+graph 38.4 dB from float32 so, 40.4 dB as stored here).
+
+Every variant is also folded and named short, neither of which changes a value. With each input's shape fixed (SCNet's
+and TIGER's graphs; MRX's takes any length and is left as it is), every value computable from the initializers and the
+shapes alone is computed once and stored as the graph computes it (SCNet's DFT matrices, made in float64 from a Range,
+which onnxruntime-web's WebGPU session neither places nor folds), but for a fill (ConstantOfShape: SCNet's LSTM zero
+states), kept with its shape stored. Identical initializers are stored once; node and value names become base-36
+counters (TIGER's 56,652 nodes carried 11.7 MB of names), the graph's inputs and outputs keep theirs.
+
+--verify runs the variant and the float32 graph on the calibration mix (default <dir>/test.f32, scripts/reference.py's;
+--calibrate another: planar float32 stereo at 44.1 kHz), cut and transformed as the package does it, and prints each
+output's largest difference over its peak and its SNR against the float32 graph's (MRX's masks as the spectra they
+mask).
+"""
+import argparse
+import hashlib
+import os
+import sys
+from pathlib import Path
+
+import numpy as np
+
+WEIGHT = {"Conv": (1,), "ConvTranspose": (1,), "MatMul": (1,), "Gemm": (1,), "LSTM": (1, 2), "GRU": (1, 2)}
+MIN, SMALL = 1024, 1024
+
+
+def users_of(g):
+    users = {}
+    for n in g.node:
+        for k, name in enumerate(n.input): users.setdefault(name, []).append((n, k))
+    return users
+
+
+def weights(g):
+    """{name: node} of the float32 initializers of over MIN values that every user takes as its weight."""
+    from onnx import TensorProto
+
+    users, out = users_of(g), {}
+    for i in g.initializer:
+        uses = users.get(i.name, [])
+        if i.data_type != TensorProto.FLOAT or int(np.prod(i.dims)) <= MIN or not uses: continue
+        if all(n.op_type in WEIGHT and k in WEIGHT[n.op_type] for n, k in uses) and len({n.op_type for n, _ in uses}) == 1:
+            out[i.name] = uses[0][0]
+    return out
+
+
+def q8(node, w):
+    """Symmetric int8 per output channel (an LSTM's or GRU's per gate row of each direction), the channel's largest
+    magnitude to 127: q in w's layout and its scale, shaped to broadcast against it."""
+    keep = (0, 1) if node.op_type in ("LSTM", "GRU") else ({"ConvTranspose": 1, "MatMul": w.ndim - 1}.get(node.op_type, 0),)
+    if node.op_type == "Gemm": keep = (0 if any(x.name == "transB" and x.i for x in node.attribute) else 1,)
+    scale = np.maximum(np.abs(w).max(axis=tuple(k for k in range(w.ndim) if k not in keep), keepdims=True), 1e-30) / 127
+    return np.clip(np.round(w / scale), -127, 127).astype(np.int8), scale.astype(np.float32)
+
+
+def rounded(node, w):
+    """w through q8 and back, in float32: what the int8 graph computes with."""
+    q, scale = q8(node, w)
+    return q.astype(np.float32) * scale
+
+
+def prepend(g, nodes):
+    body = list(g.node)
+    del g.node[:]
+    g.node.extend(nodes + body)
+
+
+def store(m, kinds):
+    """Each weight as kinds[name] says: 'fp16' (Cast), 'int8' (Cast, Mul by its scale), else float32."""
+    from onnx import helper, numpy_helper, TensorProto
+
+    g = m.graph
+    inits = {i.name: i for i in g.initializer}
+    nodes = []
+    for name, node in weights(g).items():
+        kind = kinds.get(name)
+        if kind not in ("fp16", "int8"): continue
+        w = numpy_helper.to_array(inits[name])
+        g.initializer.remove(inits[name])
+        if kind == "fp16":
+            g.initializer.append(numpy_helper.from_array(w.astype(np.float16), name + "_h"))
+            nodes.append(helper.make_node("Cast", [name + "_h"], [name], to=TensorProto.FLOAT))
+            continue
+        q, scale = q8(node, w)
+        g.initializer.extend([numpy_helper.from_array(q, name + "_q"), numpy_helper.from_array(scale, name + "_s")])
+        nodes += [helper.make_node("Cast", [name + "_q"], [name + "_c"], to=TensorProto.FLOAT), helper.make_node("Mul", [name + "_c", name + "_s"], [name])]
+    prepend(g, nodes)
+    return m
+
+
+def static(g):
+    return all(d.dim_value > 0 for i in g.input for d in i.type.tensor_type.shape.dim)
+
+
+def fold(m):
+    """Values computable from the initializers and the input shapes alone: stored when small, else made at load."""
+    import onnx
+    import onnxruntime as ort
+    from onnx import helper, numpy_helper, shape_inference
+
+    g = m.graph
+    if not static(g): return dedupe(m)
+    const, cnodes = {i.name for i in g.initializer}, []
+    for n in g.node:
+        ins = [x for x in n.input if x]
+        if n.op_type in ("Constant", "Shape") or (ins and all(x in const for x in ins) and not n.op_type.startswith("Random")):
+            cnodes.append(n)
+            const.update(n.output)
+    computed = [o for n in cnodes for o in n.output]
+    types = {v.name: v.type.tensor_type.elem_type for v in shape_inference.infer_shapes(m).graph.value_info}
+    for n in cnodes:
+        if n.op_type == "Constant": types[n.output[0]] = next(a.t.data_type for a in n.attribute if a.name == "value")
+    probe = onnx.ModelProto()
+    probe.CopyFrom(m)
+    probe.graph.output.extend(helper.make_tensor_value_info(x, types[x], None) for x in computed)
+    s = session(probe.SerializeToString())
+    shapes = {i.name: [d.dim_value for d in i.type.tensor_type.shape.dim] for i in g.input}
+    vals = dict(zip(computed, s.run(computed, {k: np.zeros(v, np.float32) for k, v in shapes.items()})))
+    # the same from another input, or the graph holds a shape that depends on the data: left as it is
+    other = s.run(computed, {k: np.random.default_rng(0).standard_normal(v).astype(np.float32) for k, v in shapes.items()})
+    if any(not np.array_equal(vals[x], y, equal_nan=vals[x].dtype.kind == "f") for x, y in zip(computed, other)): return dedupe(m)
+    # what the rest of the graph reads, stored; a fill (ConstantOfShape) kept, its shape stored
+    producer = {o: n for n in cnodes for o in n.output}
+    inner = {id(n) for n in cnodes}
+    keep, stored = set(), {}
+    want = [x for n in g.node if id(n) not in inner for x in n.input] + [o.name for o in g.output]
+    while want:
+        x = want.pop()
+        if x not in producer or x in stored or id(producer[x]) in keep: continue
+        if producer[x].op_type == "ConstantOfShape" and vals[x].size > SMALL:
+            keep.add(id(producer[x]))
+            want += list(producer[x].input)
+        else: stored[x] = vals[x]
+    body = [n for n in g.node if id(n) not in inner or id(n) in keep]
+    del g.node[:]
+    g.node.extend(body)
+    g.initializer.extend(numpy_helper.from_array(np.asarray(v), x) for x, v in stored.items())
+    return dedupe(m)
+
+
+def dedupe(m):
+    """Identical initializers stored once; those no node reads dropped."""
+    from onnx import numpy_helper
+
+    g = m.graph
+    seen, alias = {}, {}
+    for i in list(g.initializer):
+        a = numpy_helper.to_array(i)
+        key = hashlib.sha256(str((a.dtype, a.shape)).encode() + a.tobytes()).hexdigest()
+        if key in seen:
+            alias[i.name] = seen[key]
+            g.initializer.remove(i)
+        else: seen[key] = i.name
+    for n in g.node:
+        for k, x in enumerate(n.input):
+            if x in alias: n.input[k] = alias[x]
+    read = {x for n in g.node for x in n.input} | {o.name for o in g.output}
+    for i in [i for i in g.initializer if i.name not in read]: g.initializer.remove(i)
+    return m
+
+
+def shorten(m):
+    """Node and value names as base-36 counters; the inputs and outputs keep theirs."""
+    g = m.graph
+    keep = {v.name for v in list(g.input) + list(g.output)}
+    names = {}
+
+    def base36(k):
+        s = ""
+        while True:
+            s, k = "0123456789abcdefghijklmnopqrstuvwxyz"[k % 36] + s, k // 36
+            if not k: return s
+
+    def short(n):
+        if n == "" or n in keep: return n
+        if n not in names: names[n] = "_" + base36(len(names))
+        return names[n]
+
+    for i in g.initializer: i.name = short(i.name)
+    for k, n in enumerate(g.node):
+        for j, x in enumerate(n.input): n.input[j] = short(x)
+        for j, x in enumerate(n.output): n.output[j] = short(x)
+        n.name, n.doc_string = base36(k), ""  # unique: onnxruntime-web's partitioner reports nodes by name
+    del g.value_info[:]
+    m.doc_string = ""
+    return m
+
+
+# ------------------------------------------------------------------ calibration
+
+def session(model, fold=False):
+    import onnxruntime as ort
+
+    o = ort.SessionOptions()
+    o.enable_cpu_mem_arena = False
+    o.enable_mem_pattern = False
+    o.log_severity_level = 3  # the overridable weights' warnings
+    if not fold: o.graph_optimization_level = ort.GraphOptimizationLevel.ORT_DISABLE_ALL
+    return ort.InferenceSession(model if isinstance(model, bytes) else str(model), o, providers=["CPUExecutionProvider"])
+
+
+def feeds(g, mix):
+    """The graph's inputs from a mixture [2, samples] at 44.1 kHz, as separate.js cuts and transforms it: SCNet one
+    11 s segment (no window, 1/sqrt(n)), TIGER one 12 s segment of the first channel (Hann), MRX 10 s of each
+    channel at three resolutions (Hann, 1/sqrt(n), magnitudes)."""
+    ins = {i.name: [d.dim_value or None for d in i.type.tensor_type.shape.dim] for i in g.input}
+
+    def spec(x, n, hop, win, scale):
+        pad = np.pad(x, ((0, 0), (n // 2, n // 2)), mode="reflect")
+        T = 1 + (pad.shape[1] - n) // hop
+        fr = np.stack([pad[:, t * hop:t * hop + n] for t in range(T)], 1) * win
+        return np.fft.rfft(fr, axis=-1).transpose(0, 2, 1) * scale  # [C, F, T]
+
+    hann = lambda n: 0.5 - 0.5 * np.cos(2 * np.pi * np.arange(n) / n)
+    if list(ins) == ["mix_spec"]:
+        _, C2, F, T = ins["mix_spec"]
+        n = 2 * (F - 1)
+        tiger = n == 2048
+        x = mix[:C2 // 2, :(T - 1) * (n // 4)]
+        X = spec(x, n, n // 4, hann(n) if tiger else 1.0, 1.0 if tiger else n ** -0.5)[..., :T]
+        X = np.pad(X, ((0, 0), (0, 0), (0, T - X.shape[2])))
+        return {"mix_spec": np.stack([X.real, X.imag], 1).reshape(1, C2, F, T).astype(np.float32)}
+    x = mix[:, :441000]
+    return {k: np.abs(spec(x, int(k.split("_")[1]), 256, hann(int(k.split("_")[1])), int(k.split("_")[1]) ** -0.5)).astype(np.float32) for k in ins}
+
+
+def outputs(s, feed):
+    """The outputs as the stems see them: a mask times the magnitudes it masks."""
+    ys = dict(zip([o.name for o in s.get_outputs()], s.run(None, feed)))
+    return {k: (y * feed["mag_" + k[5:]][:, None] if k.startswith("mask_") else y).astype(np.float64) for k, y in ys.items()}
+
+
+def noise(y, y0):
+    """Error power over signal power, the worst output's."""
+    return max(((y[k] - y0[k]) ** 2).sum() / max((y0[k] ** 2).sum(), 1e-300) for k in y0)
+
+
+def costs(m, feed, W):
+    """The cost at int8 of all the weights rounded together, and a function giving each one's alone (the others
+    float32): noise power over output power. The weights are fed as overridable initializers, one session for all."""
+    import onnx
+    from onnx import helper, numpy_helper
+
+    inits = {i.name: i for i in m.graph.initializer}
+    probe = onnx.ModelProto()
+    probe.CopyFrom(m)
+    probe.graph.input.extend(helper.make_tensor_value_info(n, inits[n].data_type, list(inits[n].dims)) for n in W)
+    s = session(probe.SerializeToString())
+    y0 = outputs(s, feed)
+    q = {n: rounded(node, numpy_helper.to_array(inits[n])) for n, node in W.items()}
+    return noise(outputs(s, {**feed, **q}), y0), lambda n: noise(outputs(s, {**feed, n: q[n]}), y0)
+
+
+def int8(m, feed, budget):
+    """{weight: 'int8' | 'fp16'}: every weight int8 when their rounding together costs budget dB under the output or
+    less (no budget: always); else int8 for the weights cheapest per byte saved, each rounded alone, while their summed
+    cost stays within the budget (costs of separate weights add), float16 for the rest."""
+    W = weights(m.graph)
+    if budget is None: return {n: "int8" for n in W}
+    whole, alone = costs(m, feed, W)
+    print(f"  every weight int8: SNR {-10 * np.log10(whole):.1f} dB against float32")
+    if whole <= 10 ** (-budget / 10): return {n: "int8" for n in W}
+    size = {i.name: int(np.prod(i.dims)) for i in m.graph.initializer}
+    cost = {}
+    for k, n in enumerate(W):
+        cost[n] = alone(n)
+        print(f"\r  cost {k + 1}/{len(W)}", end="", file=sys.stderr)
+    print(file=sys.stderr)
+    kinds, total = {}, 0.0
+    for n in sorted(W, key=lambda n: cost[n] / size[n]):
+        if total + cost[n] <= 10 ** (-budget / 10): kinds[n], total = "int8", total + cost[n]
+        else: kinds[n] = "fp16"
+    for n in sorted((n for n in kinds if kinds[n] == "fp16"), key=lambda n: -cost[n]):
+        print(f"  float16: {W[n].name or n}, {size[n]} values, alone at int8 {-10 * np.log10(max(cost[n], 1e-30)):.1f} dB")
+    print(f"  int8: {sum(k == 'int8' for k in kinds.values())} of {len(kinds)} weights, "
+          f"{sum(size[n] for n in kinds if kinds[n] == 'int8') / sum(size[n] for n in W):.1%} of their values; predicted SNR {-10 * np.log10(max(total, 1e-30)):.1f} dB")
+    return kinds
+
+
+def compact(path, out, kind, calibration=None, budget=40.0):
+    import onnx
+
+    m = onnx.load(str(path))
+    # the trained weights, chosen before folding (never a folded constant)
+    kinds = int8(m, feeds(m.graph, calibration), budget) if kind == "int8" else {n: kind for n in weights(m.graph)}
+    m = shorten(store(fold(m), kinds))
+    m.metadata_props.append(onnx.StringStringEntryProto(key="compact", value=kind + (f", budget {budget:g} dB" if kind == "int8" and budget else "")))
+    onnx.checker.check_model(m)
+    onnx.save(m, str(out))
+    return out
+
+
+def verify(path, base, mix):
+    import onnx
+
+    a, b = session(base, True), session(path, True)
+    feed = feeds(onnx.load(str(base), load_external_data=False).graph, mix)
+    y0, y = outputs(a, feed), outputs(b, feed)
+    for k in y0:
+        d = y[k] - y0[k]
+        rel = np.abs(d).max() / np.abs(y0[k]).max()
+        print(f"  {path.name} {k}: max|diff|/max|y| = {rel:.2e}, SNR {10 * np.log10((y0[k] ** 2).sum() / max((d ** 2).sum(), 1e-300)):.1f} dB against float32")
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--model", required=True, help="scnet-large | scnet | mrx | tiger (any <dir>/<model>.onnx)")
+    ap.add_argument("--as", dest="kind", default="int8", choices=["fp32", "fp16", "int8"])
+    ap.add_argument("--budget", type=float, default=40, help="int8: the rounding's cost, dB under the output (40); 0: every weight int8")
+    ap.add_argument("--dir", default=None, help="default: $AUDIO_NEURAL_CACHE/<model> or ~/.cache/audiojs/neural/<model>")
+    ap.add_argument("--out", default=None, help="default: <dir>/<model>.<as>.onnx")
+    ap.add_argument("--calibrate", default=None, help="planar float32 stereo at 44.1 kHz (default <dir>/test.f32)")
+    ap.add_argument("--verify", action="store_true", help="the variant against the float32 graph on the calibration mix")
+    args = ap.parse_args()
+
+    cache = Path(os.environ.get("AUDIO_NEURAL_CACHE") or Path.home() / ".cache" / "audiojs" / "neural")
+    d = Path(args.dir) if args.dir else cache / args.model
+    path = d / f"{args.model}.onnx"
+    out = Path(args.out) if args.out else d / f"{args.model}.{args.kind}.onnx"
+    cal = Path(args.calibrate) if args.calibrate else d / "test.f32"
+    mix = np.fromfile(cal, np.float32).reshape(2, -1) if cal.exists() else None
+    budget = args.budget or None
+    if mix is None and ((args.kind == "int8" and budget) or args.verify): raise SystemExit(f"{cal}: no calibration mix (export with --verify writes test.f32)")
+    compact(path, out, args.kind, mix, budget)
+    print(f"wrote {out} ({out.stat().st_size / 1e6:.2f} MB, from {path.stat().st_size / 1e6:.2f})")
+    if args.verify: verify(out, path, mix)
+
+
+if __name__ == "__main__":
+    sys.exit(main())

package/scripts/export-scnet.py

@@ -25,7 +25,7 @@ The STFT is SCNet's own: n_fft 4096, hop 1024, no window (torch.stft's default,
 ones), normalized (1/sqrt(4096)), centered with reflect padding.
 
 Writes <out-dir>/<model>.onnx (scnet-large 169 MB, scnet 43 MB) and, with --fp16,
-<model>.fp16.onnx (half that), weights embedded. Default out-dir:
+<model>.fp16.onnx (half that: float16 weights, float32 compute, scripts/compact.py's), weights embedded. Default out-dir:
 $AUDIO_NEURAL_CACHE/<model>, else ~/.cache/audiojs/neural/<model>. --verify compares the stems rebuilt from the ONNX
 output with SCNet.forward on the same segment, and writes the reference
 separation test.js compares the JS pipeline against (scripts/reference.py),
@@ -187,11 +187,11 @@ def export(model, path, opset):
 
 
 def fp16(path, out):
-    """float16 weights, float32 inputs and outputs (onnxruntime's converter, as export-openunmix.py's)."""
-    import onnx
-    from onnxruntime.transformers.float16 import convert_float_to_float16
+    """float16 weights, float32 compute (scripts/compact.py --as fp16)."""
+    sys.path.insert(0, str(Path(__file__).parent))
+    import compact
 
-    onnx.save(convert_float_to_float16(onnx.load(str(path)), keep_io_types=True), str(out))
+    compact.compact(path, out, "fp16")
     print(f"wrote {out} ({out.stat().st_size / 1e6:.1f} MB)")
 
 
@@ -245,7 +245,7 @@ def main():
     ap.add_argument("--model", default="scnet-large", choices=list(MODELS))
     ap.add_argument("--out-dir", default=None, help="default: $AUDIO_NEURAL_CACHE/<model> or ~/.cache/audiojs/neural/<model>")
     ap.add_argument("--opset", type=int, default=17)
-    ap.add_argument("--fp16", action="store_true", help="also write <model>.fp16.onnx (float16 weights, float32 I/O)")
+    ap.add_argument("--fp16", action="store_true", help="also write <model>.fp16.onnx (float16 weights, float32 compute)")
     ap.add_argument("--verify", action="store_true", help="compare with SCNet.forward; write the reference separation")
     args = ap.parse_args()
 

package/separate.js

@@ -12,7 +12,7 @@
 //   of sevagh/demucs.onnx. 7.8 s segments overlap-added as demucs.apply.apply_model does; each
 //   runs as waveform + complex-as-channels spectrogram in, frequency- and time-branch estimates
 //   out, summed after the iSTFT.
-// 'complex': complex spectrogram models (SCNet: Tong, Zhang, Liu, Li, Yu, "SCNet: Sparse Compression
+// 'complex': complex spectrogram models (SCNet: Tong, Zhu, Chen, Kang, Jiang, Li, Wu, Meng, "SCNet: Sparse Compression
 //   Network for Music Source Separation", ICASSP 2024; the Band-Split RoFormer family has the same
 //   contract) with their STFT and iSTFT out of the graph: each channel's STFT in, re and im as
 //   channels, each source's out. 11 s segments overlap-added under linear fades, as
@@ -27,7 +27,7 @@
 
 import { fft, ifft } from 'fourier-transform'
 import resampleSinc from '@audio/resample-sinc'
-import { load as neuralLoad, tensor } from '@audio/neural-runtime'
+import { load as neuralLoad, tensor, fetchModel } from '@audio/neural-runtime'
 
 const PI2 = Math.PI * 2
 const isNode = typeof process !== 'undefined' && !!process.versions?.node
@@ -371,20 +371,41 @@ const TIGER = {
 	standardize: false, mono: true, script: 'export-tiger.py',
 }
 
-// Presets: files are <weights>/<name>/<target>.onnx (one graph per target) or
-// <weights>/<name>/<name>.onnx, as scripts/export-*.py write them.
+// The compact files (scripts/compact.py: int8 weights, a few float16, float32 compute), each within 0.05 dB of its
+// export's median SDR or SNR per stem (README, Size); their Hugging Face repositories, and SHA-256 checked when fetched
+const FILES = {
+	'scnet-large': { file: 'scnet-large.int8.onnx', repo: 'audiojs/scnet-large', sha256: 'b2dc586a1e0e6c0afe4915e9057ea29111397b7bbf589edc3d85de91de79cd72' },
+	scnet: { file: 'scnet.int8.onnx', repo: 'audiojs/scnet', sha256: '98228931494151762a1c4ab1ec7899a894b1f81fd4a509921a9d2f9bacc50845' },
+	mrx: { file: 'mrx.int8.onnx', repo: 'audiojs/mrx', sha256: 'd876c92d224f5d8cd2207d528b23369da895a6de71fb24132416fcb16f9f8424' },
+	tiger: { file: 'tiger.int8.onnx', repo: 'audiojs/tiger-dnr', sha256: '2189859ac7e89cf4d9d284c6f393e737aaa2b3233ed7ee424ad622012bce7bca' },
+}
+
+// The hosted revision of each compact file's repository (a commit), its URL https://huggingface.co/<repo>/resolve/<revision>/
+// <file>: what a compact preset reads without opts.weights, in the browser, and in Node fetched once into the neural cache
+// (unless the cache holds it or the export). Empty: not hosted.
+export const REVISIONS = {
+	'scnet-large': '95f22d9e68f6cab78aef358f1a0de248d4edf805',
+	scnet: 'e71bcdb42bce1d62b7fae25ba90fcbcf90f73f46',
+	mrx: 'dedc823a0bea2ed085d19e8b7f32589d35169ee8',
+	tiger: '1c959ab7ff8794dc5663b104ba301687ab95d2ab',
+}
+
+const hosted = name => models[name].repo && REVISIONS[name] ? `https://huggingface.co/${models[name].repo}/resolve/${REVISIONS[name]}/${models[name].file}` : null
+
+// Presets: files are <weights>/<name>/<target>.onnx (one graph per target) or <weights>/<name>/<name>.onnx, as
+// scripts/export-*.py write them; a compact preset's <weights>/<name>/<file> first, as scripts/compact.py writes it.
 export const models = {
 	umxhq: { modelType: 'openunmix', sampleRate: 44100, targets: ['vocals', 'drums', 'bass', 'other'], perTarget: true },
 	htdemucs: { modelType: 'hybrid', sampleRate: 44100, targets: DEMUCS },
 	htdemucs_ft: { modelType: 'hybrid', sampleRate: 44100, targets: DEMUCS, perTarget: true },
 	// SCNet-large and SCNet (starrytong/SCNet, MIT weights) as scripts/export-scnet.py exports them
-	'scnet-large': SCNET,
-	scnet: SCNET,
+	'scnet-large': { ...SCNET, ...FILES['scnet-large'] },
+	scnet: { ...SCNET, ...FILES.scnet },
 	// MRX (merlresearch/cocktail-fork-separation, MIT weights) as scripts/export-mrx.py exports it: dialogue, music
 	// and effects of a soundtrack (Divide and Remaster)
-	mrx: MRX,
+	mrx: { ...MRX, ...FILES.mrx },
 	// TIGER (JusperLee/TIGER, Apache-2.0 weights): the same three stems, three band-split models, about 50 times MRX's time
-	tiger: TIGER,
+	tiger: { ...TIGER, ...FILES.tiger },
 }
 
 async function cacheDir() {
@@ -405,18 +426,40 @@ async function weightsBase(opts, name) {
 	return pathToFileURL(w.endsWith('/') ? w : w + '/').href
 }
 
-// Weights produced locally must exist before a session loads them: name the missing file and
-// how to make it, rather than surfacing ENOENT from deep inside the runtime.
-async function checkLocal(graphs, name) {
-	let files = graphs.map(g => g.spec).filter(s => typeof s === 'string' && s.startsWith('file:'))
-	if (!files.length) return
-	let [{ existsSync }, { fileURLToPath }] = await Promise.all([import('node:fs'), import('node:url')])
-	let missing = files.map(f => fileURLToPath(f)).filter(f => !existsSync(f))
-	if (!missing.length) return
-	let script = models[name].script ?? { hybrid: 'export-htdemucs.py', complex: 'export-scnet.py' }[models[name].modelType] ?? 'export-openunmix.py'
-	throw new Error(`neural-separate: ${name} weights not found: ${missing.join(', ')}. ` +
-		`Produce them with python3 node_modules/@audio/neural-separate/scripts/${script} --model ${name}, ` +
-		`or pass opts.weights: a URL or directory holding ${name}/<file>.onnx`)
+// A preset's files in a local directory; a file missing named with how to make it, rather than ENOENT from the runtime.
+// A compact preset reads its compact file, else its export (float32). From the neural cache (no opts.weights) its compact
+// file only as hosted (its SHA-256), else the export, else the hosted file fetched into the cache, checked, written whole;
+// offline, a compact file of another SHA-256 (scripts/compact.py's own) as it is.
+async function local(graphs, name, opts) {
+	let [fs, { fileURLToPath }] = await Promise.all([import('node:fs'), import('node:url')])
+	let file = g => typeof g.spec === 'string' && g.spec.startsWith('file:') ? fileURLToPath(g.spec) : null
+	let here = g => !file(g) || fs.existsSync(file(g)), p = models[name], why = ''
+	let whole = g => ({ ...g, spec: g.spec.replace(/[^/]*$/, `${name}.onnx`), sha256: null })
+	if (p.file) graphs = await Promise.all(graphs.map(async g => {
+		if (!file(g)) return g
+		let cached = opts.weights == null
+		if (here(g) && (!cached || await sha256(fs.readFileSync(file(g))) === p.sha256)) return { ...g, sha256: null }
+		if (here(whole(g))) return whole(g)
+		if (cached && hosted(name)) try { return { ...g, spec: await fetched(name, file(g)), sha256: null } } catch (e) { why = ` (${e.message})` }
+		return { ...g, sha256: null }
+	}))
+	let missing = graphs.filter(g => !here(g)).map(file)
+	if (!missing.length) return graphs
+	let script = p.script ?? { hybrid: 'export-htdemucs.py', complex: 'export-scnet.py' }[p.modelType] ?? 'export-openunmix.py'
+	throw new Error(`neural-separate: ${name} weights not found: ${missing.join(', ')}${why}. ` +
+		`Produce them with python3 node_modules/@audio/neural-separate/scripts/${script} --model ${name}` +
+		(p.file ? ` (and scripts/compact.py --model ${name} for ${p.file})` : '') +
+		`, or pass opts.weights: a URL or directory holding ${name}/<file>.onnx`)
+}
+
+// The hosted compact file into the cache, checked against its SHA-256, written whole or not at all → its file URL
+async function fetched(name, dest) {
+	let [fs, path, { pathToFileURL }] = await Promise.all([import('node:fs'), import('node:path'), import('node:url')])
+	let bytes = await checked(hosted(name), models[name].sha256, { cache: false })
+	fs.mkdirSync(path.dirname(dest), { recursive: true })
+	fs.writeFileSync(dest + '.part', bytes)
+	fs.renameSync(dest + '.part', dest)
+	return pathToFileURL(dest).href
 }
 
 // opts.model → { graphs: [{ spec, sources, keep }], modelType, sampleRate, targets }. A graph's
@@ -429,14 +472,15 @@ async function resolveModel(opts) {
 	let pick = names => want ? names.filter(n => want.includes(n)) : names
 
 	if (typeof model === 'string' && Object.hasOwn(models, model)) {
-		let p = models[model], base = await weightsBase(opts, model)
+		// the browser without opts.weights: the hosted file
+		let p = models[model], url = !isNode && opts.weights == null && hosted(model), base = url ? '' : await weightsBase(opts, model)
 		if (want) for (let t of want) if (!p.targets.includes(t)) throw new Error(`neural-separate: ${model} has no target '${t}' (has ${p.targets.join(', ')})`)
 		let targets = pick(p.targets)
 		let sources = p.sources ?? (p.modelType === 'hybrid' || p.modelType === 'complex' ? DEMUCS : null)
 		let graphs = p.perTarget
 			? targets.map(t => ({ spec: `${base}${model}/${t}.onnx`, sources: sources ?? [t], keep: [t] }))
-			: [{ spec: `${base}${model}/${model}.onnx`, sources: sources ?? p.targets, keep: targets }]
-		if (isNode) await checkLocal(graphs, model)
+			: [{ spec: url || `${base}${model}/${p.file ?? `${model}.onnx`}`, sources: sources ?? p.targets, keep: targets, sha256: p.sha256 }]
+		if (isNode) graphs = await local(graphs, model, opts)
 		return { graphs, modelType: p.modelType, sampleRate: p.sampleRate, targets, preset: p }
 	}
 
@@ -459,8 +503,25 @@ async function resolveModel(opts) {
 // No pooled arena: one htdemucs segment peaks at 2.7 GB RSS in onnxruntime-node instead of 3.2 GB.
 const sessionOptions = { enableCpuMemArena: false, enableMemPattern: false }
 
+// A graph's session; a fetched file with a SHA-256 checked against it first
 function loader(opts) {
-	return spec => opts.session ? opts.session(spec, opts) : neuralLoad(spec, { backend: opts.device, sessionOptions })
+	return async g => {
+		if (opts.session) return opts.session(g.spec, opts)
+		let spec = g.sha256 && !g.spec.startsWith('file:') ? await checked(g.spec, g.sha256) : g.spec
+		return neuralLoad(spec, { backend: opts.device, sessionOptions })
+	}
+}
+
+// A fetched file's bytes, refused unless of its SHA-256
+async function checked(url, hash, opts) {
+	let bytes = await fetchModel(url, opts), hex = await sha256(bytes)
+	if (hex !== hash) throw new Error(`neural-separate: ${url} has SHA-256 ${hex}, expected ${hash}: a partial or altered download (clear the 'audio-neural' cache), or another build of the file`)
+	return bytes
+}
+
+async function sha256(bytes) {
+	let subtle = globalThis.crypto?.subtle ?? (await import('node:crypto')).webcrypto.subtle
+	return Array.from(new Uint8Array(await subtle.digest('SHA-256', bytes)), b => b.toString(16).padStart(2, '0')).join('')
 }
 
 function outputOf(session, out, i = 0) {
@@ -549,7 +610,7 @@ async function separateSpectral(channels, rate, opts, graphs, modelType, load) {
 	let wopts = { iterations, softmask: opts.softmask ?? false, eps: opts.eps, residual }
 	let sessions = []
 	try {
-		for (let g of graphs) sessions.push(await load(g.spec))
+		for (let g of graphs) sessions.push(await load(g))
 		let C = channels.length, N = channels[0].length
 		let { starts, size, weight } = crossfadeChunks(N, rate, opts)
 		return await overlapAdd(N, C, starts, size, weight, async (start, len) => {
@@ -643,7 +704,7 @@ async function separateHybrid(channels, rate, opts, graphs, load) {
 
 	let result = {}
 	for (let g of graphs) {
-		let session = await load(g.spec)
+		let session = await load(g)
 		try {
 			let L = session.inputs?.[0]?.dims?.[2]
 			L = typeof L === 'number' && L > 0 ? L : opts.segment ?? 343980 // htdemucs: int(7.8 · 44100)
@@ -718,7 +779,7 @@ async function separateComplex(channels, rate, opts, graphs, load, p) {
 	let { norm, scale: back } = opts.standardize ?? p?.standardize ?? true ? normalize(channels) : { norm: channels, scale: c => c }
 	let result = {}
 	for (let g of graphs) {
-		let session = await load(g.spec)
+		let session = await load(g)
 		try {
 			let T = session.inputs?.[0]?.dims?.[3]
 			T = typeof T === 'number' && T > 0 ? T : opts.frames ?? p?.frames
@@ -764,7 +825,7 @@ async function separateMultires(channels, rate, opts, graphs, load, p) {
 	let windows = opts.windows ?? p?.windows, hop = opts.hop ?? p?.hop ?? 256
 	let C = channels.length, N = channels[0].length, result = {}
 	for (let g of graphs) {
-		let session = await load(g.spec)
+		let session = await load(g)
 		try {
 			let { starts, size, weight } = crossfadeChunks(N, rate, { ...opts, chunk: opts.chunk ?? p?.chunk })
 			Object.assign(result, await overlapAdd(N, C, starts, size, weight, async (start, len) => {
@@ -847,7 +908,7 @@ async function separateWaveform(channels, rate, opts, graphs, load) {
 	let { starts, size, weight } = crossfadeChunks(N, rate, opts)
 	let result = {}
 	for (let g of graphs) {
-		let session = await load(g.spec)
+		let session = await load(g)
 		try {
 			let out = await overlapAdd(N, C, starts, size, weight, async (start, len) => {
 				let data = new Float32Array(C * len)