@audio/neural-separate 0.5.0 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +10 -0
- package/index.d.ts +3 -0
- package/package.json +2 -2
- package/scripts/export-roformer.py +70 -32
- package/separate.js +17 -2
package/README.md
CHANGED
|
@@ -63,6 +63,14 @@ together (0.9 GB of it the browser idle): wasm `scnet` 3.0 GB, `scnet-large` 4.6
|
|
|
63
63
|
on wasm 9 and 0.3 minutes. Float16 compute does not pay: `tiger`'s on WebGPU is 13 % faster and its music 4.7 dB from
|
|
64
64
|
the float32 graph's.
|
|
65
65
|
|
|
66
|
+
A voice's room (de-reverb):
|
|
67
|
+
|
|
68
|
+
| `model` | Architecture | Files | Weights | Targets | Real-time factor: Node CPU · WebGPU |
|
|
69
|
+
|---|---|---|---|---|---|
|
|
70
|
+
| `'dereverb-room'` | anvuew's room de-reverb: a mono BS-RoFormer (axial transformers over time and 63 contiguous bands of the complex spectrogram, dim 128, depth 16, 29.4 M parameters), trained on voices in rooms simulated with pyroomacoustics | 60 MB, float16 weights (export 122 MB) | GPL-3.0 | `dry`: the voice with its room off, early reflections and tail; the residual is the room | 2.3 · 0.65 (Node's WebGPU) |
|
|
71
|
+
|
|
72
|
+
Each channel heard apart, 8.7 s segments every 4.35 s as `demix()` runs it (`num_overlap: 2`), nothing normalized. Measured through [`audio`](https://github.com/audiojs/audio)'s `dereverb()` (`bench/rx/dereverb.mjs`, test): VoiceBank test utterances in 19 MIT IR Survey rooms (Traer & McDermott, PNAS 2016; T60 0.38–1.85 s), each response split at direct + 50 ms, PESQ against the dry take 1.39 → 2.65, STOI 0.77 → 0.94, DNSMOS OVRL 2.19 → 2.98, the late tail 15.6 dB down; iZotope RX 12 De-reverb tuned 1.45, its Dialogue Isolate with the reverb off 1.79. It takes the early reflections too: the output stands 5.2 dB under the direct sound and its first 50 ms, where the voice is 10 dB over the tail. Music is room to it (MUSDB18 previews, GuitarSet, sung tones and four pieces lose 6 dB in their worst octave as they are, and 9 dB of themselves in a room); `audio` passes music by a speech/music guard. The real-time factors are of 20 s of stereo on a busy 14-core M4 Max (load averages over 100), so upper bounds; the float16 file stands 74 dB from the PyTorch forward through this package's pipeline, the float32 export 123 dB.
|
|
73
|
+
|
|
66
74
|
`targets` picks a subset: `targets: ['vocals']` runs only the vocals graph of `umxhq` (against the residual, see Algorithm: 2.4× faster, vocals SDR 6.50 instead of 6.75) and of `htdemucs_ft`, and skips the other sources' iSTFT for `htdemucs`. Presets resample to their 44.1 kHz and back.
|
|
67
75
|
|
|
68
76
|
## Weights
|
|
@@ -269,6 +277,7 @@ let vocals = complex.vocals.map(ch => istft(ch, { length: left.length }))
|
|
|
269
277
|
| `window` / `normalized` / `frames` | the preset's | `'complex'`: the model's STFT window (`'ones'`, torch.stft's window=None, or `'hann'`), its 1/√n scaling, its frames per segment when the graph does not declare them |
|
|
270
278
|
| `targetRate` | the preset's rate, else the input rate | resample to the model's rate for inference; stems are resampled back to the input rate |
|
|
271
279
|
| `device` | — | passed through to `@audio/neural-runtime`'s `load()` as `backend` |
|
|
280
|
+
| `wasmPaths` | — | where onnxruntime-web's `.wasm` files are served, passed through to `load()`: needed where ORT itself comes from a CDN that rebuilds it (esm.sh), e.g. `'https://cdn.jsdelivr.net/npm/onnxruntime-web@1.30.0/dist/'` |
|
|
272
281
|
| `dtype` | `'float32'` | only `'float32'` tensor marshalling is implemented; anything else throws |
|
|
273
282
|
| `progress` | — | `({ chunk, totalChunks }) => {}` |
|
|
274
283
|
| `session` | — | overrides `@audio/neural-runtime`'s `load()` — for tests, or a custom ORT setup |
|
|
@@ -287,6 +296,7 @@ Open-source stem separation has these lineages; this package runs SCNet, Open-Un
|
|
|
287
296
|
| **SCNet** (`scnet`, `scnet-large`; starrytong) | MIT | MIT: "I confirm that the released SCNet and SCNet-large pretrained weights are distributed under the MIT License, consistent with the source code. You are welcome to redistribute the original checkpoints and format-converted versions, including ONNX exports, as part of your MIT-licensed tool, with appropriate attribution." (the author in [starrytong/SCNet#35](https://github.com/starrytong/SCNet/issues/35), 2026) | trained on MUSDB18-HQ; hosted with its configs by [ZFTurbo/Music-Source-Separation-Training](https://github.com/ZFTurbo/Music-Source-Separation-Training) (MIT), whose own MUSDB18-HQ checkpoints (SCNet XL, BS-RoFormer) carry its author's MIT statement only for some files ([#254](https://github.com/ZFTurbo/Music-Source-Separation-Training/issues/254)) |
|
|
288
297
|
| **MRX** (MERL; `mrx`) | MIT | MIT: the repository's [`.reuse/dep5`](https://github.com/merlresearch/cocktail-fork-separation/blob/main/.reuse/dep5) names the four checkpoints, "Copyright: 2023 Mitsubishi Electric Research Laboratories (MERL) / License: MIT" | trained on Divide and Remaster v2, whose music and effects come from FMA and FSD50K clips each under its own licence, some non-commercial (the DnR v3 paper's account) |
|
|
289
298
|
| **TIGER** (Tsinghua; `tiger`) | MIT (repository LICENSE; its README badge says Apache 2.0) | Apache 2.0: the [model card](https://huggingface.co/JusperLee/TIGER-DnR) states `license: apache-2.0` | trained on Divide and Remaster v1 (652 test mixtures in its paper), sourced as v2 is |
|
|
299
|
+
| **anvuew's de-reverb** (`dereverb-room`) | the architecture: BS-RoFormer (Phil Wang's [BS-RoFormer](https://github.com/lucidrains/BS-RoFormer), MIT, in ZFTurbo's Music-Source-Separation-Training) | **GPL-3.0** ([model card](https://huggingface.co/anvuew/dereverb_room)): the converted copy is hosted under the same licence, fetched at run time, never bundled | trained on convolutional reverbs from pyroomacoustics, the dry voices not named |
|
|
290
300
|
| **Bandit v2** (DnR v3) | Apache 2.0 | **CC BY-SA 4.0** ([Zenodo](https://zenodo.org/records/12701995)): a converted copy is an adaptation under the same terms | not shipped; BandIt v1's DnR checkpoints are CC BY-NC 4.0 ([Zenodo](https://zenodo.org/records/10160698)) |
|
|
291
301
|
| **Spleeter** (Deezer) | MIT | **undocumented** | the README licenses only "the code of Spleeter"; the pretrained weights' license is an open, unresolved question ([deezer/spleeter#898](https://github.com/deezer/spleeter/issues/898)) — do not assume MIT |
|
|
292
302
|
|
package/index.d.ts
CHANGED
|
@@ -131,6 +131,9 @@ export interface SeparateOptions {
|
|
|
131
131
|
hop?: number
|
|
132
132
|
/** passed through to @audio/neural-runtime's load() as `backend` */
|
|
133
133
|
device?: string
|
|
134
|
+
/** where onnxruntime-web's .wasm files are served (a URL prefix, or a file → URL map), passed through to
|
|
135
|
+
* @audio/neural-runtime's load(): for an ORT build loaded from a CDN that rebuilds it (esm.sh) */
|
|
136
|
+
wasmPaths?: string | Record<string, string>
|
|
134
137
|
/** only 'float32' is implemented; anything else throws */
|
|
135
138
|
dtype?: 'float32'
|
|
136
139
|
progress?: (p: { chunk: number; totalChunks: number }) => void
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@audio/neural-separate",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.7.0",
|
|
4
4
|
"description": "Source separation (stems): a song's by SCNet (MIT weights), Hybrid Transformer Demucs, Open-Unmix with multichannel Wiener refinement; a soundtrack's dialogue, music and effects by MRX (MIT weights) and TIGER (Apache-2.0); model-agnostic ONNX",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"sideEffects": false,
|
|
@@ -29,7 +29,7 @@
|
|
|
29
29
|
"NOTICE"
|
|
30
30
|
],
|
|
31
31
|
"dependencies": {
|
|
32
|
-
"@audio/neural-runtime": "^0.
|
|
32
|
+
"@audio/neural-runtime": "^0.2.0",
|
|
33
33
|
"@audio/resample-sinc": "^1.1.2",
|
|
34
34
|
"fourier-transform": "^2.4.1"
|
|
35
35
|
},
|
|
@@ -1,14 +1,22 @@
|
|
|
1
1
|
#!/usr/bin/env python3
|
|
2
2
|
"""Export Mel-Band RoFormer (Wang, Lu, Won, Chen, Song, "Mel-Band RoFormer for Music Source Separation", ISMIR LBD
|
|
3
|
-
2023)
|
|
3
|
+
2023) and Band-Split RoFormer (Lu, Wang, Kong, Hung, "Music Source Separation with Band-Split RoPE Transformer", ICASSP
|
|
4
|
+
2024) to the ONNX contract @audio/neural-separate expects for modelType 'complex': the network between its STFT and
|
|
4
5
|
iSTFT, which JS computes, written for the GPU.
|
|
5
6
|
|
|
6
7
|
pip install torch onnx onnxruntime pyyaml einops beartype rotary-embedding-torch librosa
|
|
7
8
|
python3 export-roformer.py --model mel-roformer --verify
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
9
|
+
python3 export-roformer.py --model dereverb-room --verify --fp16
|
|
10
|
+
|
|
11
|
+
The models, each run by ZFTurbo/Music-Source-Separation-Training's (MIT) models/bs_roformer code at a pinned commit:
|
|
12
|
+
mel-roformer Kimberley Jensen's vocals Mel-Band RoFormer (huggingface.co/KimberleyJSN/melbandroformer, MIT), with
|
|
13
|
+
the config that repository runs it by (configs/KimberleyJensen/config_vocals_mel_band_roformer_kj.yaml)
|
|
14
|
+
dereverb-room anvuew's room de-reverb, a mono BS-RoFormer (huggingface.co/anvuew/dereverb_room, GPL-3.0): a voice
|
|
15
|
+
in a room in, the voice dry out; trained on voices in rooms pyroomacoustics simulated (its card); its
|
|
16
|
+
own config, dim 128, depth 16, 63 bands, 29.4 M parameters
|
|
17
|
+
Each checkpoint and config at a pinned revision, the checkpoint checked by its SHA-256. What follows is told of
|
|
18
|
+
mel-roformer; dereverb-room is one channel, 8.7 s segments (384000 samples, hop 512, 751 frames), its bands contiguous
|
|
19
|
+
(each frequency in one band, so its masks are taken as they come) and one RMSNorm after its transformers, not one each.
|
|
12
20
|
|
|
13
21
|
One 8 s segment (352800 samples at 44.1 kHz, 801 frames) per run:
|
|
14
22
|
input mix_spec [1, 4, 1025, 801] STFT, complex as channels (L re, L im, R re, R im)
|
|
@@ -45,12 +53,16 @@ import urllib.request
|
|
|
45
53
|
from pathlib import Path
|
|
46
54
|
|
|
47
55
|
REPO = "https://raw.githubusercontent.com/ZFTurbo/Music-Source-Separation-Training/84b1eac0887756b4f1a9d7a1ff49105939749ed2/"
|
|
48
|
-
MODELS = { # checkpoint URL, its SHA-256, config path in REPO
|
|
56
|
+
MODELS = { # checkpoint URL, its SHA-256, config (a path in REPO or a URL), class, segment (samples), the stem's name
|
|
49
57
|
"mel-roformer": ("https://huggingface.co/KimberleyJSN/melbandroformer/resolve/ac9b0614ab3cd7f77219e18ba494dfd93956c348/MelBandRoformer.ckpt",
|
|
50
58
|
"87201f4d31afb5bc79993230fc49446918425574db48c01c405e44f365c7559e",
|
|
51
|
-
"configs/KimberleyJensen/config_vocals_mel_band_roformer_kj.yaml"),
|
|
59
|
+
"configs/KimberleyJensen/config_vocals_mel_band_roformer_kj.yaml", "MelBandRoformer", 352800, "vocals"),
|
|
60
|
+
"dereverb-room": ("https://huggingface.co/anvuew/dereverb_room/resolve/0b85f5b80b7f779b2dfe80f33a1b35b38af9376d/dereverb_room_anvuew_sdr_13.7432.ckpt",
|
|
61
|
+
"2edec521f09e26341c1923dc82c8c52dbc86478b42b9999f679535743c970cb3",
|
|
62
|
+
"https://huggingface.co/anvuew/dereverb_room/resolve/0b85f5b80b7f779b2dfe80f33a1b35b38af9376d/dereverb_room_anvuew.yaml",
|
|
63
|
+
"BSRoformer", 384000, "dry"),
|
|
52
64
|
}
|
|
53
|
-
SEGMENT = 352800
|
|
65
|
+
SEGMENT = 352800 # the model's, set by main()
|
|
54
66
|
STEP = 0.5
|
|
55
67
|
|
|
56
68
|
|
|
@@ -63,7 +75,7 @@ def fetch(url, path):
|
|
|
63
75
|
|
|
64
76
|
|
|
65
77
|
def load(src, name="mel-roformer"):
|
|
66
|
-
"""
|
|
78
|
+
"""The model with its weights: the model code, config and checkpoint fetched into `src` once."""
|
|
67
79
|
import hashlib
|
|
68
80
|
import torch
|
|
69
81
|
import yaml
|
|
@@ -72,16 +84,17 @@ def load(src, name="mel-roformer"):
|
|
|
72
84
|
pkg.mkdir(parents=True, exist_ok=True)
|
|
73
85
|
(pkg.parent / "__init__.py").touch()
|
|
74
86
|
(pkg / "__init__.py").touch()
|
|
75
|
-
for f in ("mel_band_roformer.py", "attend.py"):
|
|
87
|
+
for f in ("mel_band_roformer.py", "bs_roformer.py", "attend.py"):
|
|
76
88
|
fetch(REPO + "models/bs_roformer/" + f, pkg / f)
|
|
77
89
|
sys.path.insert(0, str(src / "roformer_code"))
|
|
78
90
|
sys.dont_write_bytecode = True
|
|
79
91
|
from models.bs_roformer.mel_band_roformer import MelBandRoformer
|
|
92
|
+
from models.bs_roformer.bs_roformer import BSRoformer
|
|
80
93
|
|
|
81
94
|
class Loader(yaml.SafeLoader): pass # the config's resolutions are a !!python/tuple
|
|
82
95
|
Loader.add_constructor("tag:yaml.org,2002:python/tuple", lambda l, n: tuple(l.construct_sequence(n)))
|
|
83
|
-
url, sha, config = MODELS[name]
|
|
84
|
-
cfg = yaml.load(fetch(REPO + config, src / Path(config).name).read_text(), Loader=Loader)
|
|
96
|
+
url, sha, config, cls, _, _ = MODELS[name]
|
|
97
|
+
cfg = yaml.load(fetch(config if "://" in config else REPO + config, src / Path(config).name).read_text(), Loader=Loader)
|
|
85
98
|
ckpt = fetch(url, src / Path(url).name)
|
|
86
99
|
h = hashlib.sha256()
|
|
87
100
|
with open(ckpt, "rb") as f:
|
|
@@ -89,7 +102,7 @@ def load(src, name="mel-roformer"):
|
|
|
89
102
|
h.update(b)
|
|
90
103
|
if h.hexdigest() != sha:
|
|
91
104
|
raise SystemExit(f"{ckpt}: SHA-256 {h.hexdigest()}, expected {sha}")
|
|
92
|
-
model = MelBandRoformer(**cfg["model"])
|
|
105
|
+
model = {"MelBandRoformer": MelBandRoformer, "BSRoformer": BSRoformer}[cls](**cfg["model"])
|
|
93
106
|
state = torch.load(ckpt, map_location="cpu", weights_only=True)
|
|
94
107
|
model.load_state_dict(state.get("state_dict", state) if isinstance(state, dict) else state)
|
|
95
108
|
return model.eval(), cfg
|
|
@@ -114,6 +127,16 @@ def istft(model, y, length):
|
|
|
114
127
|
return x.reshape(B, -1, model.audio_channels, length)
|
|
115
128
|
|
|
116
129
|
|
|
130
|
+
def bands(model):
|
|
131
|
+
"""(frequencies, (f·S + s) of each band's pairs, bands in order, frequencies per band): Mel-Band RoFormer's
|
|
132
|
+
overlapping mel bands, BS-RoFormer's contiguous ones"""
|
|
133
|
+
S = model.audio_channels
|
|
134
|
+
if hasattr(model, "freq_indices"):
|
|
135
|
+
return model.num_bands_per_freq.numel(), model.freq_indices.tolist(), model.num_freqs_per_band.tolist()
|
|
136
|
+
nf = [d // (2 * S) for d in model.band_split.dim_inputs]
|
|
137
|
+
return sum(nf), list(range(sum(nf) * S)), nf
|
|
138
|
+
|
|
139
|
+
|
|
117
140
|
def groups(widths, slack=1.3):
|
|
118
141
|
"""Contiguous runs of bands whose widest is at most `slack` times their narrowest: [(first, last + 1, widest)]"""
|
|
119
142
|
out, b0 = [], 0
|
|
@@ -125,7 +148,7 @@ def groups(widths, slack=1.3):
|
|
|
125
148
|
|
|
126
149
|
|
|
127
150
|
class Core:
|
|
128
|
-
"""
|
|
151
|
+
"""The model's forward between its STFT and its iSTFT, for the GPU (the module docstring says how)."""
|
|
129
152
|
|
|
130
153
|
@staticmethod
|
|
131
154
|
def build(model, T):
|
|
@@ -134,10 +157,8 @@ class Core:
|
|
|
134
157
|
import torch.nn.functional as F
|
|
135
158
|
|
|
136
159
|
f64 = torch.float64
|
|
137
|
-
Fq = model.num_bands_per_freq.numel() # 1025
|
|
138
160
|
S = model.audio_channels
|
|
139
|
-
fi = model
|
|
140
|
-
nf = model.num_freqs_per_band.tolist()
|
|
161
|
+
Fq, fi, nf = bands(model)
|
|
141
162
|
widths = [2 * n * S for n in nf] # a band's input: its (f, s) pairs, re and im
|
|
142
163
|
G = groups(widths)
|
|
143
164
|
lin = torch.matmul
|
|
@@ -186,6 +207,9 @@ class Core:
|
|
|
186
207
|
s.blocks = nn.ModuleList()
|
|
187
208
|
for time, freq in model.layers:
|
|
188
209
|
s.blocks.append(nn.ModuleList([Block(time, T), Block(freq, len(nf))]))
|
|
210
|
+
# BS-RoFormer's one norm after the transformers (Mel-Band RoFormer's end each transformer)
|
|
211
|
+
fn = getattr(model, "final_norm", None)
|
|
212
|
+
s.final = nn.Parameter(rms(fn).float(), requires_grad=False) if fn is not None else None
|
|
189
213
|
# mask estimator (one stem): its MLP's linear layers before the last batched over bands (their shapes
|
|
190
214
|
# alike), the last by group, then GLU
|
|
191
215
|
me = model.mask_estimators[0].to_freqs
|
|
@@ -239,6 +263,8 @@ class Core:
|
|
|
239
263
|
for time, freq in s.blocks:
|
|
240
264
|
x = sliced(time, x) # sequences over time, one per band
|
|
241
265
|
x = sliced(freq, x.transpose(0, 1)).transpose(0, 1) # over bands, one per frame
|
|
266
|
+
if s.final is not None:
|
|
267
|
+
x = x / x.norm(dim=-1, keepdim=True).clamp(min=1e-12) * s.final
|
|
242
268
|
ms = []
|
|
243
269
|
for k, (lo, hi, W) in enumerate(G):
|
|
244
270
|
h = x[lo:hi]
|
|
@@ -263,7 +289,7 @@ class Core:
|
|
|
263
289
|
s.layers = nn.ModuleList()
|
|
264
290
|
for attn, ff in tr.layers:
|
|
265
291
|
s.layers.append(nn.ModuleList([Attn(attn, n), FF(ff)]))
|
|
266
|
-
s.out = nn.Parameter(rms(tr.norm).float(), requires_grad=False)
|
|
292
|
+
s.out = None if isinstance(tr.norm, nn.Identity) else nn.Parameter(rms(tr.norm).float(), requires_grad=False)
|
|
267
293
|
|
|
268
294
|
def biggest(s, n):
|
|
269
295
|
"""bytes of the largest tensor one sequence of n makes"""
|
|
@@ -274,7 +300,7 @@ class Core:
|
|
|
274
300
|
for attn, ff in s.layers:
|
|
275
301
|
x = attn(x) + x
|
|
276
302
|
x = ff(x) + x
|
|
277
|
-
return x / x.norm(dim=-1, keepdim=True).clamp(min=1e-12) * s.out
|
|
303
|
+
return x if s.out is None else x / x.norm(dim=-1, keepdim=True).clamp(min=1e-12) * s.out
|
|
278
304
|
|
|
279
305
|
class Attn(nn.Module):
|
|
280
306
|
def __init__(s, a, n):
|
|
@@ -342,7 +368,7 @@ def export(model, path, opset):
|
|
|
342
368
|
with torch.no_grad():
|
|
343
369
|
torch.onnx.export(core, (spec,), str(path), input_names=["mix_spec"], output_names=["stems_spec"],
|
|
344
370
|
opset_version=opset, dynamo=False)
|
|
345
|
-
print(f"wrote {path} ({path.stat().st_size / 1e6:.1f} MB), segment {SEGMENT} samples, spec {list(spec.shape)}, groups {groups([2 * n * model.audio_channels for n in model
|
|
371
|
+
print(f"wrote {path} ({path.stat().st_size / 1e6:.1f} MB), segment {SEGMENT} samples, spec {list(spec.shape)}, groups {groups([2 * n * model.audio_channels for n in bands(model)[2]])}")
|
|
346
372
|
return core
|
|
347
373
|
|
|
348
374
|
|
|
@@ -365,7 +391,7 @@ def engine(model, out_dir, name):
|
|
|
365
391
|
blob += bytes(pad)
|
|
366
392
|
tensors[n] = {"type": kind, "shape": list(a.shape), "offset": len(blob)}
|
|
367
393
|
blob += b
|
|
368
|
-
nf = model
|
|
394
|
+
nf = bands(model)[2]
|
|
369
395
|
meta = {
|
|
370
396
|
"model": name, "frames": T, "freqs": core.Fq, "channels": core.S, "dim": model.layers[0][0].layers[0][0].to_qkv.in_features,
|
|
371
397
|
"heads": model.layers[0][0].layers[0][0].heads, "depth": len(model.layers), "bands": len(nf),
|
|
@@ -395,20 +421,20 @@ def session(path):
|
|
|
395
421
|
return ort.InferenceSession(str(path), o, providers=["CPUExecutionProvider"])
|
|
396
422
|
|
|
397
423
|
|
|
398
|
-
def mixes():
|
|
424
|
+
def mixes(C):
|
|
399
425
|
import torch
|
|
400
426
|
|
|
401
427
|
t = torch.arange(SEGMENT) / 44100
|
|
402
428
|
tone = sum(torch.sin(2 * math.pi * f * t) / k for k, f in enumerate((110, 220, 440, 880, 1760), 1))
|
|
403
|
-
return (("noise", 0.1 * torch.randn(1,
|
|
429
|
+
return (("noise", 0.1 * torch.randn(1, C, SEGMENT)), ("tones", 0.2 * torch.stack([tone, 0.8 * tone][:C])[None]))
|
|
404
430
|
|
|
405
431
|
|
|
406
432
|
def verify(model, run, label, tol):
|
|
407
|
-
"""The
|
|
433
|
+
"""The stem rebuilt from run(spec) against the model's forward: noise and tones."""
|
|
408
434
|
import numpy as np
|
|
409
435
|
import torch
|
|
410
436
|
|
|
411
|
-
for name, mix in mixes():
|
|
437
|
+
for name, mix in mixes(model.audio_channels):
|
|
412
438
|
with torch.no_grad():
|
|
413
439
|
want = model(mix).numpy()
|
|
414
440
|
got = istft(model, torch.as_tensor(run(stft(model, mix))), SEGMENT)[:, 0].numpy()
|
|
@@ -419,20 +445,28 @@ def verify(model, run, label, tol):
|
|
|
419
445
|
raise SystemExit(f"{label}: verification failed ({rel:.2e} >= {tol})")
|
|
420
446
|
|
|
421
447
|
|
|
422
|
-
def reference(model, out_dir):
|
|
423
|
-
"""The reference separation:
|
|
424
|
-
(step 0.5, no standardization: Kim's config normalizes nothing), over a 20 s mix with a noise floor
|
|
448
|
+
def reference(model, out_dir, stem):
|
|
449
|
+
"""The reference separation: the model's forward on segments as scripts/reference.py's chunked() cuts them
|
|
450
|
+
(step 0.5, no standardization: Kim's config normalizes nothing, nor anvuew's), over a 20 s mix with a noise floor;
|
|
451
|
+
a mono model's channel by channel, as separate.js runs a mono preset."""
|
|
425
452
|
import torch
|
|
426
453
|
|
|
454
|
+
import numpy as np
|
|
455
|
+
|
|
427
456
|
sys.path.insert(0, str(Path(__file__).parent))
|
|
428
457
|
import reference as ref
|
|
429
458
|
|
|
430
459
|
def run(seg):
|
|
431
460
|
with torch.no_grad():
|
|
432
|
-
|
|
461
|
+
y = model(torch.from_numpy(seg)[None]) # (1, C, L), BS-RoFormer's (1, 1 stem, C, L)
|
|
462
|
+
return (y[0] if y.ndim == 4 else y[0][None]).numpy() # (1 stem, C, L)
|
|
433
463
|
|
|
434
464
|
# under the mix a noise floor 60 dB down: each band normalizes its input, and a silent band's is its rounding
|
|
435
|
-
|
|
465
|
+
if model.audio_channels == 1:
|
|
466
|
+
each = lambda mix: np.concatenate([ref.chunked(mix[c:c + 1], run, SEGMENT, STEP)[0] for c in range(len(mix))])
|
|
467
|
+
else:
|
|
468
|
+
each = lambda mix: ref.chunked(mix, run, SEGMENT, STEP)[0]
|
|
469
|
+
ref.write(out_dir, lambda mix: {stem: each(mix)}, 20.0, 1e-3)
|
|
436
470
|
|
|
437
471
|
|
|
438
472
|
def main():
|
|
@@ -441,9 +475,13 @@ def main():
|
|
|
441
475
|
ap.add_argument("--out-dir", default=None, help="default: $AUDIO_NEURAL_CACHE/<model> or ~/.cache/audiojs/neural/<model>")
|
|
442
476
|
ap.add_argument("--opset", type=int, default=17)
|
|
443
477
|
ap.add_argument("--fp16", action="store_true", help="also write <model>.fp16.onnx (float16 weights, float32 compute)")
|
|
444
|
-
ap.add_argument("--verify", action="store_true", help="compare with
|
|
478
|
+
ap.add_argument("--verify", action="store_true", help="compare with the model's forward; write the reference separation")
|
|
445
479
|
ap.add_argument("--engine", action="store_true", help="also write <model>.engine.bin/.json: the tensors for roformer.js, the package's WebGPU engine")
|
|
446
480
|
args = ap.parse_args()
|
|
481
|
+
if args.engine and MODELS[args.model][3] != "MelBandRoformer":
|
|
482
|
+
raise SystemExit("--engine: roformer.js runs Mel-Band RoFormer alone")
|
|
483
|
+
global SEGMENT
|
|
484
|
+
SEGMENT = MODELS[args.model][4]
|
|
447
485
|
|
|
448
486
|
cache = Path(os.environ.get("AUDIO_NEURAL_CACHE") or Path.home() / ".cache" / "audiojs" / "neural")
|
|
449
487
|
out_dir = Path(args.out_dir) if args.out_dir else cache / args.model
|
|
@@ -466,7 +504,7 @@ def main():
|
|
|
466
504
|
verify(model, lambda spec: s.run(None, {"mix_spec": spec.numpy()})[0], path.name, 1e-3)
|
|
467
505
|
del s
|
|
468
506
|
gc.collect()
|
|
469
|
-
reference(model, out_dir)
|
|
507
|
+
reference(model, out_dir, MODELS[args.model][5])
|
|
470
508
|
if args.fp16:
|
|
471
509
|
fp16(path, out_dir / f"{args.model}.fp16.onnx")
|
|
472
510
|
if args.verify:
|
package/separate.js
CHANGED
|
@@ -30,7 +30,7 @@ import resampleSinc from '@audio/resample-sinc'
|
|
|
30
30
|
import { load as neuralLoad, tensor, fetchModel, fetchJson, backends } from '@audio/neural-runtime'
|
|
31
31
|
|
|
32
32
|
const PI2 = Math.PI * 2
|
|
33
|
-
const isNode =
|
|
33
|
+
const isNode = globalThis.process?.release?.name === 'node' // Node, Bun, Deno: not a process polyfill (esm.sh's, a bundler's), whose versions.node is set
|
|
34
34
|
|
|
35
35
|
// ---------------------------------------------------------------- STFT/iSTFT
|
|
36
36
|
|
|
@@ -380,6 +380,15 @@ const MEL = {
|
|
|
380
380
|
engine: { meta: 'mel-roformer.engine.json', bin: 'mel-roformer.engine.bin', sha256: '140fedcd9d03a8f918b235688c69f4dfdd86b08f9d430277659c095688a026ff' },
|
|
381
381
|
}
|
|
382
382
|
|
|
383
|
+
// anvuew's room de-reverb (a mono BS-RoFormer, GPL-3.0 weights) as scripts/export-roformer.py exports it: its STFT
|
|
384
|
+
// (Hann of 2048, hop 512, unnormalized), 8.7 s segments of 751 frames every 4.35 s, as Music-Source-Separation-Training's
|
|
385
|
+
// demix() runs it (num_overlap 2); each channel apart, nothing normalized. Its one stem is the voice dry; the residual
|
|
386
|
+
// is the room
|
|
387
|
+
const ROOM = {
|
|
388
|
+
modelType: 'complex', sampleRate: 44100, targets: ['dry'], sources: ['dry'], n: 2048, hop: 512, window: 'hann',
|
|
389
|
+
normalized: false, segment: 384000, frames: 751, step: 0.5, standardize: false, mono: true, gpu: true, script: 'export-roformer.py',
|
|
390
|
+
}
|
|
391
|
+
|
|
383
392
|
// The compact files (scripts/compact.py: int8 weights, a few float16, float32 compute), each within 0.05 dB of its
|
|
384
393
|
// export's median SDR or SNR per stem (README, Size); their Hugging Face repositories, and SHA-256 checked when fetched
|
|
385
394
|
const FILES = {
|
|
@@ -389,6 +398,8 @@ const FILES = {
|
|
|
389
398
|
tiger: { file: 'tiger.int8.onnx', repo: 'audiojs/tiger-dnr', sha256: '2189859ac7e89cf4d9d284c6f393e737aaa2b3233ed7ee424ad622012bce7bca' },
|
|
390
399
|
// float16 weights, float32 compute: half the download; int8 (259 MB) ran slower on WebGPU and 10 dB further from float32
|
|
391
400
|
'mel-roformer': { file: 'mel-roformer.fp16.onnx', repo: 'audiojs/mel-roformer', sha256: 'adcbbb826f12453fbb86652b503242d565a9c81cdef1bca2fe0e54752eba91ef' },
|
|
401
|
+
// float16 weights, float32 compute: 60 MB, its output 49 dB from the export's
|
|
402
|
+
'dereverb-room': { file: 'dereverb-room.fp16.onnx', repo: 'audiojs/dereverb-room', sha256: '99797e086a71d1cb57d50f3b5be93dcb7c89c49d1537c60487a488a24aaee9ee' },
|
|
392
403
|
}
|
|
393
404
|
|
|
394
405
|
// The hosted revision of each compact file's repository (a commit), its URL https://huggingface.co/<repo>/resolve/<revision>/
|
|
@@ -400,6 +411,7 @@ export const REVISIONS = {
|
|
|
400
411
|
mrx: 'dedc823a0bea2ed085d19e8b7f32589d35169ee8',
|
|
401
412
|
tiger: '1c959ab7ff8794dc5663b104ba301687ab95d2ab',
|
|
402
413
|
'mel-roformer': 'c1aba02c00a0bee852d6754fe6ede8164f90a4b3',
|
|
414
|
+
'dereverb-room': 'b4c2d14da9f2199099656195b06fd8253fd16e1d',
|
|
403
415
|
}
|
|
404
416
|
|
|
405
417
|
const hosted = name => models[name].repo && REVISIONS[name] ? `https://huggingface.co/${models[name].repo}/resolve/${REVISIONS[name]}/${models[name].file}` : null
|
|
@@ -420,6 +432,9 @@ export const models = {
|
|
|
420
432
|
tiger: { ...TIGER, ...FILES.tiger },
|
|
421
433
|
// Mel-Band RoFormer (Kimberley Jensen's vocals model, MIT weights) as scripts/export-roformer.py exports it for the GPU
|
|
422
434
|
'mel-roformer': { ...MEL, ...FILES['mel-roformer'] },
|
|
435
|
+
// anvuew's room de-reverb (BS-RoFormer, GPL-3.0 weights) as scripts/export-roformer.py exports it: a voice in a room
|
|
436
|
+
// in, the voice dry out
|
|
437
|
+
'dereverb-room': { ...ROOM, ...FILES['dereverb-room'] },
|
|
423
438
|
}
|
|
424
439
|
|
|
425
440
|
async function cacheDir() {
|
|
@@ -560,7 +575,7 @@ function loader(opts, preset) {
|
|
|
560
575
|
let s = own && await own
|
|
561
576
|
if (s) return s
|
|
562
577
|
let spec = g.sha256 && !g.spec.startsWith('file:') ? await checked(g.spec, g.sha256) : g.spec
|
|
563
|
-
return neuralLoad(spec, { backend: await device, sessionOptions })
|
|
578
|
+
return neuralLoad(spec, { backend: await device, sessionOptions, wasmPaths: opts.wasmPaths })
|
|
564
579
|
}
|
|
565
580
|
}
|
|
566
581
|
|