@audio/neural-separate 0.5.0 → 0.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -63,6 +63,14 @@ together (0.9 GB of it the browser idle): wasm `scnet` 3.0 GB, `scnet-large` 4.6
63
63
  on wasm 9 and 0.3 minutes. Float16 compute does not pay: `tiger`'s on WebGPU is 13 % faster and its music 4.7 dB from
64
64
  the float32 graph's.
65
65
 
66
+ A voice's room (de-reverb):
67
+
68
+ | `model` | Architecture | Files | Weights | Targets | Real-time factor: Node CPU · WebGPU |
69
+ |---|---|---|---|---|---|
70
+ | `'dereverb-room'` | anvuew's room de-reverb: a mono BS-RoFormer (axial transformers over time and 63 contiguous bands of the complex spectrogram, dim 128, depth 16, 29.4 M parameters), trained on voices in rooms simulated with pyroomacoustics | 60 MB, float16 weights (export 122 MB) | GPL-3.0 | `dry`: the voice with its room off, early reflections and tail; the residual is the room | 2.3 · 0.65 (Node's WebGPU) |
71
+
72
+ Each channel heard apart, 8.7 s segments every 4.35 s as `demix()` runs it (`num_overlap: 2`), nothing normalized. Measured through [`audio`](https://github.com/audiojs/audio)'s `dereverb()` (`bench/rx/dereverb.mjs`, test): VoiceBank test utterances in 19 MIT IR Survey rooms (Traer & McDermott, PNAS 2016; T60 0.38–1.85 s), each response split at direct + 50 ms, PESQ against the dry take 1.39 → 2.65, STOI 0.77 → 0.94, DNSMOS OVRL 2.19 → 2.98, the late tail 15.6 dB down; iZotope RX 12 De-reverb tuned 1.45, its Dialogue Isolate with the reverb off 1.79. It takes the early reflections too: the output stands 5.2 dB under the direct sound and its first 50 ms, where the voice is 10 dB over the tail. Music is room to it (MUSDB18 previews, GuitarSet, sung tones and four pieces lose 6 dB in their worst octave as they are, and 9 dB of themselves in a room); `audio` passes music by a speech/music guard. The real-time factors are of 20 s of stereo on a busy 14-core M4 Max (load averages over 100), so upper bounds; the float16 file stands 74 dB from the PyTorch forward through this package's pipeline, the float32 export 123 dB.
73
+
66
74
  `targets` picks a subset: `targets: ['vocals']` runs only the vocals graph of `umxhq` (against the residual, see Algorithm: 2.4× faster, vocals SDR 6.50 instead of 6.75) and of `htdemucs_ft`, and skips the other sources' iSTFT for `htdemucs`. Presets resample to their 44.1 kHz and back.
67
75
 
68
76
  ## Weights
@@ -269,6 +277,7 @@ let vocals = complex.vocals.map(ch => istft(ch, { length: left.length }))
269
277
  | `window` / `normalized` / `frames` | the preset's | `'complex'`: the model's STFT window (`'ones'`, torch.stft's window=None, or `'hann'`), its 1/√n scaling, its frames per segment when the graph does not declare them |
270
278
  | `targetRate` | the preset's rate, else the input rate | resample to the model's rate for inference; stems are resampled back to the input rate |
271
279
  | `device` | — | passed through to `@audio/neural-runtime`'s `load()` as `backend` |
280
+ | `wasmPaths` | — | where onnxruntime-web's `.wasm` files are served, passed through to `load()`: needed where ORT itself comes from a CDN that rebuilds it (esm.sh), e.g. `'https://cdn.jsdelivr.net/npm/onnxruntime-web@1.30.0/dist/'` |
272
281
  | `dtype` | `'float32'` | only `'float32'` tensor marshalling is implemented; anything else throws |
273
282
  | `progress` | — | `({ chunk, totalChunks }) => {}` |
274
283
  | `session` | — | overrides `@audio/neural-runtime`'s `load()` — for tests, or a custom ORT setup |
@@ -287,6 +296,7 @@ Open-source stem separation has these lineages; this package runs SCNet, Open-Un
287
296
  | **SCNet** (`scnet`, `scnet-large`; starrytong) | MIT | MIT: "I confirm that the released SCNet and SCNet-large pretrained weights are distributed under the MIT License, consistent with the source code. You are welcome to redistribute the original checkpoints and format-converted versions, including ONNX exports, as part of your MIT-licensed tool, with appropriate attribution." (the author in [starrytong/SCNet#35](https://github.com/starrytong/SCNet/issues/35), 2026) | trained on MUSDB18-HQ; hosted with its configs by [ZFTurbo/Music-Source-Separation-Training](https://github.com/ZFTurbo/Music-Source-Separation-Training) (MIT), whose own MUSDB18-HQ checkpoints (SCNet XL, BS-RoFormer) carry its author's MIT statement only for some files ([#254](https://github.com/ZFTurbo/Music-Source-Separation-Training/issues/254)) |
288
297
  | **MRX** (MERL; `mrx`) | MIT | MIT: the repository's [`.reuse/dep5`](https://github.com/merlresearch/cocktail-fork-separation/blob/main/.reuse/dep5) names the four checkpoints, "Copyright: 2023 Mitsubishi Electric Research Laboratories (MERL) / License: MIT" | trained on Divide and Remaster v2, whose music and effects come from FMA and FSD50K clips each under its own licence, some non-commercial (the DnR v3 paper's account) |
289
298
  | **TIGER** (Tsinghua; `tiger`) | MIT (repository LICENSE; its README badge says Apache 2.0) | Apache 2.0: the [model card](https://huggingface.co/JusperLee/TIGER-DnR) states `license: apache-2.0` | trained on Divide and Remaster v1 (652 test mixtures in its paper), sourced as v2 is |
299
+ | **anvuew's de-reverb** (`dereverb-room`) | the architecture: BS-RoFormer (Phil Wang's [BS-RoFormer](https://github.com/lucidrains/BS-RoFormer), MIT, in ZFTurbo's Music-Source-Separation-Training) | **GPL-3.0** ([model card](https://huggingface.co/anvuew/dereverb_room)): the converted copy is hosted under the same licence, fetched at run time, never bundled | trained on convolutional reverbs from pyroomacoustics, the dry voices not named |
290
300
  | **Bandit v2** (DnR v3) | Apache 2.0 | **CC BY-SA 4.0** ([Zenodo](https://zenodo.org/records/12701995)): a converted copy is an adaptation under the same terms | not shipped; BandIt v1's DnR checkpoints are CC BY-NC 4.0 ([Zenodo](https://zenodo.org/records/10160698)) |
291
301
  | **Spleeter** (Deezer) | MIT | **undocumented** | the README licenses only "the code of Spleeter"; the pretrained weights' license is an open, unresolved question ([deezer/spleeter#898](https://github.com/deezer/spleeter/issues/898)) — do not assume MIT |
292
302
 
package/index.d.ts CHANGED
@@ -131,6 +131,9 @@ export interface SeparateOptions {
131
131
  hop?: number
132
132
  /** passed through to @audio/neural-runtime's load() as `backend` */
133
133
  device?: string
134
+ /** where onnxruntime-web's .wasm files are served (a URL prefix, or a file → URL map), passed through to
135
+ * @audio/neural-runtime's load(): for an ORT build loaded from a CDN that rebuilds it (esm.sh) */
136
+ wasmPaths?: string | Record<string, string>
134
137
  /** only 'float32' is implemented; anything else throws */
135
138
  dtype?: 'float32'
136
139
  progress?: (p: { chunk: number; totalChunks: number }) => void
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@audio/neural-separate",
3
- "version": "0.5.0",
3
+ "version": "0.7.0",
4
4
  "description": "Source separation (stems): a song's by SCNet (MIT weights), Hybrid Transformer Demucs, Open-Unmix with multichannel Wiener refinement; a soundtrack's dialogue, music and effects by MRX (MIT weights) and TIGER (Apache-2.0); model-agnostic ONNX",
5
5
  "type": "module",
6
6
  "sideEffects": false,
@@ -29,7 +29,7 @@
29
29
  "NOTICE"
30
30
  ],
31
31
  "dependencies": {
32
- "@audio/neural-runtime": "^0.1.1",
32
+ "@audio/neural-runtime": "^0.2.0",
33
33
  "@audio/resample-sinc": "^1.1.2",
34
34
  "fourier-transform": "^2.4.1"
35
35
  },
@@ -1,14 +1,22 @@
1
1
  #!/usr/bin/env python3
2
2
  """Export Mel-Band RoFormer (Wang, Lu, Won, Chen, Song, "Mel-Band RoFormer for Music Source Separation", ISMIR LBD
3
- 2023) to the ONNX contract @audio/neural-separate expects for modelType 'complex': the network between its STFT and
3
+ 2023) and Band-Split RoFormer (Lu, Wang, Kong, Hung, "Music Source Separation with Band-Split RoPE Transformer", ICASSP
4
+ 2024) to the ONNX contract @audio/neural-separate expects for modelType 'complex': the network between its STFT and
4
5
  iSTFT, which JS computes, written for the GPU.
5
6
 
6
7
  pip install torch onnx onnxruntime pyyaml einops beartype rotary-embedding-torch librosa
7
8
  python3 export-roformer.py --model mel-roformer --verify
8
-
9
- The model is Kimberley Jensen's vocals Mel-Band RoFormer (huggingface.co/KimberleyJSN/melbandroformer, MIT), with the
10
- config ZFTurbo/Music-Source-Separation-Training (MIT) runs it by (configs/KimberleyJensen/
11
- config_vocals_mel_band_roformer_kj.yaml) and that repository's models/bs_roformer code, both at a pinned commit.
9
+ python3 export-roformer.py --model dereverb-room --verify --fp16
10
+
11
+ The models, each run by ZFTurbo/Music-Source-Separation-Training's (MIT) models/bs_roformer code at a pinned commit:
12
+ mel-roformer Kimberley Jensen's vocals Mel-Band RoFormer (huggingface.co/KimberleyJSN/melbandroformer, MIT), with
13
+ the config that repository runs it by (configs/KimberleyJensen/config_vocals_mel_band_roformer_kj.yaml)
14
+ dereverb-room anvuew's room de-reverb, a mono BS-RoFormer (huggingface.co/anvuew/dereverb_room, GPL-3.0): a voice
15
+ in a room in, the voice dry out; trained on voices in rooms pyroomacoustics simulated (its card); its
16
+ own config, dim 128, depth 16, 63 bands, 29.4 M parameters
17
+ Each checkpoint and config at a pinned revision, the checkpoint checked by its SHA-256. What follows is told of
18
+ mel-roformer; dereverb-room is one channel, 8.7 s segments (384000 samples, hop 512, 751 frames), its bands contiguous
19
+ (each frequency in one band, so its masks are taken as they come) and one RMSNorm after its transformers, not one each.
12
20
 
13
21
  One 8 s segment (352800 samples at 44.1 kHz, 801 frames) per run:
14
22
  input mix_spec [1, 4, 1025, 801] STFT, complex as channels (L re, L im, R re, R im)
@@ -45,12 +53,16 @@ import urllib.request
45
53
  from pathlib import Path
46
54
 
47
55
  REPO = "https://raw.githubusercontent.com/ZFTurbo/Music-Source-Separation-Training/84b1eac0887756b4f1a9d7a1ff49105939749ed2/"
48
- MODELS = { # checkpoint URL, its SHA-256, config path in REPO
56
+ MODELS = { # checkpoint URL, its SHA-256, config (a path in REPO or a URL), class, segment (samples), the stem's name
49
57
  "mel-roformer": ("https://huggingface.co/KimberleyJSN/melbandroformer/resolve/ac9b0614ab3cd7f77219e18ba494dfd93956c348/MelBandRoformer.ckpt",
50
58
  "87201f4d31afb5bc79993230fc49446918425574db48c01c405e44f365c7559e",
51
- "configs/KimberleyJensen/config_vocals_mel_band_roformer_kj.yaml"),
59
+ "configs/KimberleyJensen/config_vocals_mel_band_roformer_kj.yaml", "MelBandRoformer", 352800, "vocals"),
60
+ "dereverb-room": ("https://huggingface.co/anvuew/dereverb_room/resolve/0b85f5b80b7f779b2dfe80f33a1b35b38af9376d/dereverb_room_anvuew_sdr_13.7432.ckpt",
61
+ "2edec521f09e26341c1923dc82c8c52dbc86478b42b9999f679535743c970cb3",
62
+ "https://huggingface.co/anvuew/dereverb_room/resolve/0b85f5b80b7f779b2dfe80f33a1b35b38af9376d/dereverb_room_anvuew.yaml",
63
+ "BSRoformer", 384000, "dry"),
52
64
  }
53
- SEGMENT = 352800
65
+ SEGMENT = 352800 # the model's, set by main()
54
66
  STEP = 0.5
55
67
 
56
68
 
@@ -63,7 +75,7 @@ def fetch(url, path):
63
75
 
64
76
 
65
77
  def load(src, name="mel-roformer"):
66
- """MelBandRoformer with its weights: the model code, config and checkpoint fetched into `src` once."""
78
+ """The model with its weights: the model code, config and checkpoint fetched into `src` once."""
67
79
  import hashlib
68
80
  import torch
69
81
  import yaml
@@ -72,16 +84,17 @@ def load(src, name="mel-roformer"):
72
84
  pkg.mkdir(parents=True, exist_ok=True)
73
85
  (pkg.parent / "__init__.py").touch()
74
86
  (pkg / "__init__.py").touch()
75
- for f in ("mel_band_roformer.py", "attend.py"):
87
+ for f in ("mel_band_roformer.py", "bs_roformer.py", "attend.py"):
76
88
  fetch(REPO + "models/bs_roformer/" + f, pkg / f)
77
89
  sys.path.insert(0, str(src / "roformer_code"))
78
90
  sys.dont_write_bytecode = True
79
91
  from models.bs_roformer.mel_band_roformer import MelBandRoformer
92
+ from models.bs_roformer.bs_roformer import BSRoformer
80
93
 
81
94
  class Loader(yaml.SafeLoader): pass # the config's resolutions are a !!python/tuple
82
95
  Loader.add_constructor("tag:yaml.org,2002:python/tuple", lambda l, n: tuple(l.construct_sequence(n)))
83
- url, sha, config = MODELS[name]
84
- cfg = yaml.load(fetch(REPO + config, src / Path(config).name).read_text(), Loader=Loader)
96
+ url, sha, config, cls, _, _ = MODELS[name]
97
+ cfg = yaml.load(fetch(config if "://" in config else REPO + config, src / Path(config).name).read_text(), Loader=Loader)
85
98
  ckpt = fetch(url, src / Path(url).name)
86
99
  h = hashlib.sha256()
87
100
  with open(ckpt, "rb") as f:
@@ -89,7 +102,7 @@ def load(src, name="mel-roformer"):
89
102
  h.update(b)
90
103
  if h.hexdigest() != sha:
91
104
  raise SystemExit(f"{ckpt}: SHA-256 {h.hexdigest()}, expected {sha}")
92
- model = MelBandRoformer(**cfg["model"])
105
+ model = {"MelBandRoformer": MelBandRoformer, "BSRoformer": BSRoformer}[cls](**cfg["model"])
93
106
  state = torch.load(ckpt, map_location="cpu", weights_only=True)
94
107
  model.load_state_dict(state.get("state_dict", state) if isinstance(state, dict) else state)
95
108
  return model.eval(), cfg
@@ -114,6 +127,16 @@ def istft(model, y, length):
114
127
  return x.reshape(B, -1, model.audio_channels, length)
115
128
 
116
129
 
130
+ def bands(model):
131
+ """(frequencies, (f·S + s) of each band's pairs, bands in order, frequencies per band): Mel-Band RoFormer's
132
+ overlapping mel bands, BS-RoFormer's contiguous ones"""
133
+ S = model.audio_channels
134
+ if hasattr(model, "freq_indices"):
135
+ return model.num_bands_per_freq.numel(), model.freq_indices.tolist(), model.num_freqs_per_band.tolist()
136
+ nf = [d // (2 * S) for d in model.band_split.dim_inputs]
137
+ return sum(nf), list(range(sum(nf) * S)), nf
138
+
139
+
117
140
  def groups(widths, slack=1.3):
118
141
  """Contiguous runs of bands whose widest is at most `slack` times their narrowest: [(first, last + 1, widest)]"""
119
142
  out, b0 = [], 0
@@ -125,7 +148,7 @@ def groups(widths, slack=1.3):
125
148
 
126
149
 
127
150
  class Core:
128
- """MelBandRoformer.forward between its STFT and its iSTFT, for the GPU (the module docstring says how)."""
151
+ """The model's forward between its STFT and its iSTFT, for the GPU (the module docstring says how)."""
129
152
 
130
153
  @staticmethod
131
154
  def build(model, T):
@@ -134,10 +157,8 @@ class Core:
134
157
  import torch.nn.functional as F
135
158
 
136
159
  f64 = torch.float64
137
- Fq = model.num_bands_per_freq.numel() # 1025
138
160
  S = model.audio_channels
139
- fi = model.freq_indices.tolist() # (f·S + s) per band's pair, bands in order
140
- nf = model.num_freqs_per_band.tolist()
161
+ Fq, fi, nf = bands(model)
141
162
  widths = [2 * n * S for n in nf] # a band's input: its (f, s) pairs, re and im
142
163
  G = groups(widths)
143
164
  lin = torch.matmul
@@ -186,6 +207,9 @@ class Core:
186
207
  s.blocks = nn.ModuleList()
187
208
  for time, freq in model.layers:
188
209
  s.blocks.append(nn.ModuleList([Block(time, T), Block(freq, len(nf))]))
210
+ # BS-RoFormer's one norm after the transformers (Mel-Band RoFormer's end each transformer)
211
+ fn = getattr(model, "final_norm", None)
212
+ s.final = nn.Parameter(rms(fn).float(), requires_grad=False) if fn is not None else None
189
213
  # mask estimator (one stem): its MLP's linear layers before the last batched over bands (their shapes
190
214
  # alike), the last by group, then GLU
191
215
  me = model.mask_estimators[0].to_freqs
@@ -239,6 +263,8 @@ class Core:
239
263
  for time, freq in s.blocks:
240
264
  x = sliced(time, x) # sequences over time, one per band
241
265
  x = sliced(freq, x.transpose(0, 1)).transpose(0, 1) # over bands, one per frame
266
+ if s.final is not None:
267
+ x = x / x.norm(dim=-1, keepdim=True).clamp(min=1e-12) * s.final
242
268
  ms = []
243
269
  for k, (lo, hi, W) in enumerate(G):
244
270
  h = x[lo:hi]
@@ -263,7 +289,7 @@ class Core:
263
289
  s.layers = nn.ModuleList()
264
290
  for attn, ff in tr.layers:
265
291
  s.layers.append(nn.ModuleList([Attn(attn, n), FF(ff)]))
266
- s.out = nn.Parameter(rms(tr.norm).float(), requires_grad=False)
292
+ s.out = None if isinstance(tr.norm, nn.Identity) else nn.Parameter(rms(tr.norm).float(), requires_grad=False)
267
293
 
268
294
  def biggest(s, n):
269
295
  """bytes of the largest tensor one sequence of n makes"""
@@ -274,7 +300,7 @@ class Core:
274
300
  for attn, ff in s.layers:
275
301
  x = attn(x) + x
276
302
  x = ff(x) + x
277
- return x / x.norm(dim=-1, keepdim=True).clamp(min=1e-12) * s.out
303
+ return x if s.out is None else x / x.norm(dim=-1, keepdim=True).clamp(min=1e-12) * s.out
278
304
 
279
305
  class Attn(nn.Module):
280
306
  def __init__(s, a, n):
@@ -342,7 +368,7 @@ def export(model, path, opset):
342
368
  with torch.no_grad():
343
369
  torch.onnx.export(core, (spec,), str(path), input_names=["mix_spec"], output_names=["stems_spec"],
344
370
  opset_version=opset, dynamo=False)
345
- print(f"wrote {path} ({path.stat().st_size / 1e6:.1f} MB), segment {SEGMENT} samples, spec {list(spec.shape)}, groups {groups([2 * n * model.audio_channels for n in model.num_freqs_per_band.tolist()])}")
371
+ print(f"wrote {path} ({path.stat().st_size / 1e6:.1f} MB), segment {SEGMENT} samples, spec {list(spec.shape)}, groups {groups([2 * n * model.audio_channels for n in bands(model)[2]])}")
346
372
  return core
347
373
 
348
374
 
@@ -365,7 +391,7 @@ def engine(model, out_dir, name):
365
391
  blob += bytes(pad)
366
392
  tensors[n] = {"type": kind, "shape": list(a.shape), "offset": len(blob)}
367
393
  blob += b
368
- nf = model.num_freqs_per_band.tolist()
394
+ nf = bands(model)[2]
369
395
  meta = {
370
396
  "model": name, "frames": T, "freqs": core.Fq, "channels": core.S, "dim": model.layers[0][0].layers[0][0].to_qkv.in_features,
371
397
  "heads": model.layers[0][0].layers[0][0].heads, "depth": len(model.layers), "bands": len(nf),
@@ -395,20 +421,20 @@ def session(path):
395
421
  return ort.InferenceSession(str(path), o, providers=["CPUExecutionProvider"])
396
422
 
397
423
 
398
- def mixes():
424
+ def mixes(C):
399
425
  import torch
400
426
 
401
427
  t = torch.arange(SEGMENT) / 44100
402
428
  tone = sum(torch.sin(2 * math.pi * f * t) / k for k, f in enumerate((110, 220, 440, 880, 1760), 1))
403
- return (("noise", 0.1 * torch.randn(1, 2, SEGMENT)), ("tones", 0.2 * torch.stack([tone, 0.8 * tone])[None]))
429
+ return (("noise", 0.1 * torch.randn(1, C, SEGMENT)), ("tones", 0.2 * torch.stack([tone, 0.8 * tone][:C])[None]))
404
430
 
405
431
 
406
432
  def verify(model, run, label, tol):
407
- """The vocals rebuilt from run(spec) against MelBandRoformer.forward: noise and tones."""
433
+ """The stem rebuilt from run(spec) against the model's forward: noise and tones."""
408
434
  import numpy as np
409
435
  import torch
410
436
 
411
- for name, mix in mixes():
437
+ for name, mix in mixes(model.audio_channels):
412
438
  with torch.no_grad():
413
439
  want = model(mix).numpy()
414
440
  got = istft(model, torch.as_tensor(run(stft(model, mix))), SEGMENT)[:, 0].numpy()
@@ -419,20 +445,28 @@ def verify(model, run, label, tol):
419
445
  raise SystemExit(f"{label}: verification failed ({rel:.2e} >= {tol})")
420
446
 
421
447
 
422
- def reference(model, out_dir):
423
- """The reference separation: MelBandRoformer.forward on segments as scripts/reference.py's chunked() cuts them
424
- (step 0.5, no standardization: Kim's config normalizes nothing), over a 20 s mix with a noise floor."""
448
+ def reference(model, out_dir, stem):
449
+ """The reference separation: the model's forward on segments as scripts/reference.py's chunked() cuts them
450
+ (step 0.5, no standardization: Kim's config normalizes nothing, nor anvuew's), over a 20 s mix with a noise floor;
451
+ a mono model's channel by channel, as separate.js runs a mono preset."""
425
452
  import torch
426
453
 
454
+ import numpy as np
455
+
427
456
  sys.path.insert(0, str(Path(__file__).parent))
428
457
  import reference as ref
429
458
 
430
459
  def run(seg):
431
460
  with torch.no_grad():
432
- return model(torch.from_numpy(seg)[None])[0][None].numpy() # (1 stem, C, L)
461
+ y = model(torch.from_numpy(seg)[None]) # (1, C, L), BS-RoFormer's (1, 1 stem, C, L)
462
+ return (y[0] if y.ndim == 4 else y[0][None]).numpy() # (1 stem, C, L)
433
463
 
434
464
  # under the mix a noise floor 60 dB down: each band normalizes its input, and a silent band's is its rounding
435
- ref.write(out_dir, lambda mix: {"vocals": ref.chunked(mix, run, SEGMENT, STEP)[0]}, 20.0, 1e-3)
465
+ if model.audio_channels == 1:
466
+ each = lambda mix: np.concatenate([ref.chunked(mix[c:c + 1], run, SEGMENT, STEP)[0] for c in range(len(mix))])
467
+ else:
468
+ each = lambda mix: ref.chunked(mix, run, SEGMENT, STEP)[0]
469
+ ref.write(out_dir, lambda mix: {stem: each(mix)}, 20.0, 1e-3)
436
470
 
437
471
 
438
472
  def main():
@@ -441,9 +475,13 @@ def main():
441
475
  ap.add_argument("--out-dir", default=None, help="default: $AUDIO_NEURAL_CACHE/<model> or ~/.cache/audiojs/neural/<model>")
442
476
  ap.add_argument("--opset", type=int, default=17)
443
477
  ap.add_argument("--fp16", action="store_true", help="also write <model>.fp16.onnx (float16 weights, float32 compute)")
444
- ap.add_argument("--verify", action="store_true", help="compare with MelBandRoformer.forward; write the reference separation")
478
+ ap.add_argument("--verify", action="store_true", help="compare with the model's forward; write the reference separation")
445
479
  ap.add_argument("--engine", action="store_true", help="also write <model>.engine.bin/.json: the tensors for roformer.js, the package's WebGPU engine")
446
480
  args = ap.parse_args()
481
+ if args.engine and MODELS[args.model][3] != "MelBandRoformer":
482
+ raise SystemExit("--engine: roformer.js runs Mel-Band RoFormer alone")
483
+ global SEGMENT
484
+ SEGMENT = MODELS[args.model][4]
447
485
 
448
486
  cache = Path(os.environ.get("AUDIO_NEURAL_CACHE") or Path.home() / ".cache" / "audiojs" / "neural")
449
487
  out_dir = Path(args.out_dir) if args.out_dir else cache / args.model
@@ -466,7 +504,7 @@ def main():
466
504
  verify(model, lambda spec: s.run(None, {"mix_spec": spec.numpy()})[0], path.name, 1e-3)
467
505
  del s
468
506
  gc.collect()
469
- reference(model, out_dir)
507
+ reference(model, out_dir, MODELS[args.model][5])
470
508
  if args.fp16:
471
509
  fp16(path, out_dir / f"{args.model}.fp16.onnx")
472
510
  if args.verify:
package/separate.js CHANGED
@@ -30,7 +30,7 @@ import resampleSinc from '@audio/resample-sinc'
30
30
  import { load as neuralLoad, tensor, fetchModel, fetchJson, backends } from '@audio/neural-runtime'
31
31
 
32
32
  const PI2 = Math.PI * 2
33
- const isNode = typeof process !== 'undefined' && !!process.versions?.node
33
+ const isNode = globalThis.process?.release?.name === 'node' // Node, Bun, Deno: not a process polyfill (esm.sh's, a bundler's), whose versions.node is set
34
34
 
35
35
  // ---------------------------------------------------------------- STFT/iSTFT
36
36
 
@@ -380,6 +380,15 @@ const MEL = {
380
380
  engine: { meta: 'mel-roformer.engine.json', bin: 'mel-roformer.engine.bin', sha256: '140fedcd9d03a8f918b235688c69f4dfdd86b08f9d430277659c095688a026ff' },
381
381
  }
382
382
 
383
+ // anvuew's room de-reverb (a mono BS-RoFormer, GPL-3.0 weights) as scripts/export-roformer.py exports it: its STFT
384
+ // (Hann of 2048, hop 512, unnormalized), 8.7 s segments of 751 frames every 4.35 s, as Music-Source-Separation-Training's
385
+ // demix() runs it (num_overlap 2); each channel apart, nothing normalized. Its one stem is the voice dry; the residual
386
+ // is the room
387
+ const ROOM = {
388
+ modelType: 'complex', sampleRate: 44100, targets: ['dry'], sources: ['dry'], n: 2048, hop: 512, window: 'hann',
389
+ normalized: false, segment: 384000, frames: 751, step: 0.5, standardize: false, mono: true, gpu: true, script: 'export-roformer.py',
390
+ }
391
+
383
392
  // The compact files (scripts/compact.py: int8 weights, a few float16, float32 compute), each within 0.05 dB of its
384
393
  // export's median SDR or SNR per stem (README, Size); their Hugging Face repositories, and SHA-256 checked when fetched
385
394
  const FILES = {
@@ -389,6 +398,8 @@ const FILES = {
389
398
  tiger: { file: 'tiger.int8.onnx', repo: 'audiojs/tiger-dnr', sha256: '2189859ac7e89cf4d9d284c6f393e737aaa2b3233ed7ee424ad622012bce7bca' },
390
399
  // float16 weights, float32 compute: half the download; int8 (259 MB) ran slower on WebGPU and 10 dB further from float32
391
400
  'mel-roformer': { file: 'mel-roformer.fp16.onnx', repo: 'audiojs/mel-roformer', sha256: 'adcbbb826f12453fbb86652b503242d565a9c81cdef1bca2fe0e54752eba91ef' },
401
+ // float16 weights, float32 compute: 60 MB, its output 49 dB from the export's
402
+ 'dereverb-room': { file: 'dereverb-room.fp16.onnx', repo: 'audiojs/dereverb-room', sha256: '99797e086a71d1cb57d50f3b5be93dcb7c89c49d1537c60487a488a24aaee9ee' },
392
403
  }
393
404
 
394
405
  // The hosted revision of each compact file's repository (a commit), its URL https://huggingface.co/<repo>/resolve/<revision>/
@@ -400,6 +411,7 @@ export const REVISIONS = {
400
411
  mrx: 'dedc823a0bea2ed085d19e8b7f32589d35169ee8',
401
412
  tiger: '1c959ab7ff8794dc5663b104ba301687ab95d2ab',
402
413
  'mel-roformer': 'c1aba02c00a0bee852d6754fe6ede8164f90a4b3',
414
+ 'dereverb-room': 'b4c2d14da9f2199099656195b06fd8253fd16e1d',
403
415
  }
404
416
 
405
417
  const hosted = name => models[name].repo && REVISIONS[name] ? `https://huggingface.co/${models[name].repo}/resolve/${REVISIONS[name]}/${models[name].file}` : null
@@ -420,6 +432,9 @@ export const models = {
420
432
  tiger: { ...TIGER, ...FILES.tiger },
421
433
  // Mel-Band RoFormer (Kimberley Jensen's vocals model, MIT weights) as scripts/export-roformer.py exports it for the GPU
422
434
  'mel-roformer': { ...MEL, ...FILES['mel-roformer'] },
435
+ // anvuew's room de-reverb (BS-RoFormer, GPL-3.0 weights) as scripts/export-roformer.py exports it: a voice in a room
436
+ // in, the voice dry out
437
+ 'dereverb-room': { ...ROOM, ...FILES['dereverb-room'] },
423
438
  }
424
439
 
425
440
  async function cacheDir() {
@@ -560,7 +575,7 @@ function loader(opts, preset) {
560
575
  let s = own && await own
561
576
  if (s) return s
562
577
  let spec = g.sha256 && !g.spec.startsWith('file:') ? await checked(g.spec, g.sha256) : g.spec
563
- return neuralLoad(spec, { backend: await device, sessionOptions })
578
+ return neuralLoad(spec, { backend: await device, sessionOptions, wasmPaths: opts.wasmPaths })
564
579
  }
565
580
  }
566
581