@audio/neural-denoise 0.1.0 → 0.2.0

package/README.md

@@ -17,7 +17,7 @@ let better = await denoise(pcm, { sampleRate: 44100, model: 'deepfilternet3' })
 |---|---|---|
 | `sampleRate` | | required unless `audio` carries it |
 | `model` | `'rnnoise'` | `'rnnoise'`, `'deepfilternet3'`, or a handle from `load()` |
-| `limit` | `20` RNNoise, `12` DeepFilterNet3 | attenuation limit, dB: the input is mixed back in at 10^(−limit/20), so noise drops by at most `limit`; `0` for none, upstream's output. Unlimited, RNNoise removes the voice on 12 of the 824 VoiceBank+DEMAND files (STOI down by more than 0.2), on none at 20 dB, where PESQ rises from 2.11 to 2.46; unlimited, DeepFilterNet3 takes the pauses of home narrations down to −152 dBFS, the digital silence ACX Check flags, and at 12 dB keeps them at −91 to −57 dBFS with the same DNSMOS, at a cost on noisy speech: VoiceBank+DEMAND PESQ 2.67 against 3.16 unlimited ([Accuracy](#accuracy)) |
+| `limit` | `20` RNNoise, `18` DeepFilterNet3 | attenuation limit, dB: the input is mixed back in at 10^(−limit/20), so noise drops by at most `limit`; `0` for none. Unlimited, RNNoise removes the voice on 12 of the 824 VoiceBank+DEMAND files (STOI down by more than 0.2), on none at 20 dB, where PESQ rises from 2.11 to 2.46. DeepFilterNet3's 18 dB is the most before the voice itself suffers: on VoiceBank+DEMAND its DNSMOS SIG holds from 12 to 18 dB and falls past it, while BAK and PESQ keep rising; unlimited, it takes the pauses of home narrations to digital silence, which ACX Check flags ([Accuracy](#accuracy)) |
 | `weights` | | RNNoise: bytes of an upstream weight blob; DeepFilterNet3: URL or bytes of an upstream ONNX export |
 | `device` | `'auto'` | DeepFilterNet3: `@audio/neural-runtime` backend, `'node'`, `'wasm'` or `'webgpu'` |
 | `sessionOptions` | | DeepFilterNet3: ONNX Runtime session options, e.g. `{ intraOpNumThreads: 4 }` |
@@ -68,7 +68,7 @@ let voice = st.process(input480, output480)  // int16 scale in and out; voice pr
 
 ## In `audio`: the contract atom
 
-`@audio/neural-denoise/audio` is the package's [contract](https://github.com/audiojs/compile/blob/main/CONTRACT.md) manifest: `rnnoise`, a streaming atom at any sample rate, with one parameter, `limit` (dB, default 20, 0 for none). [`audio`](https://github.com/audiojs/audio) hosts it as its `rnnoise` op, and runs DeepFilterNet3 as its `deepfilter` op (the whole input before rendering, `limit` 12 by default):
+`@audio/neural-denoise/audio` is the package's [contract](https://github.com/audiojs/compile/blob/main/CONTRACT.md) manifest: `rnnoise`, a streaming atom at any sample rate, with one parameter, `limit` (dB, default 20, 0 for none). [`audio`](https://github.com/audiojs/audio) hosts it as its `rnnoise` op, and runs DeepFilterNet3 as its `deepfilter` op (the whole input before rendering, `limit` 18 by default):
 
 ```js
 import audio from 'audio'
@@ -96,9 +96,10 @@ The float code keeps C's float semantics: each float op is `Math.fround(a op b)`
 Upstream's three ONNX graphs (encoder, ERB-gain decoder, deep-filter decoder) run unchanged; the rest is ported from DeepFilterNet at d375b2d, `libDF` and `df.enhance.enhance()`:
 
 1. **STFT**: 960-point Vorbis window, 480 hop, the input padded by 960 zeros; the same FFT as RNNoise.
-2. **Features**: power in 32 ERB bands (libDF's `erb_fb`: 2 to 67 bins wide) in dB, mean-normalized with α = 0.99 (τ = 1 s); the lowest 96 bins, unit-normalized with the same α. Both run 2 frames ahead of the output (`conv_lookahead`).
+2. **Features**: power in 32 ERB bands (libDF's `erb_fb`: 2 to 67 bins wide) in dB, mean-normalized with α = 0.99 (τ = 1 s); the lowest 96 bins, unit-normalized with the same α. Both run 2 frames ahead of the output (`conv_lookahead`). `denoise()` computes them as if the input's speech (the louder half of its 50 ms frames) were at −20 dBFS, the median of the VoiceBank+DEMAND test set (−20.2): they are not level-free, the ERB means start at −60 to −90 dB and the complex features scale with the root of the level, and the model was trained at speech gains of −6, 0 and +6 dB (libDF `dataset.rs`). Heard as it is, a take 24 dB quieter lost its pauses' noise by 11 to 14 dB instead of 22 to 24, and the noise after 2.7 s of digital silence by 10 to 24 dB; at −20 dBFS it does as the louder take, at no cost on VoiceBank+DEMAND (PESQ 3.077 against 3.076, 206 files). The spectrum the gains and filters apply to stays as it was.
 3. **Network**: 32 gains (sigmoid) and 96 × 5 complex filter taps per frame.
 4. **Enhance**: every bin takes its band's gain; below bin 96 (4.8 kHz) the output is instead Σₖ spec[t − 2 + k] · coef[t, k], k = 0–4, over the unmasked spectrum; `limit` mixes the noisy spectrum back in. Inverse FFT, overlap-add, the 480-sample STFT delay removed.
+5. **Voice guard** (`denoise()`): the model takes held voicing for noise. A clean held note from VocalSet (Wilkins 2018) comes out 39 dB down, unlimited, its own local-SNR estimate at its floor (−15 dB), while speech keeps its level; sung phrases lose 24 to 34 dB. So where the input holds sustained voicing, runs of 0.3 s or more periodic at RNNoise's pitch (normalized correlation 0.45 or more, Praat's voicing threshold, the pitch searched every 20 ms) that stand 6 dB or more over the quietest 100 ms of the 20 s around (minimum statistics, Martin 2001: a buzz is that background, a note stands over it), each band holding a harmonic keeps at least c² of its input, c its correlation with the frame one and two periods earlier: the periodic share of its power, as RNNoise's pitch filter restores what its band gains would remove. Between the harmonics and in the pauses the model's output stands.
 
 The parameters come from the export's `config.ini`, so upstream's low-latency export (`DeepFilterNet3_ll_onnx.tar.gz`: no lookahead, 36 MB) loads through `weights` too.
 
@@ -112,7 +113,8 @@ The parameters come from the export's `config.ini`, so upstream's low-latency ex
 | RNNoise vs upstream C as `configure && make` builds it on arm64 (NEON, fused multiply-adds) | 38.8–62.4 dB SNR, voice probability within 0.087: upstream's builds differ from each other by this much (NEON's `vrecpeq_f32` reciprocal estimate inside tanh and sigmoid) |
 | `rnnoise.bin` | byte for byte what upstream's `dump_weights_blob` writes from the model tarball (sha256 `1ad07b42…`) |
 | FFT, window and DCT tables | equal to upstream's `rnnoise_tables.c`: 960 twiddles, 480 window values, 1024 DCT entries, the bit reversal; the FFT matches a direct DFT to 3.2e-9 |
-| DeepFilterNet3 vs Python `df.enhance.enhance()` (deepfilternet 0.5.6, PyTorch checkpoint), same input, `limit: 0` | 133.5 dB SNR on a 1.7 s utterance, 128.2 dB on 11.3 s (max \|d\| 5.7e-7); ERB widths and α identical |
+| DeepFilterNet3, `enhance()` with upstream's settings (no level change, no voice guard, whole file), vs Python `df.enhance.enhance()` (deepfilternet 0.5.6, PyTorch checkpoint), same input, `limit: 0` | 133.5 dB SNR on a 1.7 s utterance, 128.2 dB on 11.3 s (max \|d\| 5.7e-7), 133.8 dB on an utterance led by 2 s of noise, 134.5 dB on a 60 s narration, its first second included; ERB widths and α identical |
+| DeepFilterNet3 voice guard, no model (`test.js`) | a held /a/ (190 Hz, 5 Hz vibrato) 20 dB over white noise through a model that removes everything: kept within 2.8 dB, the noise around it removed; a 120 Hz buzz under the same noise: no frame guarded |
 | DeepFilterNet3 plumbing, no model (`test.js`) | identity gains and taps give the input back to 1.8e-7, whole and chunked (chunks identical to the whole run); a tap one frame ahead advances the band below 4.8 kHz by exactly 480 samples (to 8.9e-8) |
 | Worklet (`test.js`, AudioWorkletGlobalScope simulated) | the frame API's output, delayed by 448 samples, at `limit: 0`; at 12 and at the default 20, that output with the input mixed back, to 1e-7 |
 | Worklet in Chromium 153 (headless, Playwright, `scripts/worklet.mjs`) | runs, no processor error, output non-silent; live underruns in Speed |
@@ -135,12 +137,14 @@ The VoiceBank+DEMAND test set (Valentini-Botinhao 2017, CC BY 4.0): 824 utteranc
 | RNNoise, `limit: 0` (this package = upstream C) | 2.11 | 0.890 | 12.28 | 3.28 | 3.85 | 2.94 |
 | RNNoise, little model, `limit: 0` | 2.12 | 0.893 | 12.39 | 3.28 | 3.85 | 2.93 |
 | RNNoise, `limit: 20`, the default | 2.46 | 0.918 | 12.86 | 3.40 | 3.71 | 2.97 |
-| **DeepFilterNet3**, `limit: 0` (this package) | **3.16** | **0.945** | **18.96** | **3.44** | **4.08** | **3.18** |
+| **DeepFilterNet3**, upstream's settings, `limit: 0` (`enhance()`) | **3.16** | **0.945** | **18.96** | **3.44** | **4.08** | **3.18** |
 | DeepFilterNet3, Python (deepfilternet 0.5.6) | 3.16 | 0.945 | 18.96 | 3.44 | 4.08 | 3.18 |
 | DeepFilterNet3 in 1 s chunks, `limit: 0` | 3.16 | 0.945 | 18.94 | 3.44 | 4.08 | 3.18 |
-| DeepFilterNet3, `limit: 12`, the default | 2.67 | 0.939 | 16.18 | 3.49 | 3.69 | 3.03 |
+| DeepFilterNet3, `denoise()`, `limit: 0`: heard at −20 dBFS, voice guard | 3.15 | 0.945 | 18.95 | 3.44 | 4.08 | 3.18 |
+| DeepFilterNet3, `denoise()`, `limit: 18`, the default | 2.90 | 0.942 | 17.91 | 3.48 | 3.84 | 3.09 |
+| DeepFilterNet3, `limit: 12` (the default before 0.2) | 2.67 | 0.939 | 16.18 | 3.49 | 3.69 | 3.03 |
 
-Over the noisy input, paired, with 95% intervals: DeepFilterNet3 adds 1.20 ± 0.03 PESQ, 0.024 ± 0.002 STOI and 10.5 ± 0.3 dB SI-SDR, and beats every classical row's PESQ on 804 of 824 files; RNNoise adds 0.14 ± 0.04 PESQ and 3.8 ± 0.3 dB SI-SDR but loses 0.031 ± 0.004 STOI; `omlsa`, the best classical row, adds 0.43 ± 0.03 PESQ and 6.1 ± 0.2 dB SI-SDR and loses 0.006 STOI. DeepFilterNet3's default 12 dB limit leaves noise in on this set: against `limit: 0` it costs 0.49 ± 0.02 PESQ, 2.8 ± 0.2 dB SI-SDR and 0.39 BAK, and gains 0.05 SIG; it still adds 0.70 ± 0.02 PESQ over the input, 0.27 ± 0.02 over `omlsa` and 0.21 over RNNoise at its default. Pass `limit: 0` where the most noise removal matters more than room tone. The same scoring gives upstream's published anchors: noisy 1.97 and 0.921 (as the DeepFilterNet2 paper's table), DeepFilterNet3 3.16 and 0.945 against the DeepFilterNet3 paper's 3.17 and 0.944. That DeepFilterNet2 table measured RNNoise's 2022 model at 2.33 and 0.922; the current model scores lower here.
+Over the noisy input, paired, with 95% intervals: DeepFilterNet3 adds 1.20 ± 0.03 PESQ, 0.024 ± 0.002 STOI and 10.5 ± 0.3 dB SI-SDR, and beats every classical row's PESQ on 804 of 824 files; RNNoise adds 0.14 ± 0.04 PESQ and 3.8 ± 0.3 dB SI-SDR but loses 0.031 ± 0.004 STOI; `omlsa`, the best classical row, adds 0.43 ± 0.03 PESQ and 6.1 ± 0.2 dB SI-SDR and loses 0.006 STOI. A limit leaves noise in on this set, and spares the voice: paired, on a quarter of the files (206), DNSMOS SIG moves −0.002 ± 0.015 from 12 to 18 dB, then −0.008 ± 0.003 to 20, −0.019 ± 0.006 more to 24 and −0.049 ± 0.017 more unlimited, while BAK rises 0.18, 0.04, 0.07 and 0.16 and PESQ 0.25, 0.06, 0.08 and 0.10. Past 18 dB the speech itself loses, which DNSMOS reads as SIG and a listener as a filtered, phone-call voice; from 8 to 16 kHz the unlimited output carries 1.5 dB less than the clean speech, at 18 dB 0.9 (all 824 files). The default 18 dB still adds 0.93 PESQ over the input and 0.50 over `omlsa`. Pass `limit: 0` where the most noise removal matters more than room tone. The level change and the voice guard cost 0.01 PESQ unlimited and 0.004 at 18 dB. The same scoring gives upstream's published anchors: noisy 1.97 and 0.921 (as the DeepFilterNet2 paper's table), DeepFilterNet3 3.16 and 0.945 against the DeepFilterNet3 paper's 3.17 and 0.944. That DeepFilterNet2 table measured RNNoise's 2022 model at 2.33 and 0.922; the current model scores lower here.
 
 - **RNNoise's current model sometimes removes the voice.** STOI drops by more than 0.1 on 68 of 824 files and by more than 0.2 on 12; on p232_354 it goes from 0.870 to 0.154, the output 24 dB under the clean level while RNNoise's own voice probability averages 0.84. Upstream's C does the same, the portable and the NEON build alike (0.154, 0.153). DeepFilterNet3 never loses more than 0.066 STOI, `wiener` 0.065, `omlsa` 0.059. A 20 dB `limit` (the input mixed back at 0.1) caps it: losses over 0.1 fall to 6 files and over 0.2 to none (p232_354 keeps 0.785), and PESQ rises to 2.46, +0.50 ± 0.03 over the noisy input and +0.07 ± 0.02 over `omlsa`, at a BAK of 3.71 instead of 3.85. `denoise()`, the worklet and the atom apply that limit unless given another.
 - **The recipe's delivery stages are not denoisers.** Its compressor, EQ, de-esser and loudness target change the signal against the clean reference on purpose: the full recipe scores under its own denoising part. SI-SDR also falls with the 80 Hz high-pass: on its own it takes clean speech to 6.2 dB SI-SDR at PESQ 4.45 (21 test utterances). The table says what each stage does to speech in noise, not how finished a podcast sounds.
@@ -153,10 +157,14 @@ Over the noisy input, paired, with 95% intervals: DeepFilterNet3 adds 1.20 ± 0.
 | `omlsa()` | 3.46 | 4.09 | 3.19 | −98 to −69 (−75) | 0.0 dB |
 | "Enhance speech" denoising | 3.45 | 4.07 | 3.17 | −101 to −71 (−85) | −0.2 dB |
 | RNNoise, `limit: 0` | 3.42 | 4.14 | 3.19 | −127 to −106 (−112) | −0.5 dB |
-| DeepFilterNet3, `limit: 0` | 3.47 | 4.12 | 3.21 | −152 to −61 (−89) | −0.5 dB |
-| DeepFilterNet3, `limit: 12`, the default | 3.47 | 4.10 | 3.21 | −91 to −57 (−67) | −0.4 dB |
+| DeepFilterNet3, upstream's settings, `limit: 0` | 3.47 | 4.12 | 3.21 | −152 to −61 (−89) | −0.5 dB |
+| DeepFilterNet3, `denoise()`, `limit: 0` | 3.47 | 4.11 | 3.21 | −132 to −60 (−84) | −0.4 dB |
+| DeepFilterNet3, `denoise()`, `limit: 18`, the default | 3.47 | 4.11 | 3.21 | −93 to −59 (−73) | −0.4 dB |
+| DeepFilterNet3, `limit: 12` | 3.47 | 4.10 | 3.21 | −91 to −57 (−67) | −0.4 dB |
 
-These rooms are mostly quiet (raw OVRL 3.17), so the gains are small. OM-LSA keeps SIG and raises BAK a little less than the models (0.3.11's lowered SIG by 0.20); both models raise BAK and keep SIG. Both can also take pauses to dead silence (RNNoise's median floor is −112 dBFS, DeepFilterNet3's lowest −152), which ACX Check flags; a `limit` of 12 dB keeps DeepFilterNet3's floors between −91 and −57 dBFS at the same OVRL, room tone left in. On lena (a film scene with music under the voice) RNNoise lowers OVRL from 2.76 to 2.47, DeepFilterNet3 raises it to 2.83, 2.90 with the limit.
+These rooms are mostly quiet (raw OVRL 3.17), so the gains are small. OM-LSA keeps SIG and raises BAK a little less than the models (0.3.11's lowered SIG by 0.20); both models raise BAK and keep SIG. Both can also take pauses to dead silence (RNNoise's median floor is −112 dBFS, DeepFilterNet3's lowest −152), which ACX Check flags; a `limit` of 18 dB keeps DeepFilterNet3's floors between −93 and −59 dBFS at the same OVRL, room tone left in. On lena (a film scene with music under the voice) RNNoise lowers OVRL from 2.76 to 2.47, DeepFilterNet3 raises it to 2.83, 2.90 with the limit.
+
+**Singing.** VocalSet (Wilkins et al., ISMIR 2018, CC BY 4.0): 40 long tones (straight, forte, pianissimo, messa di voce; 20 singers) and 20 sung phrases (straight and vibrato), alone and in living-room noise (DEMAND, from the VoiceBank+DEMAND test set) at 15 dB SNR; level change of the voice over the frames within 20 dB of the loudest. At the default 18 dB, against 0.1's processing at the same limit (heard as it is, no voice guard): held notes −1.9 dB clean and −3.4 dB in noise, 14% of frames more than 6 dB down, against −14.2 and −16.2 dB, 96% (at 10 dB SNR −4.8 against −15.6); sung phrases −0.9 and −2.0 dB against −10.2 and −14.9; the same singers' spoken phrases −0.3 and −0.8 dB against −1.7 and −3.7 (they are quiet recordings: the −20 dBFS hearing helps them). Upstream's unlimited output takes the held notes 38.7 and 37.4 dB down. The noise in the pauses drops as before (−14.1 dB against −14.3), and a 100 or 120 Hz buzz 20 dB under the voice drops in them as without the guard (−17.2 dB), in quiet and in cafe noise: it is the background, not over it.
 
 To rerun: `node scripts/accuracy.mjs vbdemand SYSTEMS` and `node scripts/accuracy.mjs rooms` write the outputs (the data sources and checksums are in its header), `python scripts/accuracy.py vbdemand SYSTEMS` and `python scripts/accuracy.py rooms` score them, and `scripts/deepfilter-reference.py --vbdemand` writes the Python row; the classical rows come from `@audio/denoise`'s `node scripts/speech.mjs vbdemand|rooms SYSTEMS` and `python scripts/speech.py score vbdemand|rooms SYSTEMS`. Per-file scores stay in `~/.cache/audiojs/data/` (`vbdemand/scores/`, `spoken/scores.json`); the scripts regenerate outputs, skipping any already there.
 
@@ -178,7 +186,7 @@ Apple M4 Max running other jobs (load average 28 to 50 on 14 cores), so wall tim
 
 A 128-sample render quantum lasts 2.67 ms and a frame is due every 3.75 quanta: the slowest SIMD frame measured (0.94 ms) fits a quantum 2.8 times over, the mean 8 times; the JS path's slowest (up to 4.0 ms) does not, and relies on device buffers of 256 samples or more, the browsers' usual size. The live underruns came with the machine oversubscribed, the control underran too; an idle machine is untested. Memory: 3.5 MB of weights (plus a 2.8 MB copy in the kernel's memory), about 70 KB of state per channel; the Node evaluation processes peaked at 83 MB RSS.
 
-**DeepFilterNet3**, onnxruntime-node 1.30, 4 intra-op threads: real-time factor 0.047 in CPU on a 180 s file (8.4 s), 0.087 over the 824 short test utterances (2072 s of audio in 180 s; per-run overhead); Python DeepFilterNet (PyTorch 2.1, 4 threads) 0.067 on the same utterances. Peak RSS 436 MB over the test set, 563 MB for 180 s in 10 s chunks. The browser path (onnxruntime-web, wasm and WebGPU) is untested here.
+**DeepFilterNet3**, onnxruntime-node 1.30, 4 intra-op threads: real-time factor 0.047 in CPU on a 180 s file (8.4 s), 0.087 over the 824 short test utterances (2072 s of audio in 180 s; per-run overhead); Python DeepFilterNet (PyTorch 2.1, 4 threads) 0.067 on the same utterances. Peak RSS 436 MB over the test set, 563 MB for 180 s in 10 s chunks. The voice guard's pitch analysis adds 0.16 s of wall time per minute of audio to the model's 0.6 (one channel at 48 kHz, 4 threads). The browser path (onnxruntime-web, wasm and WebGPU) is untested here.
 
 ## Licenses
 
@@ -204,7 +212,7 @@ The package is BSD-3-Clause (the RNNoise translation and weights); `deepfilter.j
 J.-M. Valin, "A Hybrid DSP/Deep Learning Approach to Real-Time Full-Band Speech Enhancement", MMSP 2018, [arXiv:1709.08243](https://arxiv.org/abs/1709.08243). · H. Schröter, T. Rosenkranz, A. N. Escalante-B., A. Maier, "DeepFilterNet: Perceptually Motivated Real-Time Speech Enhancement", Interspeech 2023, [arXiv:2305.08227](https://arxiv.org/abs/2305.08227). · C. Valentini-Botinhao, "Noisy speech database for training speech enhancement algorithms and TTS models", University of Edinburgh, 2017, [doi:10.7488/ds/2117](https://doi.org/10.7488/ds/2117). · ITU-T P.862.2 (2007), wideband PESQ. · C. H. Taal, R. C. Hendriks, R. Heusdens, J. Jensen, "An Algorithm for Intelligibility Prediction of Time-Frequency Weighted Noisy Speech", IEEE TASLP 19(7), 2011. · J. Le Roux, S. Wisdom, H. Erdogan, J. R. Hershey, "SDR: Half-baked or Well Done?", ICASSP 2019. · C. K. A. Reddy, V. Gopal, R. Cutler, "DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors", ICASSP 2022. · S. A. Figueroa, "When is double rounding innocuous?", SIGNUM Newsletter 30(3), 1995. · [xiph/rnnoise](https://gitlab.xiph.org/xiph/rnnoise) at 70f1d25 · [Rikorose/DeepFilterNet](https://github.com/Rikorose/DeepFilterNet) at d375b2d.
 
 **Use when:** speech with noise a statistical denoiser leaves or smears (moving noise, a busy room); live voice in a browser (RNNoise, 30 ms); files where quality is worth an 8 MB download (DeepFilterNet3).<br>
-**Not for:** music or singing (both models keep speech and drop the rest); installs that must stay model-free ([`@audio/denoise`](https://github.com/audiojs/denoise)); reverb as such (`dereverb`); live DeepFilterNet3 (upstream's graphs take no state).
+**Not for:** music (both models keep speech and drop the rest; DeepFilterNet3's voice guard keeps held sung notes, RNNoise drops them); installs that must stay model-free ([`@audio/denoise`](https://github.com/audiojs/denoise)); reverb as such (`dereverb`); live DeepFilterNet3 (upstream's graphs take no state).
 
 ---
 

package/deepfilter.js

@@ -15,20 +15,26 @@
 // activations per second of audio. Each chunk's run starts `warmup` frames early from zero GRU state
 // (those outputs dropped) and continues `fade` frames past its end, where the next chunk crossfades in
 // (raised cosine). The GRUs keep a long memory of their start, so chunked output is another valid
-// trajectory, not the whole-file one; the STFT and the features stay exact across chunks.
+// trajectory, not the whole-file one (40 to 55 dB SNR from it on a 60 s narration; longer warm-ups don't
+// converge); the STFT and the features stay exact across chunks.
+//
+// Added, for denoise(): the network can hear its input at another level (`gain`, its features only: they are
+// not level-free, the ERB means start at fixed dB and the complex features scale with the root of the level),
+// and a voice guard (`voice`) keeps the sustained voicing it takes for noise: held sung notes, chant.
 
 import { load, tensor, fetchModel } from '@audio/neural-runtime'
 import { WINDOW, rfft, irfft, N as FFT_N, HOP as FFT_HOP, BINS } from './fft.js'
+import { pitch } from './rnnoise.js'
 
 const f = Math.fround
 
 // models/DeepFilterNet3_onnx.tar.gz at d375b2d (7,983,136 bytes, sha256 c94d91f7…): enc, erb_dec, df_dec, config.ini
 export const MODEL = 'https://raw.githubusercontent.com/Rikorose/DeepFilterNet/d375b2d8309e0935d165700c91da9de862a99c31/models/DeepFilterNet3_onnx.tar.gz'
 
-// The attenuation limit, dB, that denoise() applies to this model unless given one (0: none, upstream's
-// output). Unlimited, it takes the pauses of ten home narrations as low as -152 dBFS, the digital silence
-// ACX Check flags; limited to 12 dB, they keep -91 to -57 dBFS at the same DNSMOS (README, Accuracy).
-export const LIMIT = 12
+// The attenuation limit, dB, that denoise() applies to this model unless given one (0: none): the most before the
+// voice itself suffers. On VoiceBank+DEMAND, DNSMOS SIG holds from 12 to 18 dB and falls past it, BAK and PESQ
+// rise; unlimited, the pauses of home narrations fall to digital silence (README, Accuracy).
+export const LIMIT = 18
 
 const isNode = typeof process !== 'undefined' && !!process.versions?.node
 
@@ -114,11 +120,41 @@ const linspace = (a, b, n) => { let step = f(f(f(b) - f(a)) / (n - 1)); return F
 
 // ------------------------------------------------ enhance
 
-// enhance(x, net, opts) → Float32Array: one channel at 48 kHz, as df.enhance.enhance(model, state, x, pad=True)
-export async function enhance(x, net, { limit, chunk = 1000, warmup = 300, fade = 50 } = {}) {
+// voicing(x, T, hop) → { period, on }: per frame, RNNoise's pitch period over the frame's 20 ms window, and `on`
+// inside sustained voicing: runs of 0.3 s or more whose normalized correlation at the period is 0.45 or more
+// (Praat's voicing threshold, Boersma 1993) and whose 100 ms stand 6 dB or more above the quietest 100 ms of the
+// 20 s around (minimum statistics, Martin 2001: that minimum lies 4 to 11 dB under the mean of the DEMAND
+// noises), digital silence (under -90 dBFS) left out. A held note stands over the background; a buzz is in it.
+export function voicing(x, T, hop) {
+	let L = x.length, pt = pitch(), frame = new Float32Array(hop), period = new Int16Array(T), on = new Uint8Array(T)
+	// power per 100 ms of input, and the least of it over the 20 s around (blocks of digital silence left out)
+	let K = Math.ceil(T / 10), e = new Float64Array(K), low = new Float64Array(K).fill(Infinity)
+	for (let i = 0; i < L; i++) e[i / (10 * hop) | 0] += x[i] * x[i]
+	for (let k = 0; k < K; k++) e[k] /= Math.max(1, Math.min(L, (k + 1) * 10 * hop) - k * 10 * hop)
+	for (let k = 0; k < K; k++) if (e[k] > 1e-9) for (let j = Math.max(0, k - 100); j <= Math.min(K - 1, k + 100); j++) low[j] = Math.min(low[j], e[k])
+	// the pitch searched every 20 ms where the frame stands over the background (every 10 ms keeps held notes no better)
+	for (let t = 0; t < T; t++) {
+		for (let i = 0, j = t * hop; i < hop; i++, j++) frame[i] = j < L ? x[j] * 32768 : 0
+		let over = e[t / 10 | 0] >= 4 * low[t / 10 | 0]
+		period[t] = pt.push(frame, over && !(t & 1))
+		on[t] = over && pt.gain >= .45
+	}
+	for (let t = 0, r = 0; t <= T; t++) {
+		if (t < T && on[t]) { r++; continue }
+		if (r < 30) on.fill(0, t - r, t)
+		r = 0
+	}
+	return { period, on }
+}
+
+// enhance(x, net, opts) → Float32Array: one channel at 48 kHz, as df.enhance.enhance(model, state, x, pad=True).
+// `gain` scales the input the network hears (its features), not the spectrum its gains and filters apply to;
+// `voice` keeps sustained voicing (voicing()) the network would remove.
+export async function enhance(x, net, { limit, chunk = 1000, warmup = 300, fade = 50, gain = 1, voice = false } = {}) {
 	let { hop, n, nbErb, nbDf, order, dfLa, convLa, widths, alpha } = net.params
 	let L = x.length, T = Math.floor((L + n) / hop), lead = order - 1 - dfLa, look = Math.max(convLa, dfLa)
-	let lim = limit != null && Math.abs(limit) > 0 ? 10 ** (-Math.abs(limit) / 20) : 0
+	let lim = limit != null && Math.abs(limit) > 0 ? 10 ** (-Math.abs(limit) / 20) : 0, g1 = f(gain), g2 = f(gain * gain)
+	let vo = voice && voicing(x, T, hop)
 	// per-frame store, a ring big enough for one chunk with its context
 	chunk = Math.max(1, Math.min(chunk, T)); fade = Math.min(fade, chunk)
 	let cap = chunk + fade + warmup + look + lead + 2
@@ -138,12 +174,12 @@ export async function enhance(x, net, { limit, chunk = 1000, warmup = 300, fade
 			for (let b = 0; b < nbErb; b++) {
 				let s = 0, k = erbK[b]
 				for (let j = erbStart[b], e = j + widths[b]; j < e; j++) s = f(s + f(f(f(Sr[o + j] * Sr[o + j]) + f(Si[o + j] * Si[o + j])) * k))
-				let v = f(f(Math.log10(f(s + f(1e-10)))) * 10)
+				let v = f(f(Math.log10(f(f(s * g2) + f(1e-10)))) * 10)
 				mean[b] = f(f(v * a1) + f(mean[b] * alpha))
 				FE[slot * nbErb + b] = f(f(v - mean[b]) / 40)
 			}
 			for (let j = 0; j < nbDf; j++) {
-				let re = Sr[o + j], im = Si[o + j]
+				let re = f(Sr[o + j] * g1), im = f(Si[o + j] * g1)
 				unit[j] = f(f(f(Math.hypot(re, im)) * a1) + f(unit[j] * alpha))
 				let d = f(Math.sqrt(unit[j]))
 				FC[slot * nbDf * 2 + j] = f(re / d); FC[slot * nbDf * 2 + nbDf + j] = f(im / d)
@@ -151,6 +187,35 @@ export async function enhance(x, net, { limit, chunk = 1000, warmup = 300, fade
 		}
 	}
 
+	// sustained voicing: a band holding a harmonic of the pitch keeps at least c² of its input, c its normalized
+	// correlation with the frame one period earlier and two (the less): the periodic share of its power, as RNNoise's
+	// pitch filter restores what its band gains remove (Valin 2018). Er, Ei is the output so far.
+	let P = [[new Float32Array(BINS), new Float32Array(BINS)], [new Float32Array(BINS), new Float32Array(BINS)]]
+	function keep(t, o) {
+		let T0 = vo.period[t], f0 = n / T0
+		for (let k = 0; k < 2; k++) {
+			for (let i = 0, j = (t - 1) * hop - (k + 1) * T0; i < n; i++, j++) buf[i] = f((j >= 0 && j < L ? x[j] : 0) * WINDOW[i])
+			rfft(buf, P[k][0], P[k][1])
+		}
+		for (let b = 0; b < nbErb; b++) {
+			let j0 = erbStart[b], j1 = j0 + widths[b]
+			// h·f0, h ≥ 1, within the band's bins (sr/n apart)
+			if (Math.floor((j1 - .5) / f0) < Math.max(1, Math.ceil((j0 - .5) / f0))) continue
+			let xx = 0, ee = 0, c = 1
+			for (let j = j0; j < j1; j++) { xx += Sr[o + j] ** 2 + Si[o + j] ** 2; ee += Er[j] ** 2 + Ei[j] ** 2 }
+			for (let [Pr, Pi] of P) {
+				let xy = 0, yy = 0
+				for (let j = j0; j < j1; j++) { xy += Sr[o + j] * Pr[j] + Si[o + j] * Pi[j]; yy += Pr[j] ** 2 + Pi[j] ** 2 }
+				c = Math.min(c, xy / Math.sqrt(xx * yy + 1e-30))
+			}
+			let p = c > 0 ? c * c : 0, g = Math.min(1, Math.sqrt(ee / (xx + 1e-30)))
+			if (p <= g) continue
+			// mixing a of the input in takes the band's gain from g to p
+			let a = (p - g) / (1 - g)
+			for (let j = j0; j < j1; j++) { Er[j] = f(f(Sr[o + j] * a) + f(Er[j] * (1 - a))); Ei[j] = f(f(Si[o + j] * a) + f(Ei[j] * (1 - a))) }
+		}
+	}
+
 	let out = new Float32Array(L), syn = new Float32Array(hop), y = new Float32Array(n), d = n - hop
 	let Er = new Float32Array(BINS), Ei = new Float32Array(BINS)
 	// a chunk's run continues `fade` frames past its end; the next chunk crossfades from those spectra
@@ -191,6 +256,7 @@ export async function enhance(x, net, { limit, chunk = 1000, warmup = 300, fade
 				}
 			}
 			if (lim) for (let j = 0; j < BINS; j++) { Er[j] = f(f(Sr[o + j] * lim) + f(Er[j] * (1 - lim))); Ei[j] = f(f(Si[o + j] * lim) + f(Ei[j] * (1 - lim))) }
+			if (vo && vo.on[t]) keep(t, o)
 			if (t >= b) { tailR.set(Er, (t - b) * BINS); tailI.set(Ei, (t - b) * BINS); continue }
 			if (a > 0 && t - a < fade) {
 				let w = .5 - .5 * Math.cos(Math.PI * (t - a + .5) / fade), q = (t - a) * BINS

package/denoise.js

@@ -11,6 +11,9 @@ export { create as rnnoise, FRAME } from './rnnoise.js'
 export { MODEL } from './deepfilter.js'
 
 const RATE = 48000
+// DeepFilterNet3 hears its input with the speech at -20 dBFS, the median of the VoiceBank+DEMAND test set it scores
+// its published PESQ on (level(), -20.2): its features are not level-free (README, DeepFilterNet3)
+const LEVEL = -20
 // the attenuation limit (dB) each model gets when opts.limit is not given; 0 is upstream's output
 const LIMITS = { rnnoise: RNNOISE_LIMIT, deepfilternet3: DFN_LIMIT }
 
@@ -51,7 +54,8 @@ export default async function denoise(audio, opts = {}) {
 		let out = [], limit = opts.limit ?? LIMITS[handle.model]
 		for (let ch of channels) {
 			let x = rate === RATE ? Float32Array.from(ch) : resample(Float32Array.from(ch), { from: rate, to: RATE })
-			let y = handle.model === 'rnnoise' ? rnnoiseOffline(x, handle.net, limit) : await enhance(x, handle.net, { ...opts, limit })
+			let y = handle.model === 'rnnoise' ? rnnoiseOffline(x, handle.net, limit)
+				: await enhance(x, handle.net, { ...opts, limit, gain: 10 ** ((LEVEL - level(x)) / 20), voice: true })
 			out.push(rate === RATE ? y : fit(resample(y, { from: RATE, to: rate }), ch.length))
 		}
 		return audio instanceof Float32Array ? out[0] : Array.isArray(audio) ? out : { ...audio, channelData: out, sampleRate: rate }
@@ -60,6 +64,16 @@ export default async function denoise(audio, opts = {}) {
 	}
 }
 
+// speech level, dBFS: mean power of the louder half of 50 ms frames (LEVEL where there is none)
+function level(x) {
+	let F = 2400, p = []
+	for (let i = 0; i + F <= x.length; i += F) { let s = 0; for (let j = i; j < i + F; j++) s += x[j] * x[j]; p.push(s / F) }
+	p.sort((a, b) => b - a)
+	let k = Math.max(1, p.length >> 1), s = 0
+	for (let i = 0; i < k && i < p.length; i++) s += p[i]
+	return s > 0 ? 10 * Math.log10(s / k) : LEVEL
+}
+
 const fit = (y, n) => y.length === n ? y : (() => { let o = new Float32Array(n); o.set(y.subarray(0, n)); return o })()
 
 // One channel through RNNoise at int16 scale, delay removed: out[i] lines up with x[i]

package/index.d.ts

@@ -28,7 +28,7 @@ export interface DenoiseOptions extends LoadOptions {
   sampleRate?: number
   /** 'rnnoise' (default), 'deepfilternet3', or a handle from load() to reuse across calls */
   model?: 'rnnoise' | 'deepfilternet3' | Model
-  /** attenuation limit in dB: mixes the input back in at 10^(−limit/20); default 20 for RNNoise, 12 for DeepFilterNet3 (README, API); 0 for none, upstream's output */
+  /** attenuation limit in dB, the most the noise drops: mixes the input back in at 10^(−limit/20); default 20 for RNNoise, 18 for DeepFilterNet3 (README, API); 0 for none. DeepFilterNet3 also hears its input with the speech at −20 dBFS and keeps sustained voicing it would remove (held sung notes) */
   limit?: number
   /** DeepFilterNet3: frames (10 ms) per model run, default 1000 */
   chunk?: number

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@audio/neural-denoise",
-  "version": "0.1.0",
+  "version": "0.2.0",
   "description": "Neural speech enhancement: RNNoise (Valin 2018) in pure JS, bit-exact to its C, weights bundled; DeepFilterNet3 through ONNX Runtime, matching the Python original",
   "type": "module",
   "sideEffects": false,

package/rnnoise.js

@@ -489,6 +489,22 @@ function removeDoubling(x, T0, prevPeriod, prevGain) {
 	return T0 < PMIN ? PMIN : T0
 }
 
+// pitch() → { push(frame, search = true) → period, gain, buf }: one 10 ms frame at 48 kHz, int16 scale, as
+// rnn_compute_frame_features searches it over the last 36 ms (PITCH_BUF_SIZE, held in `buf`): the period in
+// samples (60 to 768), `gain` the normalized correlation at it; search false only takes the frame in
+export function pitch() {
+	let buf = new Float32Array(PBUF), xlp = new Float32Array(PBUF >> 1), track = { buf, period: 0, gain: 0 }
+	track.push = (x, search = true) => {
+		buf.copyWithin(0, FRAME); buf.set(x, PBUF - FRAME)
+		if (!search) return track.period
+		pitchDownsample(buf, xlp, PBUF)
+		track.period = removeDoubling(xlp, PMAX - pitchSearch(xlp, PMAX >> 1, xlp, PFRAME, PMAX - 3 * PMIN), track.period, track.gain)
+		track.gain = PG[0]
+		return track.period
+	}
+	return track
+}
+
 // ------------------------------------------------ denoiser state (DenoiseState)
 
 // create(weights) → { process(input480, output480) → VAD probability }: one 10 ms frame at 48 kHz,
@@ -496,8 +512,7 @@ function removeDoubling(x, T0, prevPeriod, prevGain) {
 export function create(net) {
 	if (!net || !net.gru1) net = model(net)
 	let analysisMem = new Float32Array(FRAME), synthesisMem = new Float32Array(FRAME)
-	let pitchBuf = new Float32Array(PBUF), memHp = new Float64Array(2), lastg = new Float32Array(NB)
-	let lastPeriod = 0, lastGain = 0
+	let pt = pitch(), pitchBuf = pt.buf, memHp = new Float64Array(2), lastg = new Float32Array(NB)
 	let dXr = new Float32Array(FREQ), dXi = new Float32Array(FREQ), dPr = new Float32Array(FREQ), dPi = new Float32Array(FREQ)
 	let dEx = new Float32Array(NB), dEp = new Float32Array(NB), dExp = new Float32Array(NB)
 	// RNNState
@@ -508,7 +523,7 @@ export function create(net) {
 	let Xr = new Float32Array(FREQ), Xi = new Float32Array(FREQ), Pr = new Float32Array(FREQ), Pi = new Float32Array(FREQ)
 	let Ex = new Float32Array(NB), Ep = new Float32Array(NB), Exp = new Float32Array(NB), Ly = new Float32Array(NB)
 	let feat = new Float32Array(NF), g = new Float32Array(NB), gf = new Float32Array(FREQ)
-	let xlp = new Float32Array(PBUF >> 1), vad = new Float32Array(1)
+	let vad = new Float32Array(1)
 	let t1 = new Float32Array(128), tmp = new Float32Array(1536), cat = new Float32Array(1536)
 	let zrh = new Float32Array(1152), recur = new Float32Array(1152)
 
@@ -521,10 +536,7 @@ export function create(net) {
 	function features(x) {
 		buf.set(analysisMem); buf.set(x, FRAME); analysisMem.set(x)
 		window(buf); rfft(buf, Xr, Xi); bandEnergy(Ex, Xr, Xi)
-		pitchBuf.copyWithin(0, FRAME); pitchBuf.set(x, PBUF - FRAME)
-		pitchDownsample(pitchBuf, xlp, PBUF)
-		let idx = removeDoubling(xlp, PMAX - pitchSearch(xlp, PMAX >> 1, xlp, PFRAME, PMAX - 3 * PMIN), lastPeriod, lastGain)
-		lastPeriod = idx; lastGain = PG[0]
+		let idx = pt.push(x)
 		for (let i = 0; i < WIN; i++) buf[i] = pitchBuf[PBUF - WIN - idx + i]
 		window(buf); rfft(buf, Pr, Pi)
 		bandEnergy(Ep, Pr, Pi); bandEnergy(Exp, Xr, Xi, Pr, Pi)