@audio/neural-denoise 0.5.0 → 0.6.1

package/README.md

@@ -117,6 +117,8 @@ Both models enhance speech, and to them everything else is noise: at their defau
 3. **Streaming** (RNNoise: `denoise()`, the contract atom): a stream decides on what has arrived. The forward recursion of the same chain for two states (music, the rest) keeps the log-odds of music, bounded at ±20 nats; a patch moves them by at most ln 100, and by 1 nat less (a prior for speech), and the stream switches to music above 5 and back below 0. Each output frame takes the decision made once the input frame two on is in (RNNoise's own delay), so nothing is added to the latency, and `denoise()` and the atom give the same samples. The price: music is denoised for its first second or so, and each switch comes about a second late.
 4. **The gain** between the enhanced signal and the input moves over 200 ms, raised cosine: offline centered on the segment boundary, streaming from the decision on. It is applied at the input's rate (the 48 kHz gain resampled with the signal), so what passes is the input itself, not its round trip through 48 kHz.
 
+`guard(x, sampleRate)` gives that offline gain for any enhancement, per sample of `x` at its own rate: 1 where it is speech or noise, 0 where it is music, ramped as above; `g·y + (1 − g)·x` passes music untouched. `audio`'s `dereverb` guards its de-reverb model by it.
+
 It runs in JS: a stream's `process()` can't await ONNX Runtime, and RNNoise needs no runtime. The three upper convolutions (through im2col) and the dense layers run in a 1,074-byte WebAssembly SIMD kernel (`gemm.wat`), the first convolution once per frame and shared by the overlapping patches (the max pooling after it commutes with each patch's standardization); 1.1 s of CPU per minute of audio, 11 s on the JS path where WebAssembly SIMD is missing (same bits).
 
 **Accuracy**, on labelled sets (`python scripts/accuracy.py guard-sets`, `node scripts/accuracy.mjs guard train|test`): the share of active 10 ms frames passed (within 40 dB of the loudest). The constants (one patch per 100 ms, the streaming bound, prior and hysteresis) were chosen on the training sets: VoiceBank+DEMAND's training speakers, the MUSDB18 training previews, Slakh2100 mixes 1 to 10, VocalSet singers 1 to 5, the Spoken Wikipedia narrations not in Rooms; the test sets below were run once. Beds: clean utterances over a song's accompaniment (the MUSDB18 stems but the vocals) 20, 15 and 10 dB under the speech; hiss: white and pink Gaussian noise at 20, 10 and 5 dB SNR. Programmes: 8 sequences of six segments, speech (five utterances of one kind, or 15 s of narration) and music (up to 20 s of a song, its accompaniment or a Slakh mix) in turn, scored 0.5 s or more from each cut.
@@ -300,7 +302,7 @@ What 0.5 changed, and where RX still wins:
 
 - **The floor.** 18 dB of attenuation leaves noise as loud as the voice 18 dB under it, and RX takes it all: before 0.5 RX won every noise row. Unlimited, the model beat RX already (tune split, −5 dB: PESQ 2.12 against 1.98); the floor keeps that where the noise stands near the voice and the 18 dB limit's room tone where it is quiet. +0.38, +0.36, +0.26 PESQ at −5, 0, 5 dB over 0.4.
 - **The guard's prior.** Speech under a bed as loud as itself reached ina as music, so 0.4 passed half of those takes untouched (PESQ 1.26). With speech favoured by 1 nat a patch, as the stream already decided, 37% of their frames pass (Music): PESQ 1.63 against RX's 1.65, OVRL 2.33 against 2.74. What still passes is a voice ina cannot tell from a singer at that level: the model keeps the voice of a song as it keeps this one, 4 to 7 dB under the mix, and ina hears speech in both, so passing less would gut songs (tune split: a voice under a bed at its level kept −3.5 to −5.9 dB and heard as speech at 0.81 or more; the training songs, at most −4.6 dB and 1.00). `music: 'enhance'` enhances it all: PESQ 1.87, OVRL 2.91, SI-SDR 11.0 dB, ahead of RX on all three, and music alone then loses 28.5 dB (RX 37.6).
-- **Rooms.** RX's reverb gain takes the late reverberation (SI-SDR 7.2 dB to the direct sound and 50 ms, against ours 5.7; at its default, reverb kept, 5.2). DeepFilterNet3 trained with reverberant speech in 10% of its mixtures (its `config.ini`, `p_reverb`), and takes part of it; `audio`'s `dereverb()` before it takes more.
+- **Rooms.** RX's reverb gain takes the late reverberation (SI-SDR 7.2 dB to the direct sound and 50 ms, against ours 5.7; at its default, reverb kept, 5.2). DeepFilterNet3 trained with reverberant speech in 10% of its mixtures (its `config.ini`, `p_reverb`), and takes part of it. `audio`'s `deepfilter()` now runs it again where its output still carries a room (`@audio/denoise-dereverb` 0.6's checks, asked of the output and of the take), on the take with the room's linear prediction taken off (that package's WPE alone): 81 of the 103 room takes, PESQ 1.81, OVRL 2.44, SI-SDR 6.2 dB (RX tuned 1.80, 2.51, 7.2; paired +0.01 ± 0.05 PESQ, −0.08 ± 0.07 OVRL); the music bed at 0 dB 1.67 / 2.44, 13 takes heard again; the other rows, clean speech, music alone, VoiceBank+DEMAND (2 of 824) and the narrations (2 of 10) as above. Chosen on the tune split, reverb5: WPE before the model PESQ 1.88 → 1.96, SI-SDR 7.3 → 7.9; `dereverb()` whole after the model 1.89, before it 1.87 (OVRL −0.09); MossFormer2_SE_48K (ClearerVoice-Studio, Apache-2.0, trained with reverberant speech in 30% of its mixtures; its 221 MB PyTorch checkpoint) 1.80, OVRL 2.35, SI-SDR 6.3. Still behind RX: OVRL and SI-SDR in rooms, the late tail the model and the linear prediction leave.
 
 To rerun: in `audio`, `node bench/rx/isolate.mjs test all rx-di rx-di-tuned dfn` (renders kept in `~/.cache/audiojs/data/rx/isolate/`), `node bench/speech.mjs test rx-di,rx-di-tuned,dfn-18` then `python bench/speech.py test rx-di,rx-di-tuned,dfn-18`; tune with `tune` in place of `test`.
 

package/audio.js

@@ -6,7 +6,7 @@
 // quanta only), so the delay is constant: 960 + 479 = 1439 samples at 48 kHz, 30 ms. Other rates go to
 // 48 kHz and back through a streaming form of @audio/resample-sinc's Lanczos kernel, the resampler denoise()
 // uses offline; its lookahead adds to the delay (1354 samples at 44.1 kHz, 30.7 ms). The output is
-// denoise()'s: at 48 kHz sample for sample; at other rates, with resample-sinc 1.2.0, except within 50 ms of
+// denoise()'s: at 48 kHz sample for sample; at other rates, with resample-sinc 1.3.0, except within 50 ms of
 // the end, where the offline resampler sees the end and a stream sees the host's trailing silence.
 //
 // limit (dB): the input is mixed back in at 10^(−limit/20), so noise drops by at most `limit`; 0 lifts it
@@ -25,6 +25,7 @@
 // Channels are denoised independently; a channel's state (70 KB, and the guard's 0.3 MB) is created on its first
 // block, as the worklet does, since hosts declare up to 32 channels.
 
+import { phases } from '@audio/resample-sinc'
 import { model, create, FRAME, LIMIT } from './rnnoise.js'
 import { weights } from './denoise.js'
 import { guardNet, online, ramp, blend } from './guard.js'
@@ -47,18 +48,34 @@ const latency = sr => sr === RATE ? DELAY + PRIME : Math.floor((taps(RATE, sr) +
 
 /**
  * @audio/resample-sinc's resample() (Lanczos, a = 16) as a stream, in its arithmetic, so its samples: `push(x)`
- * calls `emit(v)` for every output whose taps have all arrived. Output i sits at input position i·from/to;
- * taps before the start are left out and the weights renormalized, as resample() does at its edges.
+ * calls `emit(v)` for every output whose taps have all arrived. Output i sits at input position i·from/to; at integer
+ * rates it reads resample()'s phase table, at others it weighs by angle addition as resample() does there; taps before
+ * the start are left out and the weights renormalized, as resample() does at its edges.
  */
 function resampler(from, to, emit) {
 	let rate = from / to, scale = rate > 1 ? 1 / rate : 1, T = Math.ceil(HALF / scale), K = HALF / (Math.PI * Math.PI)
+	// input history: an output reads 2T samples, the newest of them just arrived
+	let mask = (1 << Math.ceil(Math.log2(2 * T + 1))) - 1, ring = new Float32Array(mask + 1), n = 0, i = 0
+	let tab = phases(from, to)
+	if (tab) {
+		let { L, M, W, J0, J1 } = tab, S = 2 * T, p = 0, base = 0
+		return x => {
+			for (let s = 0; s < x.length; s++) {
+				ring[n++ & mask] = x[s]
+				while (base + T < n) {
+					let lo = base + 1 - T, o = p * S - lo, sum = 0, w = 0
+					for (let j = Math.max(0, lo + J0[p]), e = lo + J1[p]; j < e; j++) { let k = W[o + j]; sum += ring[j & mask] * k; w += k }
+					emit(Math.fround(w !== 0 ? sum / w : 0))
+					if ((p += M) >= L) { let q = Math.floor(p / L); base += q; p -= q * L }
+				}
+			}
+		}
+	}
 	let A = new Float64Array(2 * T), B = new Float64Array(2 * T), C = new Float64Array(2 * T), D = new Float64Array(2 * T)
 	for (let j = 0, t = 1 - T; t <= T; t++, j++) {
 		let a = Math.PI * scale * t
 		A[j] = Math.sin(a); B[j] = Math.cos(a); C[j] = Math.sin(a / HALF); D[j] = Math.cos(a / HALF)
 	}
-	// input history: an output reads 2T samples, the newest of them just arrived
-	let mask = (1 << Math.ceil(Math.log2(2 * T + 1))) - 1, ring = new Float32Array(mask + 1), n = 0, i = 0
 	return x => {
 		for (let s = 0; s < x.length; s++) {
 			ring[n++ & mask] = x[s]

package/deepfilter.js

@@ -58,7 +58,7 @@ export const LEVEL = -20
 // the gain (README, Accuracy).
 export const FILL = -20
 
-const isNode = typeof process !== 'undefined' && !!process.versions?.node
+const isNode = globalThis.process?.release?.name === 'node'  // Node, Bun, Deno: not a process polyfill (esm.sh's, a bundler's), whose versions.node is set
 
 // ------------------------------------------------ model archive
 

package/denoise.js

@@ -103,6 +103,18 @@ export function mixback(x, y, { limit = DFN_LIMIT, floor = FLOOR, sampleRate = R
 	return out
 }
 
+/**
+ * guard(x, sampleRate) → per sample of x, the gain an enhancement of it takes: 1 where it is speech or noise, 0 where it
+ * is music, ramped over RAMP at 48 kHz (200 ms) centered on each change: ina's segments over the whole input, as
+ * denoise() decides for DeepFilterNet3 (guard.js offline()). Blended g·y + (1 − g)·x, a model's output y passes music
+ * untouched; a host guards any enhancement so (audio's dereverb).
+ */
+export async function guard(x, sampleRate = RATE) {
+	let nn = await guardNet(), x48 = sampleRate === RATE ? x : resample(Float32Array.from(x), { from: sampleRate, to: RATE })
+	let g = gains(offline(x48, nn), x48.length, true)
+	return sampleRate === RATE ? g : fit(resample(g, { from: RATE, to: sampleRate }), x.length)
+}
+
 // the enhanced y where the gain g (at 48 kHz) is 1, the input x where it is 0: music passes untouched
 function mix(y, x, g, rate) {
 	if (rate !== RATE) g = fit(resample(g, { from: RATE, to: rate }), x.length)

package/index.d.ts

@@ -48,6 +48,9 @@ export default function denoise<T extends Audio>(audio: T, opts?: DenoiseOptions
 /** The input x mixed back into a model's output y, y + a·(x − y): the noise drops by `limit` dB (default 18), or to `floor` dB under the voice (default 40) where it stands higher, per 10 ms frame; `limit` 0: y; `floor` 0: a constant 10^(−limit/20). denoise() applies it to DeepFilterNet3. */
 export function mixback(x: Float32Array, y: Float32Array, opts?: { limit?: number, floor?: number, sampleRate?: number }): Float32Array
 
+/** Per sample of x, the gain an enhancement of it takes: 1 where it is speech or noise, 0 where it is music (inaSpeechSegmenter's segments over the whole input, as denoise() decides), ramped over 200 ms; blend g·y + (1 − g)·x. */
+export function guard(x: Float32Array, sampleRate?: number): Promise<Float32Array>
+
 /** Load a model once to reuse it across denoise() calls. */
 export function load(model?: 'rnnoise' | 'deepfilternet3', opts?: LoadOptions): Promise<Model>
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@audio/neural-denoise",
-  "version": "0.5.0",
+  "version": "0.6.1",
   "description": "Neural speech enhancement: RNNoise (Valin 2018) in pure JS, bit-exact to its C, weights bundled; DeepFilterNet3 through ONNX Runtime, matching the Python original; music passes through untouched (inaSpeechSegmenter's speech/music/noise CNN)",
   "type": "module",
   "sideEffects": false,
@@ -38,8 +38,8 @@
     "audio.d.ts"
   ],
   "dependencies": {
-    "@audio/neural-runtime": "^0.1.0",
-    "@audio/resample-sinc": "^1.2.0"
+    "@audio/neural-runtime": "^0.2.0",
+    "@audio/resample-sinc": "^1.3.0"
   },
   "devDependencies": {
     "audio-lena": "^3.0.1",