@audio/denoise 0.6.0 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +141 -80
- package/index.d.ts +1 -1
- package/package.json +10 -10
package/README.md
CHANGED
|
@@ -15,13 +15,13 @@ Single-pass noise reduction. 13 specialised methods + an auto-classifier.
|
|
|
15
15
|
| [declick](#declick) | time | impulses | ★★★★ | medium | vinyl ticks, edit clicks, mouth clicks |
|
|
16
16
|
| [decrackle](#decrackle) | time | dense impulses | ★★★ | medium | shellac crackle |
|
|
17
17
|
| [declip](#declip) | time | hard clipping | ★★★★ | medium | clipped recordings, rails found by itself |
|
|
18
|
-
| [dewind](#dewind) |
|
|
18
|
+
| [dewind](#dewind) | freq | wind, LF rumble | ★★★ | medium | wind under a voice, its harmonics kept |
|
|
19
19
|
| [deplosive](#deplosive) | time | LF bursts | ★★★ | low | mic plosives (p, b) |
|
|
20
20
|
| [deesser](#deesser) | time | sibilance | ★★★★ | low | voice (s, sh) |
|
|
21
21
|
| [debreath](#debreath) | time | inter-word noise | ★★★ | low | breath / hiss in pauses |
|
|
22
22
|
| [desilence](#desilence) | time | pauses | ★★★ | low | remove / shorten / trim silence, split by pause |
|
|
23
23
|
| [dereverb](#dereverb) | freq | late reverb | ★★★ | high | speech in a room |
|
|
24
|
-
| [dewow](#dewow) | time + freq | pitch drift | ★★★ | medium |
|
|
24
|
+
| [dewow](#dewow) | time + freq | pitch drift | ★★★ | medium | disc wow (33⅓ / 45 / 78 rpm) in music; tape by its pilot tone |
|
|
25
25
|
|
|
26
26
|
For broader DSP needs use [stretch](https://github.com/audiojs/stretch), [shift](https://github.com/audiojs/shift), [pitch](https://github.com/audiojs/pitch), [beat](https://github.com/audiojs/beat).
|
|
27
27
|
|
|
@@ -153,46 +153,78 @@ The same mixtures through 0.2.0 (its Q 30 notches at the lines it found, run on
|
|
|
153
153
|
|
|
154
154
|
### `dewind`
|
|
155
155
|
|
|
156
|
-
|
|
156
|
+
Takes wind out from under a voice or an instrument and leaves their harmonics. Wind is turbulence at the microphone: noise under a few hundred Hz with no period, in gusts (Nelke & Vary, IWAENC 2014), where a voice's low end is a row of harmonics. A high-pass can only take everything under its cutoff, the voice's low harmonics with the wind; here each STFT bin under `cutoff` is weighed against a wind spectrum read from the frame itself.
|
|
157
|
+
|
|
158
|
+
The frame is the power of two nearest 85 ms (4096 samples at 44.1 and 48 kHz), so a 100 Hz voice's harmonics stand 8 bins apart with valleys between them. The wind spectrum is the periodogram's morphological opening over 5 bins: every peak narrower than that is cut, and a steady harmonic under a Hann window is 4 bins wide, so the harmonics go and the broad wind stays (the inverse harmonic mask of Nelke, Naylor & Vary, ICASSP 2015, without a pitch track). Over 100 Hz it is held to 4× the least it has been over the last 1.5 s relative to its level under 100 Hz (minimum statistics, Martin 2001): wind keeps its shape while it gusts, and a voice's valleys, onsets and unvoiced sounds don't pass for it. The gain is OM-LSA's form (Cohen & Berdugo 2001): a Wiener gain on the decision-directed a priori SNR, raised to the speech presence probability, which takes a fixed 15 dB prior (Gerkmann & Hendriks 2012) and an a priori absence of 0.2 on a harmonic (a peak 6 dB over the wind), 0.9 elsewhere. So a harmonic keeps what of it stands over the wind, and the wind's own random peaks don't come through as musical noise.
|
|
159
|
+
|
|
160
|
+
It runs while wind blows: the 20–300 Hz band aperiodic (its normalized autocorrelation, taken through the frame's spectrum over the window's, under ½ at 2.5–25 ms; a harmonic H in noise N reads H / (H + N), Boersma 1993) and its noise over the 300–2000 Hz band, floored 20 dB under its peak over the last seconds, three frames in a row (a 150 ms gap between words shows it); then held 1 s, through the words, whose low end hides it. A bass line, a kick drum's body or a voice's low end repeats; a room's quiet rumble in a pause is no wind next to a voice. With no wind the output equals the input, sample for sample, `frameSize − 1` samples later (85 ms at 48 kHz, the manifest's declared latency).
|
|
157
161
|
|
|
158
162
|
```js
|
|
159
|
-
dewind(data, {
|
|
163
|
+
dewind(data, { fs: 48000 }) // in place
|
|
164
|
+
let write = dewind({ fs: 48000 }) // stream: write(chunk) → the samples done, write() → the rest
|
|
160
165
|
```
|
|
161
166
|
|
|
162
167
|
| Param | Default | |
|
|
163
168
|
|---|---|---|
|
|
164
|
-
| `
|
|
165
|
-
| `
|
|
166
|
-
| `
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
| |
|
|
174
|
-
|
|
175
|
-
|
|
|
176
|
-
|
|
|
177
|
-
|
|
|
178
|
-
|
|
|
179
|
-
|
|
|
180
|
-
|
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
| bass line | 100.0% → 0.0% | −32.8 → 0.0 | −18.7 → 0.0 | −5.8 → 0.0 |
|
|
169
|
+
| `cutoff` | `1500` | Hz, the top of the band wind is taken from; read every frame |
|
|
170
|
+
| `attenuation` | `-20` | dB, the most a bin is turned down; `0` takes nothing |
|
|
171
|
+
| `frameSize` | 85 ms | STFT frame, a power of two; the hop a quarter |
|
|
172
|
+
|
|
173
|
+
`python scripts/wind.py fetch`, then `node scripts/lowend.js dewind` puts clean speech and music through it, then speech with wind: synthetic (Gaussian noise shaped and gusting as Nelke & Vary measure it), generated (the SC-Wind-Noise-Generator, Mirabilii et al., IWAENC 2022: spectrum and gusts by wind speed) and recorded (twelve CC0 recordings of wind on a microphone, freesound.org), half of the generated and recorded tuning the defaults, half below. VoiceBank+DEMAND test utterances (every fourth: p232 male, p257 female), ten Spoken Wikipedia narrations, the music `repair` uses, 0.2.0 → now; voiced frames thinned: the voiced 10 ms frames whose level under 250 Hz fell by more than 3 dB; their periodic part: whose periodic energy there (r·E, Boersma 1993) did:
|
|
174
|
+
|
|
175
|
+
| | untouched | voiced frames thinned | their periodic part | 20–63 Hz | 63–125 Hz | 125–250 Hz |
|
|
176
|
+
|---|---:|---:|---:|---:|---:|---:|
|
|
177
|
+
| speech, male | 0% → 0% | 5.1% → 9.1% | 4.9% → 7.2% | −0.2 → −1.7 | 0.0 → −0.5 | 0.0 → −0.1 |
|
|
178
|
+
| speech, female | 0% → 0% | 8.8% → 16.2% | 8.5% → 11.6% | −1.2 → −4.5 | −0.1 → −0.7 | 0.0 → 0.0 |
|
|
179
|
+
| narrations | 0% → 20% | 2.4% → 6.0% | 2.4% → 5.2% | −0.1 → −0.7 | −0.1 → −0.3 | 0.0 → −0.2 |
|
|
180
|
+
| audio-lena | 100% → 100% | 0.0% → 0.0% | 0.0% → 0.0% | 0.0 → 0.0 | 0.0 → 0.0 | 0.0 → 0.0 |
|
|
181
|
+
| Vibe Ace (jazz) | 0% → 0% | 0.1% → 1.2% | 0.1% → 0.9% | 0.0 → −0.2 | 0.0 → −0.1 | 0.0 → −0.1 |
|
|
182
|
+
| Brahms (strings) | 0% → 100% | 0.0% → 0.0% | 0.0% → 0.0% | 0.0 → 0.0 | 0.0 → 0.0 | 0.0 → 0.0 |
|
|
183
|
+
| Nutcracker | 0% → 100% | 0.0% → 0.0% | 0.0% → 0.0% | 0.0 → 0.0 | 0.0 → 0.0 | 0.0 → 0.0 |
|
|
184
|
+
| trumpet | 100% → 100% | 0.0% → 0.0% | 0.0% → 0.0% | 0.0 → 0.0 | 0.0 → 0.0 | 0.0 → 0.0 |
|
|
185
|
+
| bass line | 0% → 100% | 0.0% → 0.0% | 0.0% → 0.0% | 0.0 → 0.0 | 0.0 → 0.0 | 0.0 → 0.0 |
|
|
186
|
+
|
|
187
|
+
A recording's own low rumble, where it outweighs the mid band in a pause, reads as wind: VoiceBank's room tone before each take engages it for the take's first second, and narrations recorded at home in their pauses. The rumble goes; the voiced frames that thin are onsets, whose low end an 85 ms frame smears into a broad bump that reads as floor, and frames whose low end is mostly that rumble. Orchestral music, a trumpet and a bass line come back untouched; a jazz track's drums engage it for moments.
|
|
184
188
|
|
|
185
189
|
Wind at a speech-to-wind ratio of +10, 0 and −10 dB, the error to the clean speech taken away (dB):
|
|
186
190
|
|
|
187
191
|
| | +10 dB | 0 dB | −10 dB |
|
|
188
192
|
|---|---:|---:|---:|
|
|
189
|
-
|
|
|
190
|
-
|
|
|
193
|
+
| synthetic | −0.1 → 5.1 | 2.4 → 9.9 | 5.6 → 13.1 |
|
|
194
|
+
| generated | −0.1 → 4.4 | 1.3 → 8.3 | 4.1 → 11.3 |
|
|
195
|
+
| recorded | −0.4 → 3.8 | 1.1 → 7.5 | 3.2 → 10.2 |
|
|
196
|
+
|
|
197
|
+
The wind removed and the speech kept, by phase inversion (Hagerman & Olofsson, Acta Acustica 2004: the op on s + n and on s − n, ŝ = (y₊ + y₋)/2, n̂ = (y₊ − y₋)/2), dB:
|
|
198
|
+
|
|
199
|
+
| | wind removed, +10 dB | 0 dB | −10 dB | speech kept, +10 dB | 0 dB | −10 dB |
|
|
200
|
+
|---|---:|---:|---:|---:|---:|---:|
|
|
201
|
+
| synthetic | 2.2 → 9.3 | 5.3 → 14.0 | 7.0 → 15.0 | −0.1 → −0.3 | −0.4 → −0.8 | −0.7 → −1.8 |
|
|
202
|
+
| generated | 1.7 → 8.8 | 3.5 → 12.4 | 5.4 → 13.3 | −0.1 → −0.4 | −0.5 → −1.2 | −0.7 → −3.0 |
|
|
203
|
+
| recorded | 1.3 → 8.0 | 3.1 → 10.8 | 4.0 → 11.4 | −0.1 → −0.4 | −0.4 → −1.1 | −0.3 → −2.1 |
|
|
204
|
+
|
|
205
|
+
PESQ (wideband), STOI and DNSMOS P.835 of the same outputs, the three winds together (`python scripts/wind.py score DIR/now test`; scored at 16 kHz as `scripts/speech.py` does):
|
|
206
|
+
|
|
207
|
+
| | PESQ | STOI | SIG | BAK | OVRL |
|
|
208
|
+
|---|---:|---:|---:|---:|---:|
|
|
209
|
+
| +10 dB: input | 1.76 | 0.939 | 3.51 | 3.36 | 2.89 |
|
|
210
|
+
| 0.2.0 | 1.67 | 0.937 | 3.47 | 3.44 | 2.90 |
|
|
211
|
+
| now | 2.30 | 0.935 | 3.34 | 3.61 | 2.88 |
|
|
212
|
+
| 0 dB: input | 1.21 | 0.874 | 3.25 | 2.44 | 2.31 |
|
|
213
|
+
| 0.2.0 | 1.23 | 0.870 | 3.26 | 2.66 | 2.41 |
|
|
214
|
+
| now | 1.58 | 0.875 | 3.20 | 3.19 | 2.58 |
|
|
215
|
+
| −10 dB: input | 1.06 | 0.754 | 2.06 | 1.43 | 1.43 |
|
|
216
|
+
| 0.2.0 | 1.09 | 0.753 | 2.36 | 1.67 | 1.63 |
|
|
217
|
+
| now | 1.16 | 0.756 | 2.83 | 2.41 | 2.08 |
|
|
218
|
+
| all: input | 1.34 | 0.856 | 2.94 | 2.41 | 2.21 |
|
|
219
|
+
| 0.2.0 | 1.33 | 0.853 | 3.03 | 2.59 | 2.31 |
|
|
220
|
+
| now | 1.68 | 0.855 | 3.12 | 3.07 | 2.51 |
|
|
191
221
|
|
|
192
|
-
|
|
222
|
+
STOI barely moves: its bands begin at 150 Hz, above most of the wind. In light wind DNSMOS's SIG drops 0.17; an ideal Wiener gain from the wind's own spectrum under 1.5 kHz lost 0.12 there on the tuning material.
|
|
193
223
|
|
|
194
|
-
|
|
195
|
-
|
|
224
|
+
0.2.0 high-passed only between the words: a voiced low end sent its filter out, so the wind under the words stayed. The wind now goes under the words too, between and under the harmonics; what a voice loses is mostly the harmonics the wind buries (its low end at −10 dB), and onsets.
|
|
225
|
+
|
|
226
|
+
**Use when:** wind on a microphone under a voice or an instrument, gusting or steady; a recording's low rumble (traffic, a fan) where it outweighs the mid band in the pauses.<br>
|
|
227
|
+
**Not for:** a lone thump (`deplosive`), a steady low tone or hum, which has a period (`dehum`, `highpass`), or broadband noise (`omlsa`, `wiener`). Air rushing over the whole band is taken only under `cutoff`. A neural model (`@audio/neural-denoise`'s DeepFilterNet3) takes more of the wind and keeps more of the voice; dewind leaves a sound with no aperiodic low end as it is (music, a voice with no rumble under it), streams at a frame's delay, and needs no model.
|
|
196
228
|
|
|
197
229
|
|
|
198
230
|
### `deplosive`
|
|
@@ -313,11 +345,13 @@ wiener(data, { fs, noiseFrames: 6 }) // noise from the
|
|
|
313
345
|
|
|
314
346
|
### `omlsa`
|
|
315
347
|
|
|
316
|
-
Optimally-Modified Log-Spectral Amplitude estimator (Cohen & Berdugo 2001) with IMCRA noise estimation (Cohen 2003). The log-spectral amplitude gain when speech is present, weighed against a floor by the speech presence probability p: `G = max(G_H1, G_min)^p · G_min^(1−p)` (eq. 16 with G<sub>H1</sub> floored at G<sub>min</sub>: as written it took a bin below G<sub>min</sub> wherever G<sub>H1</sub> was under it, to −18 dB at G<sub>min</sub> −12, so `gMin` was not the floor). The a priori SNR is decision-directed on G<sub>H1</sub> (eq. 18), the a priori speech absence is estimated from
|
|
348
|
+
Optimally-Modified Log-Spectral Amplitude estimator (Cohen & Berdugo 2001) with IMCRA noise estimation (Cohen 2003). The log-spectral amplitude gain when speech is present, weighed against a floor by the speech presence probability p: `G = max(G_H1, G_min)^p · G_min^(1−p)` (eq. 16 with G<sub>H1</sub> floored at G<sub>min</sub>: as written it took a bin below G<sub>min</sub> wherever G<sub>H1</sub> was under it, to −18 dB at G<sub>min</sub> −12, so `gMin` was not the floor). The a priori SNR is decision-directed on G<sub>H1</sub> (eq. 18), the a priori speech absence is estimated from the spread of an a priori SNR over time and neighbouring bins (§4), the noise spectrum comes from IMCRA. As Cohen's own `omlsa.m`, a bin whose speech absence reaches 0.9 counts as noise: noise that happens to peak keeps G<sub>min</sub> and leaves no musical tone, so what remains of the noise is the noise, G<sub>min</sub> quieter. Time constants, the decision-directed α among them, are set for 8 ms frames and rescaled to the actual frame step (`omlsa.m` rescales by the reciprocal, which lengthens them at 48 kHz).
|
|
317
349
|
|
|
318
350
|
α sets how soon ξ follows a word's start after a pause, and how much gain a noise peak gets. Until 0.3 it was 0.98 per frame, whatever the frame (0.985 per 8 ms at 48 kHz, 0.972 at 44.1), and a word's second and third frames after a pause lost 10 and 5 dB. It is now the lowest value per 8 ms that leaves steady white and pink noise without musical noise: 0.97 tracking the noise (log kurtosis ratio 0.00 at G<sub>min</sub> −15 and −25 dB; 0.96 left 1.16 and 0.49 at −25, the paper's 0.92 0.99 and 1.81 at −15), 0.95 on a learned noise (`omlsa.m`'s; 0.92 left 0.19 and 0.42). The paper's cap q ≤ q<sub>max</sub> = 0.95 (Table 1) in place of the gate kept a word's second frame after a pause another 3 dB but left musical noise on steady noise (0.24 and 0.46 at α 0.97), so the gate stays. The table under [Speech](#speech) has what each keeps. `scripts/reference.py` holds a numpy version written from the papers; on 16 kHz VoiceBank frames it gives `omlsa.m`'s noise track, speech absence and gains to the last bit (but for near-empty bins, where `omlsa.m`'s absolute 1e-10 floors bind), and test.js holds this code to it.
|
|
319
351
|
|
|
320
|
-
|
|
352
|
+
The a priori SNR the speech absence is read from is smoothed in the cepstrum (Breithaupt, Gerkmann & Martin 2008): the speech power's maximum-likelihood estimate λ·max(γ−1, ξ<sub>ml,min</sub>), its cepstrum smoothed over time per quefrency, hardly where speech lives (the envelope's low quefrencies, the pitch's peak), much where a lone noise peak lives, and back. The decision-directed ξ (Cohen's, `qFrom: 'dd'`) lags a word's start after a pause, and takes a short burst of noise the tracker cannot follow for speech; the cepstral one follows a word's envelope at once and smooths a lone spectral peak away. Its constants are Table 1's, per 16 ms of frame step, but under 1.25 ms of quefrency Gerkmann & Hendriks's (ICASSP 2012: 0 and 0.2, where the 2008 paper's 0.5 and 0.7 lag an onset as the decision-directed ξ does) and above it 0.9: the least that leaves steady white and pink noise on a learned noise free of musical noise (0.85 left 0.22 on pink). On the training speech, tracked: PESQ 1.823 → 1.825, STOI 0.832 → 0.834, OVRL 2.525 → 2.543, BAK 3.00 → 3.04, musical noise 0.82 → 0.80; 40 ms tone bursts in steady noise pass at −6.5 dB (were −2.6). The cepstral ξ as the gain's own ξ as well lost voiced speech where the pitch glides (SIG −0.04), so the gain keeps the decision-directed one.
|
|
353
|
+
|
|
354
|
+
A noise that holds still can be learned instead: `profile`, the noise's PSD (`noiseProfile` of a stretch where it plays alone), or, in the batch form, `noiseFrames`/`profileFrom`/`profileTo` naming that stretch. The noise is then held (`known` of @audio/noise-estimate), and p reads the observation: γ averaged over 105.5 Hz and over 543 Hz of the frame, at fixed priors (Gerkmann, Breithaupt & Martin 2008: the averaged γ is χ² with r degrees of freedom from the Hann window's correlation across bins, ξ<sub>fix</sub> the a priori SNR that minimizes false alarms plus misses, eq. 13; at 48 kHz r 5.7 and 24, ξ<sub>fix</sub> 9.4 and 5.1 dB), and 0 where the estimated q reaches 0.9, as before. Until 0.4 it read γ in each bin alone, at ξ<sub>H1</sub> = 15 dB (Gerkmann & Hendriks 2012), and speech 0–5 dB over the noise lost 7.8 dB on the training speech, where the MMSE Wiener gain keeps 3.9. The paper averages over 64 ms of frames as well; presence then outlasted a sound by those frames, and in the half second after music stopped the noise came through in tones (log kurtosis ratio 0.45 at G<sub>min</sub> −12 dB, 1.77 at −20): over the frame alone. On the training speech, G<sub>min</sub> −12 dB: PESQ 1.861 → 1.867, OVRL 2.564 → 2.572, BAK 3.05 → 3.08, musical noise 0.53 → 0.46, speech 0–5 dB over the noise −7.8 → −5.1 dB, the noise taken in pauses 9.7 → 10.1 dB. Steady noise under speech and music goes exactly G<sub>min</sub> down, with no musical noise from −12 to −20 dB (log kurtosis ratio 0.00); in the half second after music stops, 0.05 at −12 dB and 0.61 at −20 (0.3: 0.11 and 1.00; `scripts/broadband.mjs`). Tracking the noise from the print (Gerkmann & Hendriks 2012's MMSE tracker started on it, RX's "adaptive") lost OVRL 0.04 and left musical noise (0.40 on pink): the print is held. `processor(opts)` is the gain as an @audio/stft frame process, for a host running its own frames: audio's `denoise` op learns a print from a range and runs it so.
|
|
321
355
|
|
|
322
356
|
```js
|
|
323
357
|
omlsa(data, { fs })
|
|
@@ -331,6 +365,7 @@ omlsa(data, { fs, profileFrom: 0, profileTo: fs / 2, gMin: -12 }) // the noise
|
|
|
331
365
|
| `alpha` | `0.97`; `0.95` with a `profile` | Decision-directed smoothing per 8 ms of frame step, rescaled to the actual step (alias of `alphaDD`) |
|
|
332
366
|
| `xiMin` | `10^−2.5` | A priori SNR floor (−25 dB) |
|
|
333
367
|
| `qPrior` | estimated | A fixed a priori speech absence instead of the estimate |
|
|
368
|
+
| `qFrom` | `'cts'` | The a priori SNR the speech absence is read from: smoothed in the cepstrum, or `'dd'`, decision-directed (Cohen's `omlsa.m`) |
|
|
334
369
|
| `frameSize` | power of two nearest 32 ms | STFT frame: 512 at 16 and 22.05 kHz, 1024 at 44.1, 2048 at 48 (`frame(fs)`) |
|
|
335
370
|
| `hopSize` | `frameSize/4` | |
|
|
336
371
|
| `profile` | tracked | A known noise PSD (`frameSize/2+1` bins), held |
|
|
@@ -437,63 +472,76 @@ Rails are found at the 20 dB level clipped asymmetrically (with the 10 dB level
|
|
|
437
472
|
|
|
438
473
|
### `dereverb`
|
|
439
474
|
|
|
440
|
-
Late reverberation off
|
|
475
|
+
Late reverberation off a voice, one microphone, the whole take at once. First weighted prediction error, WPE (Nakatani et al. 2010): in each STFT bin, what the frames 50 to 160 ms back predict of the current one is the room's tail, and is subtracted; the prediction is fitted over the whole take, each frame weighted by its inverse power, three times. The room is one for the take and the voice is not, so the fit learns the room. One microphone cannot invert a room, and the prediction cancels a dB or two of the tail; but its taps measure the room. The power they carry from the past into a frame is the late reverberation's power, which Lebart et al. (2001) and Habets (2010) model from a decay time and a direct-to-reverberant ratio: here neither is estimated, and on a dry voice the taps are near zero. A log-spectral-amplitude gain (Ephraim & Malah 1985) takes that power off what WPE left, floored at −14 dB, and no bin leaves louder than it came. Below 500 Hz the estimate falls 3 dB an octave: there a voice's harmonics hold their phase longest and the taps learn some of them as room. A bin cut to digital silence (an edit, a gate) is left out of the fit. About 0.1 s per second of 48 kHz sound.
|
|
441
476
|
|
|
442
477
|
```js
|
|
443
478
|
dereverb(data, { fs: 48000 })
|
|
479
|
+
dereverb(data, { fs: 48000, strength: 0.5 }) // a lighter hand
|
|
444
480
|
```
|
|
445
481
|
|
|
446
482
|
| Param | Default | |
|
|
447
483
|
|---|---|---|
|
|
448
|
-
| `
|
|
484
|
+
| `strength` | `1` | Scale of the late-reverberation estimate: 0 is the linear prediction alone, 2 takes more of the tail and more of the voice |
|
|
485
|
+
|
|
486
|
+
Measured (`python scripts/dereverb.py`, test set): 42 VoiceBank test utterances (1.6–5.9 s), and four 41–60 s takes of ten utterances by one speaker, in 19 MIT IR Survey rooms the tuning never heard (T60 0.38–1.85 s). Each room's response is split at direct + 50 ms: under it, the voice as the room colours it; over it, the tail to take. Voice lost: the output's power where the voice is 10 dB over the tail; tail taken: where the tail is 10 dB over the voice. PESQ and STOI against the dry take; SRMR (Falk et al. 2010) higher is drier; DNSMOS at the input's loudness; a dry voice through it, PESQ. 0.2 → 0.3:
|
|
487
|
+
|
|
488
|
+
| | voice lost dB | tail taken dB | PESQ | STOI | SRMR | OVRL | dry voice, PESQ |
|
|
489
|
+
|---|---|---|---|---|---|---|---|
|
|
490
|
+
| short takes in | | | 1.39 | 0.774 | 3.98 | 2.19 | 4.64 |
|
|
491
|
+
| short, `dereverb` | −0.32 → −0.81 | −1.7 → **−6.2** | 1.45 → **1.50** | 0.793 → 0.768 | 4.56 → **6.90** | 2.28 → **2.47** | 4.42 → 4.25 |
|
|
492
|
+
| long takes in | | | 1.37 | 0.739 | 3.43 | 1.98 | 4.64 |
|
|
493
|
+
| long, `dereverb` | −0.13 → −0.25 | −1.1 → **−3.3** | 1.43 → **1.49** | 0.756 → 0.749 | 3.83 → **4.99** | 2.03 → **2.43** | 4.58 → 4.58 |
|
|
494
|
+
|
|
495
|
+
`strength` 0, 1, 2, 4: on the short takes the tail −1.4, −6.2, −7.7, −9.0 dB for the voice −0.12, −0.81, −1.18, −1.64, a dry voice's PESQ 4.51, 4.25, 4.17, 4.10; on the long ones the tail −0.9, −3.3, −4.5, −6.0 for the voice −0.02, −0.25, −0.41, −0.65. A sung voice passes (VocalSet: SI-SDR 19.4 → 35.3 dB, 125–250 Hz −2.55 → −0.45 dB). STOI falls 0.025 on the short takes, the gain's cost; PESQ, SRMR and DNSMOS rise.
|
|
449
496
|
|
|
450
|
-
|
|
451
|
-
|
|
497
|
+
0.2 fitted the prediction recursively, over its last second: it learned some of the voice as room (−0.32 dB of the voice for 1.7 dB of the tail), and alone it cannot take much more (at 30 taps, about 4 dB of the tail for 0.6 dB of the voice). Late-reverb subtraction at each room's measured T60 (Lebart; Habets's κ) took 6 dB for −0.4 dB of the voice on the training takes, but it needs the T60, which free-decay estimates did not give on 3 s takes (correlation −0.3 to 0.2 with the measured one), and it took a dry voice for a room (PESQ 3.7). DeepFilterNet3 (`@audio/neural-denoise`), a denoiser that also learned to take reverberation, took 6.6–8.1 dB of the tail on the training takes and 2.1–2.9 dB of the first 50 ms, early reflections with the tail: PESQ 1.71–1.74 against 1.70 here, STOI 0.84 against 0.82.
|
|
498
|
+
|
|
499
|
+
**Use when:** a voice in a room, one microphone, one room per take.<br>
|
|
500
|
+
**Not for:** music or anything holding a pitch: a held note is predictable, and taken for room (music loses 2–5.5 dB under 1 kHz; 0.2 lost 1–4.3). Noise: denoise first.
|
|
452
501
|
|
|
453
502
|
|
|
454
503
|
## Pitch drift
|
|
455
504
|
|
|
456
505
|
### `dewow`
|
|
457
506
|
|
|
458
|
-
Wow & flutter correction. A speed change of the
|
|
507
|
+
Wow & flutter correction. A speed change of the disc or tape moves every frequency in the recording by one ratio at one instant; a singer's vibrato, a glide or a melody moves one note and its harmonics. dewow tracks the partials (phase-vocoder frequency over a 0.19 s frame), cuts them into steady pieces, and fits Godsill & Rayner's speed model (*Digital Audio Restoration*, 1998, ch. 8) with each piece's centre tied to its pitch class: music reuses its pitches, so every return of a note measures the speed, where free centres let errors add up along the chain of overlapping notes. Harmonics of a note count as one source; a hop counts where two independent sources agree. Of that curve it applies only what a second test proves: a disc's wow, a sinusoid at a turntable's rotation (33⅓, 45 or 78 rpm within 4 %, or its 2nd harmonic), fitted to the partials and kept where its amplitude stands 4.5 times over its own bootstrapped noise, holds from one half of its 20 s window to the other and reaches 0.1 %. A pilot tone running through the recording (above 5 kHz) is read instead, flutter too; `mode: 'reference'` reads a tone given or found (a pilot, a calibration tone, mains hum). The sound is read back through a variable-rate windowed sinc. Nothing proven, nothing changed: the clip comes back bit-exact. Whole-signal, length-preserving. `wowFlutter()` alone is the meter.
|
|
459
508
|
|
|
460
509
|
```js
|
|
461
|
-
dewow(data, { fs }) // from the music
|
|
462
|
-
dewow(data, { fs, mode: 'reference'
|
|
463
|
-
dewow(data, { fs,
|
|
464
|
-
wowFlutter(data, { fs }) // → { speed, times, wow, flutter, confidence }
|
|
510
|
+
dewow(data, { fs }) // a disc's wow from the music; a pilot tone if there is one
|
|
511
|
+
dewow(data, { fs, mode: 'reference' }) // the tone found: a pilot, else 50/60 Hz hum
|
|
512
|
+
dewow(data, { fs, mode: 'reference', refFreq: 1000 }) // a 1 kHz calibration tone
|
|
513
|
+
wowFlutter(data, { fs }) // → { speed, times, wow, flutter, lines, reference, confidence }
|
|
465
514
|
```
|
|
466
515
|
|
|
467
516
|
| Param | Default | |
|
|
468
517
|
|---|---|---|
|
|
469
518
|
| `mode` | `'partial'` | `'partial' \| 'reference' \| 'pitch'` (`pitch`: one voice's f0 against its own trend; takes its vibrato for speed) |
|
|
470
|
-
| `refFreq` |
|
|
519
|
+
| `refFreq` | found | Hz — the tone for `reference`; omitted: a pilot above 5 kHz, else 50/60 Hz hum |
|
|
471
520
|
| `smooth` | `0.05` | s — splits the curve into wow (slower) and flutter |
|
|
472
521
|
| `wow` / `flutter` | `true` / `true` | correct each part |
|
|
473
522
|
| `maxDeviation` | `0.05` | clamp on the speed ratio (±5 %) |
|
|
474
|
-
| `minTrack` | `0.
|
|
523
|
+
| `minTrack` | `0.2` | s — shortest steady piece of a partial used (`partial`) |
|
|
475
524
|
| `minFreq` / `maxFreq` | `50` / `2000` | Hz — where partials (`partial`) or the f0 (`pitch`) are looked for |
|
|
476
525
|
| `keepLength` | `true` | output length equals input |
|
|
477
526
|
|
|
478
|
-
`node scripts/dewow.js` reads clean speech (audio-lena, two Spoken Wikipedia narrations), music ("Vibe Ace", "Dance of the Sugar Plum Fairy", Brahms' Hungarian Dance No. 5, a trumpet loop, three GuitarSet takes) and singing (five VocalSet excerpts) at a varying speed —
|
|
527
|
+
`node scripts/dewow.js` reads clean speech (audio-lena, two Spoken Wikipedia narrations), music ("Vibe Ace", "Dance of the Sugar Plum Fairy", Brahms' Hungarian Dance No. 5, a trumpet loop, three GuitarSet takes, six MUSDB18 and four BabySlakh mixes) and singing (five VocalSet excerpts) at a varying speed — a disc turning off-centre at 33⅓, 45 or 78 rpm, or tape's random wow (0.5–6 Hz) — at 0.3, 1 and 2 % peak, and measures the pitch error left on the audio itself (local lag against the clean sound, differentiated; cents RMS, mean over clips; doing nothing leaves the wow):
|
|
479
528
|
|
|
480
529
|
| | wow | 0.3 % | 1 % | 2 % |
|
|
481
530
|
|---|---|---:|---:|---:|
|
|
482
|
-
|
|
|
483
|
-
| |
|
|
484
|
-
|
|
|
485
|
-
| | tape | 1.
|
|
486
|
-
| singing |
|
|
487
|
-
| |
|
|
488
|
-
|
|
|
489
|
-
| | tape | 1.2 → 0.2 | 4.1 → 0.3 | 8.3 → 0.6 |
|
|
531
|
+
| music | disc 33⅓ | 3.7 → 1.7 | 12.2 → 2.1 | 24.5 → 4.3 |
|
|
532
|
+
| | disc 45 | 3.7 → 1.6 | 12.2 → 3.0 | 24.5 → 6.0 |
|
|
533
|
+
| | disc 78 | 3.7 → 1.2 | 12.2 → 2.9 | 24.5 → 13.4 |
|
|
534
|
+
| | tape | 1.4 → 1.4 | 4.6 → 4.6 | 9.2 → 9.2 |
|
|
535
|
+
| speech, singing | any | unchanged | unchanged | unchanged |
|
|
536
|
+
| steady notes (C4 E4 G♯4 D5) | disc 33⅓ / 45 / 78 | 3.7 → 0.0 / 0.1 / 0.1 | 12.3 → 0.1 / 0.2 / 0.5 | 24.6 → 0.3 / 0.5 / 0.9 |
|
|
537
|
+
| | tape | 1.3 → 1.3 | 4.5 → 4.5 | 9.0 → 9.0 |
|
|
490
538
|
|
|
491
|
-
|
|
539
|
+
The longer the recording, the more turns of the disc and returns of each note it holds: on whole pieces (`node scripts/dewow.js whole`) disc wow at 33⅓ rpm comes out of the Sugar Plum Fairy (120 s) at 0.4, 0.6 and 0.8 cents from 3.7, 12.2 and 24.5; of "Vibe Ace" (61 s) at 0.6, 1.0 and 1.9; of the Brahms (46 s, strings with vibrato) at 3.0, 2.4 and 3.2. Flutter (random, 6–30 Hz, 0.1 % peak, over 1 % tape wow) is not read from the music; a 19 kHz pilot 40 dB under the program, found by itself, reads it all: 4.2–4.5 → 0.1 cents on speech, music and singing. 50 Hz hum 30 dB down is a poor reference where the program has its own energy at 50–100 Hz: singing 4.3 → 2.2, speech 4.3 → 3.7, music 4.4 → 4.2 (`reference`).
|
|
492
540
|
|
|
493
|
-
|
|
541
|
+
Clean, every clip comes back bit-exact: 25 tuning and 19 held-out clips, the 504 clean VoiceBank+DEMAND training and 824 test utterances, a vibrato voice (±50 cents at 5.5 Hz), a 220 → 330 Hz glide, a vibrato voice over steady notes. With wow, no clip came out worse. Held out (never tuned on: two narrations, six MUSDB18 test mixes, three GuitarSet takes, four BabySlakh mixes, four VocalSet singers), music under disc wow at 0.3 / 1 / 2 %: 33⅓ rpm 3.7 → 2.3, 12.2 → 2.8, 24.5 → 6.9; 45 rpm 1.9, 2.8, 7.7; 78 rpm 2.0, 5.7, 17.6. 0.2 on the tuning music: 33⅓ rpm 3.7 → 3.6, 12.2 → 10.5, 24.5 → 18.0, 3 of 17 clean clips moved and 4 tape cases made worse (held out: 12.2 → 11.4 at 1 %); it did correct steady notes under random tape wow (4.5 → 0.5 cents), which 0.3 leaves. What dewow cannot do, measured: random tape wow, read from the music alone, is left — over a few seconds the notes' own pitch movement is as large as the wow, and no test told the two apart without also passing a voice's intonation; one voice or instrument alone gives no evidence, so speech and singing are corrected only through a tone; a clip under 5 s is never corrected from the music.
|
|
494
542
|
|
|
495
|
-
**Use when:**
|
|
496
|
-
**Not for:**
|
|
543
|
+
**Use when:** disc transfers (33⅓, 45, 78 rpm) of music with more than one voice, the record off-centre or warped; any transfer with a pilot tone (found by itself) or a calibration tone (`reference`).<br>
|
|
544
|
+
**Not for:** random tape wow read from the music alone (left as it is); a solo voice or instrument without a tone to read; dropouts; azimuth/time-skew.
|
|
497
545
|
|
|
498
546
|
|
|
499
547
|
## Gates & inter-word
|
|
@@ -524,24 +572,26 @@ debreath(data, { range: -10 }) // -10 dB between
|
|
|
524
572
|
| `attack` | `0.005` | s — the gain rises over this before speech starts |
|
|
525
573
|
| `release` | `0.1` | s — and falls over this after speech ends |
|
|
526
574
|
|
|
527
|
-
`snrTh` and `flatTh` (0.1) are gone: they tuned the old detector, whose floor was the 10th-percentile frame energy of the whole input. Under noise that percentile is the noise, and every word under 9 dB over it went down. Measured with `python scripts/vad.py` (VoiceBank+DEMAND test set, ten Spoken Wikipedia narrations; defaults chosen on the training subset and ten other narrations), 0.1.8 → 0.2.
|
|
575
|
+
`snrTh` and `flatTh` (0.1) are gone: they tuned the old detector, whose floor was the 10th-percentile frame energy of the whole input. Under noise that percentile is the noise, and every word under 9 dB over it went down. Measured with `python scripts/vad.py` (VoiceBank+DEMAND test set, ten Spoken Wikipedia narrations; defaults chosen on the training subset and ten other narrations), 0.1.8 → 0.2.1, frames turned down by over 3 dB:
|
|
528
576
|
|
|
529
577
|
| | voiced | word edges |
|
|
530
578
|
|---|---:|---:|
|
|
531
|
-
| VoiceBank+DEMAND, 824 noisy | 5.87 → **0.
|
|
532
|
-
| the same, clean | 0.03 → 0.09 % | 1.22 → **0.
|
|
533
|
-
| 10 narrations | 0.07 → 0.
|
|
579
|
+
| VoiceBank+DEMAND, 824 noisy | 5.87 → **0.06** % | 24.55 → **0.30** % |
|
|
580
|
+
| the same, clean | 0.03 → 0.09 % | 1.22 → **0.60** % |
|
|
581
|
+
| 10 narrations | 0.07 → 0.05 % | 0.23 → 0.03 % |
|
|
534
582
|
|
|
535
583
|
Breaths (350 ms of noise through three wide resonances, 500/1500/2500 Hz) put into the narrations' pauses, ending G before the next phrase, 35 and 25 dB under the speech: median gain, share turned down by 6 dB or more.
|
|
536
584
|
|
|
537
585
|
| G | −35 dB | −25 dB |
|
|
538
586
|
|---|---|---|
|
|
539
|
-
| 0.05 s | −6.6 dB, 53 % →
|
|
540
|
-
| 0.15 s | −2.9 dB, 44 % → **−
|
|
541
|
-
| 0.3 s | −4.0 dB, 50 % → **−11.
|
|
542
|
-
| 0.5 s | −11.5 dB, 55 % → **−12.0 dB,
|
|
587
|
+
| 0.05 s | −6.6 dB, 53 % → **−11.1 dB, 55 %** | −0.2 dB, 45 % → 0.0 dB, 13 % |
|
|
588
|
+
| 0.15 s | −2.9 dB, 44 % → **−11.7 dB, 81 %** | −0.1 dB, 44 % → **−11.4 dB, 74 %** |
|
|
589
|
+
| 0.3 s | −4.0 dB, 50 % → **−11.8 dB, 75 %** | −3.2 dB, 50 % → **−11.8 dB, 69 %** |
|
|
590
|
+
| 0.5 s | −11.5 dB, 55 % → **−12.0 dB, 91 %** | −11.5 dB, 55 % → **−11.6 dB, 73 %** |
|
|
591
|
+
|
|
592
|
+
A breath within 0.3 s of a vowel, parted from it by less than 0.15 s, reads as the word's onset and stays: that is where consonants lie, parted from the vowel at most by a stop's closure. A loud breath that close to a phrase went down more often before (45 → 13 %), and so did a quarter of the word edges in noise.
|
|
543
593
|
|
|
544
|
-
|
|
594
|
+
Noise after speech goes down: 1.5 s of noise after 1.5 s of a VoiceBank utterance, frames from 0.4 s on turned down by 6 dB or more, white, pink and brown 94 %, the office 99 %; noises with talkers in them (bus, cafe, public square) stay, as speech. 0.2.0 kept white and pink noise after speech whole (its VAD read voicing off the Wiener estimate, which peaks on any noise). Music, frames within 30 dB of the loudest turned down by over 3 dB: Vibe Ace 37.4 → 0.67 %, Nutcracker 22.1 → 0 %, trumpet 0 → 0 %, four sung excerpts (VocalSet) up to 21.6 → up to 0.39 %, but Brahms (strings) 4.29 → 9.35 %: an orchestra raises its own floor (0.2.0: 0.02 %).
|
|
545
595
|
|
|
546
596
|
**Use when:** breath, mouth noise, hiss in pauses on a voiceover.<br>
|
|
547
597
|
**Not for:** a breath that runs into a word; whispered speech (it holds no voicing: it goes down whole).
|
|
@@ -570,15 +620,15 @@ splitSilence(data, { fs }) // → one Floa
|
|
|
570
620
|
|
|
571
621
|
Returns `{ data, segments, removed, map }`; `project(map, t)` re-times markers/subtitles (`@audio/subtitle` `project()` takes the same map).
|
|
572
622
|
|
|
573
|
-
0.1 read silence under 11 dB over the input's 10th-percentile frame energy: under noise that percentile is the noise, and whole words went. Measured with `python scripts/vad.py` (as `debreath` above), `shorten` at its defaults, 0.1.1 → 0.2.
|
|
623
|
+
0.1 read silence under 11 dB over the input's 10th-percentile frame energy: under noise that percentile is the noise, and whole words went. Measured with `python scripts/vad.py` (as `debreath` above), `shorten` at its defaults, 0.1.1 → 0.2.1, frames cut:
|
|
574
624
|
|
|
575
625
|
| | voiced | word edges | removed |
|
|
576
626
|
|---|---:|---:|---:|
|
|
577
|
-
| VoiceBank+DEMAND, 824 noisy | 3.89 → **0.
|
|
578
|
-
| the same, clean | 0.00 → 0.
|
|
579
|
-
| 10 narrations | 0.00 → 0.
|
|
627
|
+
| VoiceBank+DEMAND, 824 noisy | 3.89 → **0.02** % | 3.93 → **0.08** % | 290 → 40 of 2072 s |
|
|
628
|
+
| the same, clean | 0.00 → 0.05 % | 0.01 → 0.19 % | 167 → 156 s |
|
|
629
|
+
| 10 narrations | 0.00 → 0.04 % | 0.00 → 0.03 % | 50 → 56 of 600 s |
|
|
580
630
|
|
|
581
|
-
Breaths in the narrations' pauses (as above) ending 0.3 s before the next phrase: 56 → 75 % of each removed at −35 dB, 50 → 69 % at −25 dB. Music, frames within 30 dB of the loudest cut: Vibe Ace 10.5 → 0
|
|
631
|
+
Breaths in the narrations' pauses (as above) ending 0.3 s before the next phrase: 56 → 75 % of each removed at −35 dB, 50 → 69 % at −25 dB. Noise after speech is a pause (0.2.0 read white or pink noise there as speech and cut none of it). Music, frames within 30 dB of the loudest cut: Vibe Ace 10.5 → 0.5 %, Nutcracker 16.8 → 0 %, sung (VocalSet m8) 25.3 → 0 %, but Brahms (strings) 3.5 → 9.1 % (0.2.0: 0.02 %).
|
|
582
632
|
|
|
583
633
|
**Use when:** podcast/lecture pause tightening, split-by-silence, auto-trim.<br>
|
|
584
634
|
**Not for:** music (a rest is a pause to this VAD); overlapping speech; whispered speech (no voicing: it reads as pause).
|
|
@@ -654,16 +704,22 @@ Ten Spoken Wikipedia narrations, 60 s each (volunteers at home; raw OVRL 3.17):
|
|
|
654
704
|
|
|
655
705
|
In the rows above, `omlsa` 0.3 and `wiener` 0.3 take the decision-directed α per 8 ms of frame step, as their papers quote it, where it was per frame (the a priori SNR's memory ran 1.8× longer at 48 kHz than at 44.1); `omlsa`'s is the lowest per 8 ms that leaves steady noise free of musical noise, 0.97 tracking and 0.95 on a learned noise, `wiener`'s Ephraim & Malah's 0.98; `omlsa`'s G<sub>min</sub> is the floor; and minimum statistics (`wiener`, `specsub`; noise-estimate 2.2) starts its smoother on the mean of the first frames, not on one periodogram that left 2.6 % of bins 10 dB low for the first 1.5 s. Against 0.2 on the test set: `omlsa` PESQ 2.40 → 2.36, STOI 0.916 → 0.919, SI-SDR 14.5 → 14.0, SIG 3.39 → 3.38, OVRL 2.86 → 2.84; `wiener` 2.34 → 2.36, 0.910 → 0.911, 14.0 → 14.3, 3.38 → 3.40, 2.81 → 2.84; `specsub` 2.24 → 2.26, 0.920 → 0.919, 12.5 → 12.8, 3.35 → 3.36, 2.77 → 2.79 (chosen on the training subset, where tracked `omlsa` went STOI 0.826 → 0.832, PESQ 1.83 → 1.82). On a learned noise (`omlsa` with `profile`, as audio's `denoise` runs it, G<sub>min</sub> −12 dB, the noise learned from the lead-in): PESQ 2.48 → 2.45, STOI 0.915 → 0.919, SIG 3.36 → 3.38, OVRL 2.89 → 2.88.
|
|
656
706
|
|
|
657
|
-
|
|
707
|
+
`omlsa` 0.4 reads the speech absence from the a priori SNR smoothed in the cepstrum, and on a learned noise the speech presence from γ averaged over neighbouring bins ([`omlsa`](#omlsa)). Chosen on the training subset; on the test set, once, 0.3 → 0.4, tracked: PESQ 2.362 → 2.364, STOI 0.919 → 0.920, SI-SDR 14.04 → 14.25 dB, SIG 3.382 → 3.384, BAK 3.43 → 3.47, OVRL 2.838 → 2.853, musical noise 0.49 → 0.48; on a learned noise (`node scripts/speech.mjs vbdemand omlsa-learned`, as audio's `denoise` runs it, 12 dB): 2.453 → 2.455, 0.919 → 0.920, 15.05 → 15.11, 3.380 → 3.367, 3.53 → 3.54, 2.875 → 2.874, 0.39 → 0.34. The clean utterances in: PESQ 4.288 → 4.287 tracked, 4.299 → 4.288 learned; steady white and pink noise alone 15 dB down at log kurtosis ratio 0.00 and 0.02. (The learned form was scored on the test set once before, its presence averaged over 64 ms of frames as the paper has it: PESQ 2.421, OVRL 2.885; the musical noise it left after music stops, measured on the training material, then took it to the frame alone.)
|
|
708
|
+
|
|
709
|
+
The open classical denoisers on the same 824, each given the noise as ours is: noisereduce 3.0.3 (stationary spectral gating, the lead-in as its noise clip) at 12 dB (`prop_decrease` 0.75): PESQ 2.409, STOI 0.921, SI-SDR 12.8, SIG 3.37, BAK 3.51, OVRL 2.861, musical noise 0.03; at full reduction 2.357, 0.910, 12.4, 3.22, 3.72, 2.825, 0.74; logmmse 1.5 (a port of Loizou's `logmmse.m`, noise from the lead-in, updated by its VAD) 2.350, 0.900, 14.7, 3.29, 3.52, 2.799, 0.73; FFmpeg 8.0.1's `afftdn` learning the lead-in, 12 dB: 2.048, 0.920, 8.6, 3.25, 3.14, 2.657, 0.20 (`anlmdn` at its defaults leaves the input as it is). Neural, for scale: DeepFilterNet3 limited to 12 dB 2.670, 0.939, 16.2, 3.49, 3.69, 3.026, 0.06; RNNoise 2.109, 0.890, 12.3, 3.28, 3.85, 2.935, 0.90. iZotope RX was not measured: no RX plug-in is installed where these ran.
|
|
710
|
+
|
|
711
|
+
Tried on the training subset and not taken: the cepstral a priori SNR as the gain's ξ as well (tracked SIG −0.04, music cut 9.6 → 15.6 %); Gerkmann & Hendriks's MMSE noise tracker (2012) in place of IMCRA (tracked PESQ +0.04, OVRL +0.03, but STOI −0.008, clean speech in PESQ 4.24 → 4.08, music cut 9.6 → 28.5 %) or of minimum statistics in `wiener` (PESQ 1.82 → 1.87, steady-noise musical noise 0.53 → 0.79, music cut 14.5 → 28.5 %); that tracker started on a learned print (RX's "adaptive": OVRL −0.04, musical noise 0.40 on pink noise); the cepstral ξ in `wiener` (STOI +0.014, SIG −0.06); the averaged presence in the tracked gain (PESQ +0.015, a word's second frame after a pause 0.7 dB lower); the gain smoothed over frequency, in dB or linear, ±100 Hz (musical noise 0.46 → 0.22–0.29, speech cut by more than 3 dB 9.5 → 25–38 %). With the noise known exactly (the true noise's periodogram, recursively smoothed) the learned form scores PESQ 2.11 on the training subset (1.87 with the print) and leaves musical noise of 0.01 (0.46): what remains to be had is in the estimate of a noise that changes, not in the gain.
|
|
712
|
+
|
|
713
|
+
What each keeps of the speech, by shadow filtering (`node scripts/broadband.mjs`: each frame's gain, computed on the noisy mix, applied to the clean speech and to the noise alone): a word's second and third frame after 85 ms of pause, in ten Spoken Wikipedia narrations under pink noise 10 dB down; speech 20–50 dB under the take's loudest frame, and bins where it stands 0–5 dB over the noise (the MMSE-optimal Wiener gain keeps −3.9 dB there), in the test set's every fourth utterance; the noise taken; the share of clean speech, and of clean music, cut by more than 3 dB. dB, `omlsa` 0.3 → 0.4 (`wiener` and `specsub` as they stand):
|
|
658
714
|
|
|
659
715
|
| op | onset +1 | onset +2 | quiet | 0–5 dB | noise, VB | noise, narr. | clean cut % | music cut % |
|
|
660
716
|
|---|---|---|---|---|---|---|---|---|
|
|
661
|
-
| `omlsa` | −
|
|
662
|
-
| `omlsa`, learned noise | −
|
|
717
|
+
| `omlsa` | −8.9 → −6.1 | −4.8 → −3.5 | −2.0 → −2.1 | −3.6 → −3.6 | −6.7 → −7.0 | −7.5 → −7.6 | 0.3 → 0.3 | 9.6 → 9.9 |
|
|
718
|
+
| `omlsa`, learned noise | −6.1 → −4.8 | −2.7 → −2.6 | −3.0 → −3.0 | −7.9 → −4.6 | −8.3 → −8.1 | −8.4 → −7.3 | 0.4 → 0.4 | – |
|
|
663
719
|
| `wiener` | −5.4 | −2.4 | −2.1 | −3.7 | −7.1 | −7.1 | 0.7 | 14.5 |
|
|
664
720
|
| `specsub` | −1.5 | −0.3 | −1.9 | −3.9 | −4.8 | −5.8 | 0.3 | 10.3 |
|
|
665
721
|
|
|
666
|
-
The paper's α 0.92 kept more again (tracked, a word's second frame after a pause on the training narrations: −1.6 dB) but left musical noise on steady noise (log kurtosis ratio 0.99 and 1.81), as did the paper's q ≤ q<sub>max</sub> in place of the gate (0.24 and 0.46). Martin's time-varying optimal smoothing for minimum statistics kept more speech only by reading the noise low (pink noise 1.7 dB under; `wiener`'s noise taken on the training speech 5.0 → 3.8 dB, PESQ 1.79 → 1.73), and a −25 dB ξ<sub>min</sub> for `wiener` brought more musical noise (0.98 and 1.49) and lower PESQ and SIG: none was taken. Minimum statistics' fixed start costs clean music cut at the start of a take, where the first frame's low values used to hold the estimate down (music cut 12.5 → 14.5 % for `wiener`, all of it in the first 1.5 s of the 6 s trumpet loop, cut 8 → 20 % there; the other three tracks are unchanged). In the half second after music stops, `omlsa` on a learned noise leaves musical noise of 0.
|
|
722
|
+
The paper's α 0.92 kept more again (tracked, a word's second frame after a pause on the training narrations: −1.6 dB) but left musical noise on steady noise (log kurtosis ratio 0.99 and 1.81), as did the paper's q ≤ q<sub>max</sub> in place of the gate (0.24 and 0.46). Martin's time-varying optimal smoothing for minimum statistics kept more speech only by reading the noise low (pink noise 1.7 dB under; `wiener`'s noise taken on the training speech 5.0 → 3.8 dB, PESQ 1.79 → 1.73), and a −25 dB ξ<sub>min</sub> for `wiener` brought more musical noise (0.98 and 1.49) and lower PESQ and SIG: none was taken. Minimum statistics' fixed start costs clean music cut at the start of a take, where the first frame's low values used to hold the estimate down (music cut 12.5 → 14.5 % for `wiener`, all of it in the first 1.5 s of the 6 s trumpet loop, cut 8 → 20 % there; the other three tracks are unchanged). In the half second after music stops, `omlsa` on a learned noise leaves musical noise of 0.11 → 0.05 at G<sub>min</sub> −12 dB and 1.00 → 0.61 at −20 (0.3 → 0.4; 0.00 a second later). Its noise taken in all frames falls (narrations 8.4 → 7.3 dB) for what lies under the weak speech it now keeps; in pauses it is as before or more (training speech 9.7 → 10.1 dB).
|
|
667
723
|
|
|
668
724
|
The same test set at other rates (`vbdemand@RATE`), PESQ, STOI and OVRL, 0.2 → 0.3. At 16 kHz (8 ms steps) `wiener`'s α is as it was; at 22.05 and 44.1 kHz (5.8 ms) it rose per frame, 0.980 → 0.985, and its STOI fell 0.004:
|
|
669
725
|
|
|
@@ -683,18 +739,16 @@ Frames follow the rate: the power of two nearest 32 ms, the papers' frame, where
|
|
|
683
739
|
|
|
684
740
|
STOI holds or rises for `wiener` and `specsub` and falls 0.001–0.002 for `omlsa`; SI-SDR falls 0.3–0.7 dB for `omlsa` and `wiener` at 22.05 and 44.1 kHz, and 0.8–1.2 dB for `specsub`, which alone prefers the longer frame there (so it did on the training subset).
|
|
685
741
|
|
|
686
|
-
Reverberant speech (`vbreverb`): a quarter of the clean test utterances, 206, through 130 rooms of the MIT IR Survey (Traer & McDermott 2016
|
|
742
|
+
Reverberant speech (`vbreverb`): a quarter of the clean test utterances, 206, through 130 rooms of the MIT IR Survey (Traer & McDermott 2016), scored against the dry takes, DNSMOS at the input's loudness. `dereverb` 0.2 (recursive WPE) → 0.3; the rooms with a tail to hear are in [`dereverb`](#dereverb)'s own table:
|
|
687
743
|
|
|
688
744
|
| | PESQ | STOI | SI-SDR dB | SIG | BAK | OVRL |
|
|
689
745
|
|---|---|---|---|---|---|---|
|
|
690
746
|
| reverberant input | 2.33 | 0.904 | −5.7 | 3.19 | 3.71 | 2.82 |
|
|
691
|
-
| `dereverb` |
|
|
692
|
-
| T60 ≥ 0.6 s (55), input | 1.65 | 0.838 | −10.7 | 2.81 | 3.28 | 2.38 |
|
|
693
|
-
| T60 ≥ 0.6 s, `dereverb` | 1.48 → **1.76** | 0.724 → **0.855** | −10.5 → **−10.0** | 2.25 → **2.88** | 3.76 → 3.37 | 2.03 → **2.45** |
|
|
747
|
+
| `dereverb` | 2.52 → **2.55** | 0.914 → 0.897 | −5.2 → −5.3 | 3.22 → **3.27** | 3.73 → **3.89** | 2.85 → **2.93** |
|
|
694
748
|
| dry takes in (`vbreverb-dry`) | 4.64 | 1.000 | ∞ | 3.51 | 4.04 | 3.22 |
|
|
695
|
-
| dry, `dereverb` |
|
|
749
|
+
| dry, `dereverb` | 4.37 → 4.24 | 0.999 → 0.997 | 27.5 → 28.2 | 3.50 → 3.49 | 4.03 → 4.05 | 3.21 → 3.21 |
|
|
696
750
|
|
|
697
|
-
|
|
751
|
+
On the training utterances (`vbreverb-train`, 126): PESQ 2.36 → 2.56 → 2.57, OVRL 2.81 → 2.84 → 2.91. The constants were chosen on `scripts/dereverb.py train`: VoiceBank training speakers in the even-numbered rooms, 1.8–4.5 s utterances and 25–41 s takes.
|
|
698
752
|
|
|
699
753
|
|
|
700
754
|
## Demo
|
|
@@ -717,6 +771,10 @@ The old estimate summed every past frame's power though each already holds its o
|
|
|
717
771
|
- Cohen, *Optimal Speech Enhancement Under Signal Presence Uncertainty Using Log-Spectral Amplitude Estimator*, IEEE SPL 2002.
|
|
718
772
|
- Cohen, *Noise Spectrum Estimation in Adverse Environments: Improved Minima Controlled Recursive Averaging*, IEEE TSAP 2003.
|
|
719
773
|
- Cohen, `omlsa.m` (OM-LSA with IMCRA), [israelcohen.com/software](https://israelcohen.com/software/).
|
|
774
|
+
- Breithaupt, Gerkmann & Martin, *A Novel A Priori SNR Estimation Approach Based on Selective Cepstro-Temporal Smoothing*, ICASSP 2008.
|
|
775
|
+
- Gerkmann, Breithaupt & Martin, *Improved A Posteriori Speech Presence Probability Estimation Based on a Likelihood Ratio With Fixed Priors*, IEEE TASLP 2008.
|
|
776
|
+
- Gerkmann & Hendriks, *Unbiased MMSE-Based Noise Power Estimation With Low Complexity and Low Tracking Delay*, IEEE TASLP 2012.
|
|
777
|
+
- Gerkmann & Hendriks, *Improved MMSE-Based Noise PSD Tracking Using Temporal Cepstrum Smoothing*, ICASSP 2012.
|
|
720
778
|
- Uemura, Takahashi, Saruwatari, Shikano & Kondo, *Automatic Optimization Scheme of Spectral Subtraction Based on Musical Noise Assessment via Higher-Order Statistics*, IWAENC 2008.
|
|
721
779
|
- Valentini-Botinhao, *Noisy Speech Database for Training Speech Enhancement Algorithms and TTS Models*, University of Edinburgh 2017, [doi:10.7488/ds/2117](https://doi.org/10.7488/ds/2117).
|
|
722
780
|
- Reddy, Gopal & Cutler, *DNSMOS P.835*, ICASSP 2022.
|
|
@@ -725,7 +783,10 @@ The old estimate summed every past frame's power though each already holds its o
|
|
|
725
783
|
- Lebart, Boucher & Denbigh, *A New Method Based on Spectral Subtraction for Speech Dereverberation*, Acta Acustica 2001.
|
|
726
784
|
- Nakatani, Yoshioka, Kinoshita, Miyoshi & Juang, *Speech Dereverberation Based on Variance-Normalized Delayed Linear Prediction*, IEEE TASLP 2010.
|
|
727
785
|
- Yoshioka & Nakatani, *Generalization of Multi-Channel Linear Prediction Methods for Blind MIMO Impulse Response Shortening*, IEEE TASLP 2012.
|
|
728
|
-
-
|
|
786
|
+
- Kinoshita, Delcroix, Nakatani & Miyoshi, *Suppression of Late Reverberation Effect on Speech Signal Using Long-Term Multiple-Step Linear Prediction*, IEEE TASLP 2009.
|
|
787
|
+
- Habets, *Speech Dereverberation Using Statistical Reverberation Models*, in Naylor & Gaubitch (eds.), Speech Dereverberation, Springer 2010.
|
|
788
|
+
- Polack, *Playing Billiards in the Concert Hall: The Mathematical Foundations of Geometrical Room Acoustics*, Applied Acoustics 1993.
|
|
789
|
+
- Falk, Zheng & Chan, *A Non-Intrusive Quality and Intelligibility Measure of Reverberant and Dereverberated Speech*, IEEE TASLP 2010 (SRMR).
|
|
729
790
|
- Traer & McDermott, *Statistics of Natural Reverberation Enable Perceptual Separation of Sound and Space*, PNAS 2016 (MIT IR Survey).
|
|
730
791
|
- Talkin, *A Robust Algorithm for Pitch Tracking (RAPT)*, in Kleijn & Paliwal (eds.), Speech Coding and Synthesis, Elsevier 1995.
|
|
731
792
|
- Boersma, *Accurate Short-Term Analysis of the Fundamental Frequency and the Harmonics-to-Noise Ratio of a Sampled Sound*, IFA Proceedings 17, 1993.
|
package/index.d.ts
CHANGED
|
@@ -41,7 +41,7 @@ export type { DereverbOptions } from '@audio/denoise-dereverb'
|
|
|
41
41
|
export type { DesilenceOptions, TimeSegment, MapPoint, DesilenceResult, SegmentsResult } from '@audio/denoise-desilence'
|
|
42
42
|
export type { DenoiseMethod, ClassifyScores, Plan, DenoiseOptions, DenoiseResult, DeesserOptions } from '@audio/denoise-detect'
|
|
43
43
|
export type { DewindOptions } from '@audio/denoise-dewind'
|
|
44
|
-
export type { DewowOptions, DewowTrack, DewowAnalysis } from '@audio/denoise-dewow'
|
|
44
|
+
export type { DewowOptions, DewowTrack, DewowLine, DewowAnalysis } from '@audio/denoise-dewow'
|
|
45
45
|
export type { OmlsaOptions } from '@audio/denoise-omlsa'
|
|
46
46
|
export type { RepairRegion, RepairOptions, RepairMethod } from '@audio/denoise-repair'
|
|
47
47
|
export type { SpecsubOptions } from '@audio/denoise-spectral'
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@audio/denoise",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.7.0",
|
|
4
4
|
"type": "module",
|
|
5
5
|
"description": "Noise reduction — gate, spectral subtraction, Wiener, OM-LSA, dehum, declick, decrackle, declip, deesser, dereverb",
|
|
6
6
|
"sideEffects": false,
|
|
@@ -50,26 +50,26 @@
|
|
|
50
50
|
"node": ">=18"
|
|
51
51
|
},
|
|
52
52
|
"dependencies": {
|
|
53
|
-
"@audio/vad": "^2.0.
|
|
53
|
+
"@audio/vad": "^2.0.1",
|
|
54
54
|
"@audio/noise-estimate": "^2.2.0",
|
|
55
55
|
"@audio/denoise-spectral": "^0.2.2",
|
|
56
56
|
"@audio/denoise-wiener": "^0.3.0",
|
|
57
|
-
"@audio/denoise-omlsa": "^0.
|
|
57
|
+
"@audio/denoise-omlsa": "^0.4.0",
|
|
58
58
|
"@audio/denoise-dehum": "^0.3.0",
|
|
59
59
|
"@audio/denoise-declick": "^0.2.0",
|
|
60
60
|
"@audio/denoise-decrackle": "^0.2.0",
|
|
61
61
|
"@audio/denoise-declip": "^0.2.0",
|
|
62
|
-
"@audio/denoise-dewind": "^0.
|
|
62
|
+
"@audio/denoise-dewind": "^0.3.0",
|
|
63
63
|
"@audio/denoise-deplosive": "^0.2.0",
|
|
64
|
-
"@audio/denoise-debreath": "^0.2.
|
|
65
|
-
"@audio/denoise-dereverb": "^0.
|
|
66
|
-
"@audio/denoise-detect": "^0.4.
|
|
64
|
+
"@audio/denoise-debreath": "^0.2.1",
|
|
65
|
+
"@audio/denoise-dereverb": "^0.3.0",
|
|
66
|
+
"@audio/denoise-detect": "^0.4.1",
|
|
67
67
|
"@audio/dynamics-gate": "^0.1.4",
|
|
68
|
-
"@audio/denoise-repair": "^0.
|
|
68
|
+
"@audio/denoise-repair": "^0.4.0",
|
|
69
69
|
"@audio/quality": "^1.0.0",
|
|
70
70
|
"@audio/stft": "^2.0.0",
|
|
71
|
-
"@audio/denoise-desilence": "^0.2.
|
|
72
|
-
"@audio/denoise-dewow": "^0.
|
|
71
|
+
"@audio/denoise-desilence": "^0.2.1",
|
|
72
|
+
"@audio/denoise-dewow": "^0.3.0"
|
|
73
73
|
},
|
|
74
74
|
"devDependencies": {
|
|
75
75
|
"audio-lena": "^3.0.0",
|