AI Noise Removal

For four decades, noise reduction meant statistics: measure the hiss, then subtract it. That works until a dog barks. The current generation of noise removal takes the opposite approach — instead of modelling your noise, a neural network has learned what a human voice is, and it keeps that while discarding everything else. Here's how the field got there, and what actually runs inside the model.

The wall classical noise reduction hits

Classical spectral noise reduction — the family of techniques descended from spectral subtraction work of the late 1970s — is built on one load-bearing assumption: noise is stationary. The algorithm listens during the gaps in your speech, builds a statistical profile of what the background sounds like in each frequency band, and then subtracts that profile from everything. For the noises the assumption fits — fan hum, air conditioning, electrical hiss, the steady roar of a road — it works remarkably well, and cheaply.

The assumption breaks the moment noise stops being predictable. A bark, a dropped fork, a slammed door, keyboard clatter, a television in the next room, a second person talking — none of these were in the profile the algorithm learned two seconds ago, so they pass straight through, often sounding more conspicuous against the newly cleaned background. Worse, if a non-stationary sound happens to land during a speech gap, it contaminates the noise estimate, and the subtraction starts eating your voice instead. This isn't a tuning problem. It's a structural limit: a system that only knows what your noise looked like a moment ago cannot handle noise that refuses to repeat itself.

The neural turn: 2018–2023

The escape route was to stop modelling the noise and start modelling the speech. The landmark demonstration that this could run in real time on ordinary hardware was RNNoise, published by Jean-Marc Valin in 2018. RNNoise was deliberately a hybrid: a conventional DSP front end computed per-band signal features, and a tiny recurrent neural network — small enough to run comfortably on a single CPU core — predicted how much gain to apply in each band, frame by frame. It wasn't the biggest or best model imaginable; it was proof that learned speech enhancement fit inside a real-time budget.

Commercial products followed quickly. Krisp built a business on machine-learned noise removal for calls, running the model on the user's own machine, and NVIDIA's RTX Voice (2020) put GPU-powered denoising in front of millions of gamers and streamers. By the time remote work became the default, "the AI noise cancellation" had gone from research demo to checkbox feature.

The open-source state of the art then advanced again with DeepFilterNet, developed by Hendrik Schröter and colleagues and published across 2022–23. DeepFilterNet is perceptually motivated: it groups the spectrum into ERB bands that mirror the frequency resolution of human hearing, applies learned gains there, and then adds a second mechanism called deep filtering to reconstruct the fine spectral detail of the voice. It runs in real time on a CPU, and it's open source — which is why it, rather than a proprietary black box, is what you'll find embedded in a growing number of applications.

Separation, not subtraction

The conceptual shift matters more than the architecture. A classical reducer removes an estimate of the noise; a neural enhancer keeps an estimate of the speech. During training, the model hears thousands of hours of clean voice recordings mixed with an enormous variety of noises — traffic, babble, machinery, music, rain — and is graded on how well it recovers the clean voice from the mixture. What it internalises is not a catalogue of noises but the deep regularities of speech itself: harmonic structure that moves the way pitch moves, formants that sit where vocal tracts put them, energy that starts and stops on syllable rhythms.

That's why the failure modes are inverted. The classical reducer fails on any noise it hasn't profiled; the neural model handles novel noise easily but is most challenged by sounds that resemble its target. A generic model has learned to preserve human speech, not your speech — so a second voice in the room is passed through faithfully, whether you wanted it or not. Solving that requires a further, personalised model, which is its own research area.

How it works under the hood

Real-time speech enhancement is frame-based. The incoming stream — typically at a 48 kHz sample rate — is chopped into short overlapping frames, on the order of 10 ms per hop, and transformed into the frequency domain. Rather than asking the network to reason about a thousand raw frequency bins, models like DeepFilterNet first collapse them into a few dozen perceptual bands, which slashes the compute and matches how the ear actually resolves frequency. Recurrent layers carry memory across frames, which is how the model distinguishes a consonant burst (part of a syllable in progress) from a click (an isolated transient) — context that no single frame contains.

Per-band gains alone leave speech sounding slightly hollow, because a coarse band can't separate a harmonic from noise sitting between harmonics. Deep filtering addresses this by predicting short complex-valued filters over a small time–frequency neighbourhood, letting the model rebuild the voice's fine structure instead of merely attenuating around it. The whole pipeline is causal or nearly so, with at most a few frames of look-ahead — the price of which is a fixed, known amount of latency.

On-device or in the cloud

There are two places a model like this can run, and the trade is stark. In the cloud, you can deploy models of any size — but every second of your microphone leaves your machine, the round trip adds latency you can't engineer away, and someone is paying per minute of inference forever. On-device, the model must fit a real-time CPU budget, but the latency is fixed and small, it works on a train with no signal, and the audio never goes anywhere. It's telling that the systems which shaped this field — RNNoise, Krisp, DeepFilterNet — were all engineered for local execution. For a microphone, the private, predictable option is also the practical one.

In DeskBroadcast

DeskBroadcast embeds DeepFilterNet 3 — an open model, MIT/Apache-2.0 licensed — directly inside the app binary. It runs fully on-device: nothing leaves the Mac. The model operates at 48 kHz on 480-sample (10 ms) frames with roughly 40 ms of end-to-end latency. When you enable it, it replaces the classical spectral stage and the click suppressor rather than stacking on top of them — the latency budget stays the same, and you never pay twice for overlapping jobs. Measured on real speech, the voice passes at −0.2 dB while broadband noise drops 45–49 dB, and noise in the gaps around speech drops about 40 dB. The strength slider maps to the model's maximum-attenuation cap, so lower settings retain a little natural room sound rather than running a weaker model. And the classical reducer remains available as a toggle if you prefer the old-school approach for steady, well-behaved noise.