How AI Music Source Separation Works (and Why It Got So Good)

The problem: un-mixing a song

A mix is the sum of many sounds. Recovering the parts from the sum is mathematically "under-determined": countless combinations of signals could add up to the same result. In the famous "cocktail party problem", people solve this effortlessly, picking out one voice in a noisy room, because the brain knows what voices sound like. Separation algorithms need that same kind of prior knowledge.

Before deep learning

Early approaches relied on hand-designed assumptions:

• Stereo position: vocals are usually centred, so subtracting one channel from the other cancels them, along with the bass and drums.
• Repetition: methods such as REPET assumed the accompaniment repeats while the voice varies.
• Matrix factorisation: non-negative matrix factorisation (NMF) modelled a spectrogram as a combination of recurring spectral patterns.
• Harmonic/percussive splitting: sustained tones form horizontal lines on a spectrogram and drum hits form vertical ones, which median filters can separate.

These ideas work in specific cases but break easily, leaving the "watery" artefacts many people associate with old vocal removers.

The deep-learning shift

Neural networks replaced hand-written rules with rules learned from data. Researchers built datasets of songs with their isolated stems. MUSDB18 (150 tracks split into vocals, drums, bass and other) became the standard benchmark. A model sees the mix, guesses the stems, compares the guess with the truth, and adjusts millions of internal parameters to do better next time.

Key open models along the way:

Two ways to see sound: masks and waveforms

Spectrogram masking models turn audio into a time-frequency grid with the short-time Fourier transform, then predict a "mask" for each source. That's a value per grid cell saying how much of that cell belongs to the vocals, the drums, and so on. Multiplying the mix by each mask and converting back gives the stems.

Waveform models work on the raw sample values directly. They're better at keeping phase and sharp attacks intact, so drums stay punchy, but they have to learn frequency structure themselves.

HTDemucs, which powers this site's Vocal Remover and Stem Splitter, does both. One U-Net branch processes the waveform and another processes the spectrogram. In the middle, a transformer lets the two branches attend to each other across several seconds of music, so the model can use context such as "this phrase was sung a bar ago". The outputs of both branches are added together for each stem.

How quality is measured

Researchers report the signal-to-distortion ratio (SDR): how much of the estimated stem is the true source compared with everything else (bleed from other instruments plus artefacts), in decibels. Every additional ~3 dB roughly halves the relative error energy. HTDemucs scores around 9 dB on average on the MUSDB18-HQ benchmark, and the latest RoFormer systems report higher scores, especially for vocals.

SDR is useful, but ears matter more. Two stems with the same score can sound different: one with faint bleed, another with "underwater" artefacts. That's why it's worth listening to your results in context.

Running AI in a web browser

Until recently, separation meant installing Python tools or uploading songs to a cloud service. Two web technologies changed that:

• ONNX, a portable format for neural networks, lets a model trained in PyTorch run in other environments, including browsers via ONNX Runtime Web.
• WebGPU gives web pages access to the graphics card, which performs the matrix multiplications neural networks need far faster than the processor can.

Full Audio Tools downloads the HTDemucs model once (about 180 MB), stores it in your browser, and runs it on your own GPU. Your music never leaves your device. The trade-off is that very large models, such as the biggest RoFormer variants, are still too heavy for most browsers.

Current limits

• Overlapping sounds (a sax and a voice on the same note) remain hard to split cleanly.
• Low-bitrate sources contain compression artefacts that confuse models.
• Stem definitions are fixed by the training data: four-stem models put guitars and keys together in "other".

Frequently asked questions