Sound, Fourier, and DSP

Play the exact same note - say the A above middle C - on a violin, a flute, and a piano, and you can tell them apart instantly with your eyes closed. The pitch is identical. Something else is different. That something is the reason audio turns out to be, underneath, pure math, and why an equalizer, an MP3 file, and a spectrogram app all work.

A pure, featureless tone - the beep of a hearing test - is a single sine wave. But almost no real sound is a bare sine. A voice, a violin, a struck drum: their waveforms are jagged, complicated, nothing like that smooth curve. So how can a jagged sound and a smooth beep be the same kind of thing?

First, a guess

Build a sound out of sines

Each slider below is the loudness of one pure sine tone: the lowest is the fundamental, and the rest are its harmonics at two times, three times, and so on. The top panel adds them all up into one waveform. Try the presets, then drag the sliders and watch a smooth wave grow teeth.

You just did additive synthesis by hand. The jagged shapes were never anything but a handful of sines stacked up. Now run the idea backward: given a recorded sound, figure out how much of each sine is in it. That backward step is the whole game.

Fourier's claim, and two ways to look at a sound

The mathematician Joseph Fourier proved something that sounds too strong to be true: any repeating signal, however complicated, can be written as a sum of sine waves at whole-number multiples of one base frequency, each with its own amplitude.

That sum is exactly the summation you met in math, one term per harmonic. It gives you two equally true ways to hold the same sound in your head. The time domain is the wiggly waveform on top of the widget. The frequency domain is the bar chart underneath: the recipe of how much of each harmonic. The bar chart is the sound's spectrum, and moving between these two pictures is what nearly all audio software does all day.

Why audio is a branch of computer science

A microphone turns sound into a smooth voltage, but a computer cannot store a smooth anything. So it measures the voltage tens of thousands of times a second and keeps just those numbers. That is sampling. CD audio samples $44,100$ times per second, which is why an audio file is just a long array of numbers.

To find the spectrum of that array - to pull out the harmonic recipe - a computer runs the Fast Fourier Transform, or FFT: the algorithm that does the backward step, waveform to spectrum, and does it fast. Its speed is the kind of thing the Big-O lens measures - about $n\log n$ work instead of $n^2$, which is the difference between real-time audio and a stutter.

And once a sound is in the frequency domain, you can throw parts of it away on purpose. Your ears cannot hear very high frequencies, and a loud tone hides a soft one nearby. An MP3 computes the spectrum, deletes the components you would never notice, and stores only what is left. That is how a song shrinks to a tenth of its size with no obvious loss - it is the spectrum, minus the parts you cannot hear.

The same idea, in code

Building a sound from harmonics is one summation loop, exactly what the widget does each frame:

import numpy as np

sample_rate = 44100
t = np.linspace(0, 1, sample_rate, endpoint=False)   # one second of samples
f0 = 220.0                                            # fundamental: the pitch

amps = [1.0, 0.5, 0.33, 0.25, 0.2, 0.17]             # a sawtooth-ish recipe
wave = sum(a * np.sin(2 * np.pi * (n + 1) * f0 * t)  # one term per harmonic
           for n, a in enumerate(amps))

spectrum = np.abs(np.fft.rfft(wave))                 # backward step: waveform -> harmonics
# spectrum spikes at 220, 440, 660, ... Hz: the recipe reappears.

Check yourself