From Neurons to Networks

Activation functions

The nonlinearity that makes depth matter

~10 minLesson 14 of 57Written by Claude Opus 4.8

The tiny non-linear squish inside every neuron - the one thing that lets a deep network bend, instead of quietly collapsing into a single straight line.

A neuron does two things: it takes a weighted sum of its inputs - a straight, linear operation - and then it passes that number through a small activation function that bends it. That bend is not a detail. Strip it out and something startling happens: a 100-layer network, for all its millions of weights, collapses into a single straight line. The activation is the entire reason depth buys you power.

The picture to hold in your head

Think of each layer as a sheet of stretchy graph paper. A linear layer can only slide, rotate, and stretch the sheet - every straight line stays straight. Stack a hundred such slides and the sheet is still flat: no matter the dataset, your decision boundary is one straight cut. The activation function is what lets each layer fold the sheet. Fold, then slide, then fold again, and you can wrap the paper around any shape. That folding is where curved boundaries - and real intelligence - come from.

Why a stack of lines is still one line

This isn't hand-waving; it's a one-line proof. Take two linear layers with no activation between them. The first maps input $x$ to $h = W_{1} x + b_{1}$ ; the second maps $h$ to $y = W_{2} h + b_{2}$ . Substitute the first into the second:

Two linear layers fold into one

y = W_{2} (W_{1} x + b_{1}) + b_{2} = W^{'} (W_{2} W_{1}) x + b^{'} (W_{2} b_{1} + b_{2})

The whole two-layer stack is just $y = W^{'} x + b^{'}$ - a single linear layer with new weights $W^{'}$ and bias $b^{'}$ . The same trick collapses ten layers, or ten thousand. Depth without a non-linearity is an illusion: you paid for a deep network and received one matrix multiply.

So we insert a non-linear function $ϕ$ after each layer: $h = ϕ (W_{1} x + b_{1})$ . Because $ϕ$ is not linear, you cannot fold the two multiplies into one - every layer now genuinely adds expressive power. The question becomes: which $ϕ$ ? Four choices dominate, and the demo below lets you feel each one's personality - its shape and, just as importantly, the shape of its gradient.

Four squishes and their gradients

Meet the cast. Sigmoid gently squashes any number into $(0, 1)$ . Tanh does the same into $(- 1, 1)$ but is centred on zero. ReLU - the modern workhorse - simply zeroes out negatives and leaves positives untouched. Leaky ReLU lets a whisper of the negative side through so a neuron never fully dies. Here are the exact functions and their derivatives (the derivative is what gradient descent multiplies through - watch it closely):

The four, with derivatives

σ (x) = \frac{1}{1 + e ^{- x}}, σ^{'} (x) = σ (x) (1 - σ (x)) \leq \frac{1}{4}

tanh (x) = \frac{e ^{x} - e ^{- x}}{e ^{x} + e ^{- x}}, tanh^{'} (x) = 1 - tanh^{2} (x)

ReLU (x) = max (0, x), ReLU^{'} (x) = {10 x > 0 x < 0

LeakyReLU (x) = {x α x x > 0 x \leq 0, α \approx 0.01

In the plotter, the left panel is the function; the right panel is its gradient $f^{'} (x)$ . Swap the activation with the dropdown, then drag the slider to move the input $x$ . The amber dashed tangent on the left is the slope at that point - and its steepness is exactly the height of the amber dot on the right. Slide a sigmoid out to $x = 5$ and watch the tangent go flat and the gradient collapse toward zero. That flattening is the famous vanishing gradient, and it's the reason ReLU took over.

Activation plotter: swap the function, slide x, then flip to the decision boundary.

activationinput x = 0.00

f(x) = 0.000f'(x) = 0.000

function f(x)gradient f'(x) = tangent slopedashed line at 1 for scale

Dead zone: for x ≤ 0 the gradient is exactly 0 - this unit is switched off and stops learning until its input turns positive.

Why ReLU is the default - and what "vanishing gradient" means

Training sends error signals backward through the network, and the chain rule multiplies one activation's derivative at every layer it passes. Sigmoid's derivative never exceeds

0.25

; tanh's saturates to

0

at the tails. Multiply twenty numbers each below

1

and the signal reaching the early layers is microscopic - they barely learn. This is the vanishing-gradient problem. ReLU sidesteps it: on the positive side its derivative is exactly

1

, so the signal passes through undimmed, layer after layer. That single property is why ReLU (and its cousins) unlocked genuinely deep networks. Its one wart - a neuron stuck at

x \leq 0

has gradient

0

and can "die" - is exactly what Leaky ReLU patches with a small negative slope.

The output layer speaks probability: softmax

Hidden layers squish per-neuron, but the final layer of a classifier needs to answer "how likely is each class?" - a whole vector of numbers that are all positive and sum to $1$ . That's the job of softmax, which exponentiates the raw scores $z_{i}$ (called logits) and normalizes them:

Softmax turns scores into a distribution

softmax (z)_{i} = \frac{e ^{z_{i}}}{j \sum e ^{z_{j}}}

Exponentiating keeps everything positive and exaggerates the leader; dividing by the sum forces the outputs to add to $1$ . The largest logit becomes the most probable class - and when a language model predicts the next token, this is the exact step that turns a vector of scores over the vocabulary into the probabilities it samples from.

import numpy as np

def sigmoid(x): return 1 / (1 + np.exp(-x))
def tanh(x):    return np.tanh(x)
def relu(x):    return np.maximum(0, x)
def leaky(x, a=0.01): return np.where(x > 0, x, a * x)

def softmax(z):                      # output layer: scores -> probabilities
    e = np.exp(z - z.max())          # subtract max for numerical stability
    return e / e.sum()               # positive, and sums to 1

The whole cast in a few lines of NumPy

Check yourself

Why does stacking linear layers with no activation collapse into a single layer?

For very large positive x, the sigmoid's gradient becomes:

Recall: why is a non-linear activation essential, and why is ReLU the common default?

Try to state it, then check.

Without a non-linearity, any depth of linear layers folds into a single linear map

y = W^{'} x + b^{'}

, so the network can only draw straight boundaries. A non-linearity lets each layer bend the space, so stacking layers composes complex curved boundaries. ReLU,

max (0, x)

, is the default because its gradient is exactly

1

for

x > 0

- it neither saturates nor vanishes, so error signals pass undimmed through many layers and deep nets train fast.

Primary source

Play with the same idea live in TensorFlow Playground - swap the activation on a curved dataset and watch the decision boundary morph in real time (try the two-spiral problem with Linear vs ReLU). For the deepest intuition on how these bends compose into learning, 3Blue1Brown's Neural Networks series is the gold standard.

Ask your teacher

Here's the thread that ties this to today's models: inside every transformer's feed-forward block sits an activation - usually GELU, a smooth cousin of ReLU that curves gently through the origin instead of kinking sharply. It's the same job you saw here, just polished. Sit with this: non-linearity is the entire reason a deep LLM is more expressive than one giant matrix multiply - without it, GPT-scale weights would fold, by the proof above, into a single linear map. Ask me about GELU vs ReLU, why "dead" ReLU units happen, or how softmax temperature changes how a model samples its next word.