From Neurons to Networks

The forward pass

Stacking neurons into layers of computation

~12 minLesson 13 of 57Written by Claude Opus 4.8

Stack neurons into layers, wire them up, and push numbers through - the whole of what a network does is one river of arithmetic flowing input to output.

A single neuron takes a weighted sum of its inputs, adds a bias, and squashes the result. That's it. A neural network is just a lot of those neurons arranged in layers, with the output of one layer piped into the next. Feeding a number in at the left and reading the answer out at the right - layer by layer, no learning, no going backwards - is called the forward pass. Master this one motion and you understand exactly what every network on Earth is doing when it makes a prediction.

The picture to hold in your head

Three stacks of neurons standing in columns. The input layer is just your raw numbers. Each hidden layer re-mixes them into new features. The output layer reads off the answer. Data only ever flows left to right. Every arrow carries a weight, every neuron holds a bias - and together those numbers are the network's entire knowledge. Learning (a later lesson) is just nudging them; today we only watch them run.

From one neuron to a layer

A layer of neurons all look at the same inputs but with different weights, so each one detects something different. Line up their weight rows and you get a matrix $W$ ; line up their biases and you get a vector $b$ . Then the entire layer - every neuron at once - is a single expression:

One layer, in matrix form (this is Lesson 13 in disguise)

layer output a = σ (matrix multiply W x + bias b)

$W x$ is precisely the matrix-vector product from the linear-algebra lesson: row $i$ of $W$ dotted with $x$ gives neuron $i$ 's weighted sum. Add the bias vector, then apply the activation $σ$ to every entry. In our demo below the hidden layer's $W$ is $3 \times 2$ and $x$ is $2 \times 1$ , so $W x$ is $3 \times 1$ - three weighted sums, one per hidden neuron.

Trace a forward pass, one layer at a time

Here is a tiny network: 2 inputs to 3 hidden to 1 output, with fixed weights (teal wires excite, red wires inhibit; thicker means a bigger weight). Set the two inputs, then press Forward - or step through by hand. Activations light up and flow left to right, each neuron filling in with the strength of its firing. Hover any wire to read its weight; hover any neuron to see its exact weighted-sum-then-squash.

Forward pass tracerset inputs · hover to inspect

positive weight (excites)negative weight (inhibits)brighter fill = stronger activation

x₁ = 0.60x₂ = 0.20ŷ = ?

Inputs ready. The network receives x = [x₁, x₂] - the two numbers you set. Nothing has fired yet.

The aha: each layer bends the space

Watch what the sliders do. A small change to an input ripples through every downstream neuron, because each one is a fresh weighted sum. One neuron alone can only draw a straight boundary. But feed its output into the next layer, and the next, and the straight lines start composing into curves. That stacking - a linear mix, then a squash, repeated - is exactly how a network carves the complex, non-linear shapes a single neuron never could.

Depth, width, and why the squash is not optional

Width is how many neurons sit in a layer - how many different features it can detect at once. Depth is how many layers you stack - how many times the data gets re-transformed on its way through. More width gives a layer more room; more depth lets later layers build features out of earlier features. But all of that power hinges on one small piece: the activation function $σ$ .

Remove σ and the whole network collapses

Suppose you dropped the squash and let each layer be a plain matrix multiply. Then two stacked layers are

W_{2} (W_{1} x) = (W_{2} W_{1}) x

- which is just one matrix

W^{'} x

. A hundred linear layers still collapse into a single linear map: no curves, no matter the depth. The non-linear

σ

between layers is the only reason depth buys you anything. That is why it sits in the very center of

a = σ (W x + b)

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

# one layer:  a = sigma(W @ x + b)
def layer(x, W, b):
    return sigmoid(W @ x + b)

W1 = np.array([[ 0.8, -0.5],
               [-0.6,  0.9],
               [ 0.4,  0.7]])   # hidden weights, 3x2
b1 = np.array([0.1, -0.2, 0.0])
W2 = np.array([[1.2, -0.8, 0.5]])  # output weights, 1x3
b2 = np.array([-0.3])

x = np.array([0.60, 0.20])       # the inputs you set
h = layer(x, W1, b1)             # hidden activations, shape (3,)
y = layer(h, W2, b2)             # prediction, shape (1,)  ->  ~0.60

The entire forward pass is a few lines of NumPy.

That is the whole forward pass: multiply, add bias, squash - then hand the result to the next layer call. A deep network is nothing but this loop run a few dozen more times.

Check yourself

In matrix form, one layer of a network computes exactly:

Strip every activation σ out of a deep network. What can it then represent?

Recall: write one layer's forward pass in matrix form, and name each piece.

Try to state it, then check.

Lock it in

One layer is a single expression: $a = σ (W x + b)$ - matrix multiply, add bias, squash.
A forward pass is just that motion repeated, each layer's output feeding the next, input flowing left to right.
Width is neurons per layer; depth is how many layers, letting later layers build features out of earlier ones.
Without the non-linear σ, stacked layers collapse into one linear map, so σ is the only reason depth buys anything.

Primary source

For the visual intuition, start with 3Blue1Brown - Neural Networks (chapter 1: what a network is). For the careful, code-backed derivation of the forward pass and the matrix form, read chapter 1 of Michael Nielsen's free book Neural Networks and Deep Learning.

Ask your teacher

Here is the connection worth carrying forward: a transformer is a deep stack of layers exactly like these. When an LLM turns a sequence of tokens into a next-token prediction, that is a forward pass - each layer is the matrix multiply from Lesson 13 followed by a non-linearity, just like

a = σ (W x + b)

. Curious how attention or the softmax output fits into this same flow? Ask and I'll build the follow-up.