Architectures Before Transformers

Convolutional neural networks

Sharing weights across space for vision

~12 minLesson 20 of 57Written by Claude Opus 4.8

A photo is a grid of numbers, and the thing that makes it a cat is local: an ear is a small arrangement of nearby pixels, not a relationship between the top-left and bottom-right corners. A fully-connected layer ignores that - it wires every pixel to every neuron and burns millions of weights re-learning "an edge is an edge" at each spot. A convolutional layer bakes the structure in: take one tiny filter, slide it everywhere, and at each stop compute a single dot product between the filter and the patch beneath it. Same nine weights, reused across the whole image. That one move - a shared local dot product - is the entire idea, and it is what powered vision for a decade.

The one-line idea

Hold a small stencil of weights (the filter or kernel) over a patch of the image, multiply overlapping numbers and add them up - one number comes out. Now slide the stencil one step and do it again. Sweep the whole image and those numbers form a new grid: a feature map that lights up wherever the filter's pattern appears. The filter is the pattern it detects.

Slide a filter, take a dot product

Line the filter up over a patch of the image the same size as itself. Multiply each filter weight by the pixel under it, sum all nine products, and write that single number into the output at the filter's current position. Slide one column right and repeat; when you reach the edge, drop to the next row. This sweep is the convolution (in deep learning we actually compute the un-flipped version, cross-correlation, but everyone calls it convolution):

S (i, j) = (I * K) (i, j) = m = 0 \sum k - 1 n = 0 \sum k - 1 I (i + m, j + n) K (m, n) .

Here $I$ is the image, $K$ is the $k \times k$ kernel, and $S (i, j)$ is one output pixel - nothing but the dot product of the kernel with the patch it currently covers. Two properties fall out immediately, and they are the whole reason CNNs work. Locality: each output only looks at a small $k \times k$ neighborhood - a pixel sees its neighbors, nothing far away. Weight sharing: the same nine numbers are reused at every position, so an edge detector learned in one corner works in all of them.

A filter is a feature detector

Change the nine weights and you change what lights up. A vertical-edge kernel subtracts the left column from the right, so it reads near zero on flat regions and spikes where brightness jumps sideways. A blur kernel averages the patch; a sharpen kernel amplifies the center against its neighbors. In a real CNN these weights are not hand-set - backpropagation learns them, discovering the edge, corner, and texture detectors that happen to reduce the loss. Below, you set them yourself. Move your pointer across the image to slide the filter and watch its output pixel compute in real time.

Sliding-kernel convolver move over the image · edit the filter · try presets

Left: an 8×8 grayscale image (a bright bar on a dark field). Right: the 6×6 feature map being built. Move your pointer over the image (or drag, on touch) to reposition the amber window; the panel below shows the exact dot product for that one output pixel. Map colours: positive negative activation.

Filter preset:

filter K (editable)

⊙

patch under window

products

-2

0

8

-4

0

16

-2

0

8

→ Σ

24output S(2,2)

S(2,2) =  (-1·2) + (0·8) + (1·8) + (-2·2) + (0·8) + (2·8) + (-1·2) + (0·8) + (1·8) = 24

And then it shrinks. Stack a convolution, then a 2×2 max-pool (keep the biggest value in each block), then convolve again. Each stage trades spatial size for depth of meaning: from many raw pixels to a few rich features:

8×8
image

conv 3×3
→

6×6
feature map

max-pool
2×2 →

3×3
pooled

conv 3×3
→

1×1
deep feature

Counting the weights CNNs save

Take a small $32 \times 32$ colour image - that is $32 \times 32 \times 3 = 3072$ input numbers. Wire it to a modest hidden layer of $1000$ units with a dense layer and you pay $3072 \times 1000 \approx 3.07$ million weights for one layer. Now a convolutional layer with $64$ filters of size $3 \times 3 \times 3$ :

one filter (3 \times 3 \times 3 + 1) \times 64 = 28 \times 64 = 1792 weights.

Roughly 1,700× fewer - and, crucially, that count does not grow when the image gets bigger, because the same filters slide over a larger canvas. Locality makes each connection cheap; weight sharing makes them few. In general a conv layer costs $(k \cdot k \cdot C_{in} + 1) \cdot C_{out}$ weights, set by the filter, not the image.

The aha: a hierarchy built from small parts

One conv layer sees only a

3 \times 3

window - a tiny receptive field. But stack layers with pooling in between (which downsamples: a

2 \times 2

max-pool halves each side,

6 \times 6 \to 3 \times 3

, keeping the strongest response and adding a little translation-tolerance), and a neuron three layers deep indirectly sees a wide swath of the original image. So early filters fire on edges, the next layer combines edges into corners and textures, the next into object parts (an eye, a wheel), and the top into whole objects. Meaning is composed bottom-up from local pieces - the same compositional trick as recursion, drawn in pixels.

import numpy as np

def conv2d(image, kernel):            # 'valid' convolution, single channel
    kh, kw = kernel.shape
    H,  W  = image.shape
    out = np.zeros((H - kh + 1, W - kw + 1))   # 8x8, 3x3 -> 6x6
    for i in range(out.shape[0]):        # slide down the rows
        for j in range(out.shape[1]):    # slide across the cols
            patch = image[i:i+kh, j:j+kw]      # the local receptive field
            out[i, j] = np.sum(patch * kernel)  # dot product = one output pixel
    return out

edge = np.array([[-1, 0, 1],
                 [-2, 0, 2],
                 [-1, 0, 1]])       # vertical-edge (Sobel) detector
feature_map = conv2d(img, edge)         # same 9 weights, every location

Valid convolution in NumPy - nine weights, reused everywhere

Check yourself

Reusing one 3×3 filter across every position of the image gives a CNN:

A 2×2 max-pool with stride 2 turns a 6×6 feature map into:

Recall what a single convolution output value is, and the two properties that make a conv layer so much cheaper than a dense one.

Try to state it, then check.

One output pixel

S (i, j) = \sum_{m, n} I (i + m, j + n) K (m, n)

is just the dot product of the filter with the local patch it covers; slide the filter to fill the whole feature map. It is cheap because of locality (each output connects only to a small

k \times k

patch - sparse) and weight sharing (the same filter is reused at every position), so the parameter count depends on filter size, not image size. Pooling then downsamples the map, growing the receptive field as you stack layers.

Lock it in

A conv layer holds one small filter (kernel), slides it everywhere, and at each stop computes a dot product with the patch beneath - the filter is the pattern it detects.
Two properties make it work: locality (each output sees only a $k \times k$ neighborhood) and weight sharing (the same weights reused at every position), so the parameter count is set by filter size, not image size.
That is orders of magnitude fewer weights than a dense layer (about $1, 700 \times$ in the worked example), and the count does not grow when the image gets bigger.
Stacking conv layers with pooling grows the receptive field, so filters compose bottom-up: edges into corners and textures, into object parts, into whole objects - and backprop learns the weights rather than hand-setting them.
CNNs bet that nearby-is-relevant (great for images); attention drops that constraint for a global receptive field from layer one, which is why transformers replaced CNNs for language.

Primary source

The canonical treatment is Stanford's CS231n - Convolutional Neural Networks for Visual Recognition; its notes derive the output-size formula

⌊(W - k + 2 p) / s ⌋ + 1

, filters, stride, padding, pooling, and parameter counts in full. For a gentler, interactive on-ramp with runnable exercises, work through Google's Machine Learning Crash Course. Between them you get both the rigor and the hands-on feel.

Ask your teacher

Here is the connection that makes the last third of this course click. Convolution's defining move is locality: a pixel only ever sees its immediate neighbors, and reaching a distant pixel requires stacking many layers so receptive fields slowly grow. Attention (Lessons 31-32) throws that constraint away - every token attends to every other token in one step, a fully global receptive field from layer one. Seeing them side by side is exactly why the difference lands: CNNs bet that nearby-is-relevant (great for images), while transformers refuse to assume where the relevant context sits - which is why they replaced CNNs for language, where a pronoun can depend on a word forty positions back. Ask me about stride and padding, why max-pool beats average-pool, dilated convolutions, or how a Vision Transformer drops convolution entirely and lets attention look at image patches instead.