RNNs and LSTMs, the pre-transformer world

Before attention, models read a sentence strictly left to right through a single evolving memory - and the further back a word sat, the more surely it was forgotten.

For the decade before Transformers, a neural network that read text had no way to see a whole sentence at once. It processed one token at a time, folding each new word into a single fixed-size hidden state - a running summary of everything seen so far - and passing that summary forward to the next step. This is a recurrent neural network (RNN). The design is elegant and the failure is famous: by the time the model reaches the end of a long sentence, the beginning has faded almost to nothing, and the learning signal that should reconnect them vanishes on the way back. LSTMs patched the wound; attention removed it entirely. This lesson is the "before" picture that makes attention feel inevitable.

One token at a time: the hidden state

Give the model a vocabulary vector $x_t$ for the token at position $t$ and a hidden state $h_{t-1}$ carried in from the step before. The cell mixes them, squashes the result through a $\tanh$ so values stay in $[-1,1]$, and emits $h_t$. That $h_t$ is both the output at step $t$ and the memory handed to step $t+1$. Start with $h_0=0$ and roll forward: $h_1$ knows the first word, $h_2$ knows the first two, and $h_T$ is supposed to know the whole sentence - all compressed into one vector of, say, a few hundred numbers. Nothing about the architecture reserves space for word 1 once word 40 arrives; older information simply gets overwritten as the state churns.

Learning across time: BPTT and the fading signal

To train the loop we unroll it into a deep feed-forward network - one layer per time step, all sharing the same weights - and run ordinary backpropagation along it. This is backpropagation through time (BPTT). The catch lives in the chain rule. To learn how an early word should influence a late prediction, the gradient must travel back across every step in between, and at each hop it is multiplied by the same recurrent Jacobian $\partial h_t/\partial h_{t-1}=\mathrm{diag}({\tanh }^{\prime })W_h$. Over $k$ steps that is roughly the factor $\lambda$ raised to the $k$: if $\lambda <1$ the gradient shrinks geometrically toward zero - it vanishes; if $\lambda >1$ it blows up - it explodes. Watch it happen.

LSTMs & GRUs: a memory that survives

The fix, from Hochreiter & Schmidhuber, is to stop overwriting the memory on every step and instead gate it. An LSTM adds a second track, the cell state $c_t$, that is updated additively rather than by a fresh matrix multiply: a forget gate $f_t$ decides how much of the old cell to keep, and an input gate $i_t$ decides how much new content $g_t$ to write. $c_t=f_t\odot c_{t-1}+i_t\odot g_t$. The magic is the derivative along this track: $\partial c_t/\partial c_{t-1}=f_t$, so whenever the network learns to hold a gate open ($f_t\approx 1$) the local factor is $\approx 1$ and the gradient neither shrinks nor grows - a constant error carousel that carries the signal across hundreds of steps. A GRU is a lighter cousin with two gates instead of three and no separate cell state, but the same additive-highway idea. Toggle LSTM gates in the demo and re-run the backprop: the green pulse now reaches the first word almost intact.

import numpy as np

def rnn_step(h, x, Wh, Wx, b):
    return np.tanh(Wh @ h + Wx @ x + b)   # mix old memory h with new token x

h = np.zeros(D)
for x in sequence:                         # strictly one token at a time (sequential!)
    h = rnn_step(h, x, Wh, Wx, b)          # h carries everything seen so far

# BPTT multiplies the recurrent Jacobian once per step:
#   dL/dh_0  ~  prod_t ( diag(1 - h_t**2) @ Wh )     # tanh' = 1 - h**2
#   factor norm < 1  ->  gradient VANISHES over long spans
#   factor norm > 1  ->  gradient EXPLODES

# LSTM swaps the raw product for a gated, ADDITIVE cell state:
c = f * c_prev + i * g          # f = forget gate; f ~ 1  =>  dC_t/dC_(t-1) ~ 1
                                # a near-constant highway  =>  the signal survives

The bottleneck Transformers escaped

Even a perfectly-tuned LSTM has one flaw it cannot gate away: step $t$ cannot begin until step $t-1$ is done. The recurrence is inherently sequential, so a thousand-token document is a thousand dependent steps - you cannot parallelize across the sequence, and training crawls. Attention (Lesson 31) breaks both curses at once: it lets every token look directly at every other token in a single parallel operation, with a constant-length path between any two positions. No loop, no recurrent Jacobian, no vanishing gradient, and the whole sequence computed at once on a GPU. Everything you just felt struggling here is exactly what attention was invented to fix.

Check yourself