Encoder-decoder and the fixed-vector bottleneck

The whole point of attention, seen through the problem it was built to kill: every sentence - five words or fifty - forced through one fixed-length vector.

In 2014 the first neural machine translators worked in two moves. An encoder RNN read the source sentence one word at a time and boiled it down to a single fixed-length vector - its final hidden state. A decoder RNN then wrote the translation one word at a time, with nothing to go on but that one vector. It was elegant, it beat the phrase-based systems of its day - and it had a fatal flaw. Every sentence, however long, had to squeeze through the same fixed vector. That funnel is the bottleneck this lesson is about, and removing it is exactly what attention (Lesson 31) was invented to do.

The encoder-decoder pipeline

Two networks, one handoff. The encoder steps through the source tokens $x_1,\dots ,x_n$, updating a hidden state each step, $h_t=f(h_{t-1},x_t)$. When it reaches the end it stops and keeps only the last state. That final vector - often called the context vector or thought vector - is the entire handoff:

The decoder is a second RNN that starts from $c$ and emits target words one at a time, each conditioned on $c$ and everything written so far, $s_t=g(s_{t-1},y_{t-1},c)$. It factorises the probability of the whole output sentence as a product of per-word predictions:

Notice that $c$ is the only thing carrying the source into every term of that product. During training we use teacher forcing: instead of feeding the decoder its own (possibly wrong) previous guess, we feed the true previous target word $y_{t-1}$. That keeps early mistakes from snowballing while the model is still learning - at inference time it has to fall back on its own outputs.

# ENCODER: read the whole source, keep only the final hidden state.
h = zeros(d)                       # the "thought vector" - a FIXED size d
for x in source_tokens:            # x_1 ... x_n, any length
    h = rnn_step(h, embed(x))      # everything must fit inside this one h
context = h                        # <-- the entire sentence, squeezed into d numbers

# DECODER: write the target, conditioned ONLY on that one vector.
s = context                        # start from the squeezed summary
y_prev = "<START>"
for t in range(target_len):
    s = rnn_step(s, embed(y_prev))
    logits = W @ s                 # a distribution over the vocabulary
    guess  = argmax(logits)
    y_prev = target[t] if training else guess   # teacher forcing feeds the TRUE word

The fixed-vector bottleneck

Here is the crack. The context vector $c$ has a fixed size $d$ chosen once, at build time. A five-word sentence and a fifty-word sentence both get exactly $d$ numbers. But the amount of meaning to carry grows with length, so a long sentence has to overwrite detail to fit - and because the encoder reads left to right, it is the start of the sentence that fades first. Sutskever, Vinyals & Le measured exactly this: translation quality (BLEU) held up on short inputs and slid on long ones. Their now-famous hack - reverse the source words before encoding - shortened the path from the first source words to $c$ and bought several points of BLEU. A hack that helps is a bottleneck confirmed.

Drag the slider below to grow the sentence. The fixed vector in the middle never changes size - watch more and more words get forced through its same handful of slots while the reconstruction fidelity meter falls.

Check yourself