Map of the field

How the whole field fits together: the path from word vectors to modern LLMs in a handful of steps, each linking to the lessons that teach it, and the primary papers behind them.

The path in eight steps

1. 2013
   Words become vectors

   word2vec shows that meaning can live in geometry: train a model to predict neighboring words and directions in the vector space start to encode analogies. This is the seed idea every later layer builds on. See Word embeddings.

2. 2014 to 2015
   Sequence to sequence, then attention

   Encoder-decoder networks learn to map one sequence to another by squeezing the input into a single vector, and attention removes that bottleneck by letting the decoder look back at every input position by relevance. See Encoder-decoder, Attention, and Graphs to attention.

3. 2017
   The transformer

   Attention Is All You Need drops recurrence entirely: self-attention plus feed-forward blocks, wrapped in residuals and layer normalization, all parallelizable. This is the architecture that scaled. See Self-attention and QKV, Multi-head attention, The transformer architecture, Residuals, LayerNorm, and the FFN, and Positional encodings.

4. 2018 to 2019
   Pretraining takes over

   BERT and GPT-2 show that pretraining one big model on raw text, then reusing it, beats training task-specific models from scratch. The decoder-only, next-token recipe is the branch that kept scaling. See Language modeling, Decoder-only and GPT, and Recursion to autoregression.

5. 2020
   Scale becomes a law

   GPT-3 makes in-context learning a headline, and the scaling-laws paper shows loss falling as a smooth power law in model size, data, and compute, so the payoff of scale is predictable in advance. See Pretraining at scale and Scaling laws.

6. 2021 to 2022
   Compute-optimal and aligned

   Chinchilla corrects the field: for a fixed budget, grow model and data together, roughly 20 tokens per parameter. In parallel, RLHF turns a raw next-token model into one that follows instructions, chain-of-thought prompting unlocks multi-step reasoning, and LoRA makes tuning cheap. See Scaling laws, Fine-tuning, RLHF and DPO, and Prompting and in-context learning.

7. 2022 to 2023
   Grounded and agentic

   The model stops being the whole system. Retrieval-augmented generation feeds it real, current data; tool use and the ReAct loop let it act; and DPO simplifies alignment into a single preference loss. See Retrieval-augmented generation, Agents and tools, and RLHF and DPO.

8. Running throughout
   The efficiency thread

   Alongside the capability story runs a quieter one: making these models fit and run. Local and windowed attention beat the quadratic wall, the KV cache trades memory for speed, quantization shrinks the weights, and smarter decoding picks better tokens for less. See The Big-O attention wall, Sliding window to local attention, Memoization to the KV cache, Bit tricks to quantization, and Heaps to beam search.

The primary sources

The papers this course cites, in the order they land on the timeline. Each is the original source for one of the ideas above, worth reading in full once the lesson has prepared you for it.

1. 1.

   Sutskever, Vinyals, and Le (2014). Sequence to Sequence Learning with Neural Networks.

   The encoder-decoder that framed generation as mapping one sequence to another, and exposed the single-vector bottleneck attention would later remove.

   Taught in Encoder-decoder.

2. 2.

   Vaswani et al. (2017). Attention Is All You Need.

   The transformer. If you read one paper here, read this one: self-attention, multi-head, positional encoding, and the block that every LLM is still made of.

   Taught in The transformer architecture.

3. 3.

   Kaplan et al. (2020). Scaling Laws for Neural Language Models.

   The empirical power laws that turned scale from a guess into a plan: loss falls predictably as model, data, and compute grow.

   Taught in Scaling laws.

4. 4.

   Hoffmann et al. (2022). Training Compute-Optimal Large Language Models (Chinchilla).

   The correction to the earlier law: most big models were undertrained. For a fixed budget, grow parameters and tokens together, roughly 20 tokens per parameter.

   Taught in Scaling laws.

5. 5.

   Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models.

   Freeze the base weights, train tiny low-rank updates. The idea that made fine-tuning affordable on a single machine.

   Taught in Fine-tuning.

6. 6.

   Wei et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.

   Ask the model to show its steps and multi-step accuracy jumps, with no weight changes. The prompt-side counterpart to scaling.

   Taught in Prompting and in-context learning.

7. 7.

   Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.

   Bolt a retriever onto the model so answers come from a real, updatable store instead of only frozen weights. The blueprint behind most production LLM apps.

   Taught in Retrieval-augmented generation and Hashing to nearest neighbors.

8. 8.

   Yao et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models.

   Interleave a short thought with each tool call in a loop. The pattern that turned a text model into an agent that can look things up and act.

   Taught in Agents and tools.

9. 9.

   Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model.

   Drop the separate reward model and RL loop; train straight on preference pairs with one classification-style loss. Alignment made simpler and more stable.

   Taught in RLHF and DPO.

10. 10.

    Beltagy, Peters, and Cohan (2020). Longformer: The Long-Document Transformer.

    Windowed plus global attention brings the quadratic cost down to linear in sequence length, one of the standard ways past the attention wall.

    Taught in Sliding window to local attention.