LLMs as the engine

Every agent you will ever build wraps one component: a large language model. Strip away the tools, memory, and orchestration and what remains is a next-token predictor, a function from text to text. Build that engine from first principles and its two hard constraints - it is stateless and context-bounded - stop surprising you when you design agents.

The whole engine: predict the next token

A decoder-only Transformer does exactly one thing. It reads a list of token IDs and emits one raw score (a logit) for every entry in its vocabulary - typically 50,000 to 200,000+ possible tokens. A softmax turns those scores into probabilities that sum to 1. The model draws one token from that distribution, appends it to the sequence, and runs the whole network again on the longer sequence. That loop, called autoregression, is generation.

Three consequences fall straight out of this design, and every one of them shapes how agents behave:

1. No plan exists. The model commits one token at a time. When it "reasons," it is literally emitting reasoning tokens that then condition later tokens. The scratchpad is the computation.
2. Compute per token is fixed. Each token costs roughly 2x the parameter count in FLOPs.^[1] A model cannot think harder on one token, only longer, by emitting more of them.
3. Everything is conditioning. System prompt, retrieved docs, prior turns, tool outputs, and the model's own words are all just tokens in context that shift the next-token distribution.

"More tokens equals more serial computation" is the mechanical reason chain-of-thought and reasoning models work - you buy accuracy with tokens. That is the subject of the next lesson, Reasoning and test-time compute.

Tokens: the model never sees your text

Transformers operate on integers, not characters. A tokenizer encodes your string into token IDs before the model sees anything, and decodes IDs back into text afterward. Nearly every modern model - GPT, Llama, Mistral, Claude, Gemini - builds its tokenizer with Byte Pair Encoding (BPE): start from raw bytes, then repeatedly merge the most frequent adjacent pair into a new token until you hit a target vocabulary size. The saved merge rules are the tokenizer.^[2] Tokens are sub-word chunks - usually bigger than a letter, smaller than a word. A rough rule for English: about 4 characters is 1 token, and 100 tokens is about 75 words. The leading space is part of the token, so " France" is a different token from "France".

Feed the five tokens of "The capital of France is" into the network and it returns a probability for every possible next token. For this prompt the distribution is sharply peaked:

Non-English text and code learn fewer merges, so they cost more tokens per word - inflating both price and how fast you burn through the context window.^[2]

Sampling: turning a distribution into a choice

The network gives you a distribution. Sampling parameters decide how a single token is drawn from it. Crucially, they do not change what the model knows or predicts - they only reshape or truncate the bars before the draw.^[3]

1. Temperature. Divides the logits by T before softmax. T < 1 sharpens the distribution (more confident); T > 1 flattens it (more random). T = 0 disables sampling entirely, giving greedy decoding, always the argmax.
2. Top-k. Keep only the k highest-probability tokens, renormalize, then sample. A fixed cutoff - it ignores how confident the model actually is on this particular step.
3. Top-p (nucleus). Keep the smallest set of tokens whose cumulative probability is at least p (for example 0.9), renormalize, sample. Adaptive: few tokens when confident, many when uncertain. For the France prompt above, top-p 0.9 keeps just " Paris" plus " the".

Here is the required intuition made concrete: the same prompt run five times at two temperatures. Notice two things: at T = 0 the output is identical every run, and temperature only matters when the distribution is not already peaked:

The context window: a hard token budget

The context window is the maximum number of tokens the model can attend to at once - and it must hold everything: system prompt, tool definitions, conversation history, retrieved documents, and the generated output (including any hidden reasoning tokens).^[6] When it fills up, the earliest tokens must be dropped or summarized. There is no memory beyond it. In an agent, tool schemas, history, and retrieved docs compete for the same fixed space as the answer you still need to generate.

Windows grew from about 2k tokens (GPT-3) to 128k to 1M+ on frontier models - but bigger is not reliably usable. The lost in the middle effect shows retrieval accuracy follows a U-shaped curve over position: highest when the needed fact sits at the start or end of context, degrading by more than 30% when it is buried in the middle.^[4]

In-context learning: teach it in the prompt

In-context learning (ICL) is the ability to perform a new task purely from instructions or examples placed in the prompt - with no gradient updates and no fine-tuning. Zero-shot describes the task; few-shot adds a handful of input-to-output demonstrations and the model infers the pattern. This is the property that made prompt engineering viable at all, and it is widely considered an ability that emerges sharply above a scale threshold.^[5]

# Two demonstrations teach the task, no retraining involved.
Review: "Loved it, would buy again."    Sentiment: positive
Review: "Broke on day two. Avoid."      Sentiment: negative
Review: "Arrived late but works fine."  Sentiment:   # model completes: neutral

Chain-of-thought - prompting for intermediate steps before the answer - is ICL's most famous trick, and it too emerges only at scale.^[5] We put ICL to work in Prompting for agents.

Two properties that shape every agent

Everything above collapses into two constraints you must design around. Miss them and your agent architecture will fight the engine instead of using it.

# The model is stateless. YOU carry the state, and resend it all every turn.
history = [system, user_1, assistant_1, user_2]   # just a growing list of tokens
reply   = model.generate(history)                 # one stateless call, remembers nothing
history = history + [reply, user_3]               # you append the new turn...
model.generate(history)                           # ...and the model re-reads the WHOLE thing

These two facts are why the rest of this course exists. Statelessness is why agents need external memory and a persistent agent loop. Context-boundedness is why retrieval and context engineering exist: you are constantly deciding which tokens earn a place in a budget that is always too small.

Check yourself