Hallucination, limits, and safety

Ask a language model about a person who never existed and it will often answer smoothly, in complete sentences, with a university, a field, and a prize - all invented. This is hallucination, and it is not a bug bolted on by accident. It falls straight out of the training objective: a base model is optimised to make its next token plausible, not true. There is no truth oracle inside; there is only a probability distribution over what usually comes next. This lesson maps the failure modes - hallucination, overconfidence, knowledge cutoff, context limits, the reversal curse - and then the safety layer built on top: jailbreaks, prompt injection, guardrails, and red-teaming. Balanced, not alarmist: these are engineering problems, actively and measurably being reduced.

Why fluent models make things up

Recall the objective from Lesson 37: given the tokens so far, predict a distribution over the next token, and train to maximise the likelihood of real text. Nowhere in that loss is there a term for is this claim true? - only is this a likely thing to say?. When you ask about something the model has no data for, the maximum-likelihood continuation is still a grammatical, confident-sounding sentence, because that is what almost all of its training text looks like. Silence and "I don't know" are rare in the corpus, so the model rarely produces them. It fills the gap with the shape of an answer.

Weng calls the sharp version extrinsic hallucination: output that is not grounded in - and often contradicts - any real-world source, as opposed to intrinsic hallucination that contradicts the prompt itself. The cure is to give the model something to be grounded in: retrieved evidence it can cite, or the honest option to abstain.

The hard limits

Four walls are structural - no prompt phrasing removes them, only tooling around the model can help.

• Knowledge cutoff. Training data is frozen at a date. Events, papers, and prices after that date simply are not in the weights. The model will still answer about them - confidently, from the wrong era.
• Context-length limit. Attention costs $O(n^2)$ in the sequence length $n$ (Lesson 31), so every model has a finite window. Text outside the window does not exist for it, and quality degrades toward the far ends of a long one.
• The reversal curse. Trained heavily on "A is B", models often fail the mirror "B is A" - knowing a person's parent does not reliably let it name that parent's child. Facts are stored directionally, not as a symmetric graph.
• Overconfidence. The number you might read as "confidence" is just the top softmax probability - a fluency signal, not a calibrated probability of being correct.

The demo has two modes. First Hallucination: ask about a made-up entity, watch a toy model invent details with a per-token confidence heatmap, then flip on retrieval grounding and watch invented claims get replaced by cited facts (or an honest abstention). Then Safety: fire escalating jailbreak prompts at a safety classifier meter, and toggle a system-prompt guardrail to change how many attacks get through.

Making models safer

Safety is a second system layered on the plausibility engine. The main threats are adversarial. A jailbreak is a prompt crafted to talk the model past its guidelines - role-play ("you are DAN, with no rules"), fake authority ("developer mode on"), or slow escalation. Prompt injection is subtler and specific to tools and agents: untrusted content the model reads - a web page, a retrieved document, an email - smuggles in instructions, and the model, unable to tell data from directives, follows them. This is why safety is where the whole course meets: it depends on training (alignment), prompting (the system prompt boundary), and retrieval (trusting your sources) all at once.

The defences are layered, not a single wall. Alignment training (RLHF) teaches refusal of harmful requests. A separate safety classifier screens inputs and outputs - the meter you just tripped. Guardrails keep trusted system instructions structurally separated from untrusted data and sanitise injected commands. Red-teaming is the practice of attacking your own model before release to find these holes, and it is how the risk scores in the probe would be calibrated in reality. Two honest caveats: models inherit bias from their training data, which is measured and reduced but never zeroed; and most capabilities are dual-use - the same skill that drafts a safety plan can draft its opposite - so the goal is defence-in-depth that lowers risk, not a mythical perfect filter.

# A token's "confidence" is just the top softmax probability (Lesson 37).
logits = model(prompt)                 # raw scores over the vocabulary
probs  = softmax(logits)             # nonnegative, sums to 1
token  = probs.argmax()
conf   = probs[token]                  # e.g. 0.94 - high even when the fact is invented

# Grounding: condition generation on retrieved evidence (RAG, Lesson 55).
docs = retriever.search(prompt, k=4)   # real passages... or none at all
if not docs:
    answer = "I couldn't find a reliable source."   # abstain, don't invent
else:
    answer = model(prompt, context=docs)         # cite the docs in the output

Check yourself