From GPT to Production
Fine-tuning and instruction tuning
Specializing a foundation model for tasks
Pretraining leaves you with a base model: a spectacular next-token predictor that has read much of the internet - and does exactly what it was trained to do, which is continue the text, not answer it. Ask a base model "Explain gravity to a child" and it may reply with "Explain gravity to a physicist. Chapter 2 - Forces & Motion." because that is a plausible continuation of the web page it thinks it is reading. Fine-tuning is the second, far cheaper stage that teaches the model to behave - and modern tricks like LoRA pull it off by nudging a tiny sliver of the parameters while the giant backbone stays frozen. This lesson is that whole handoff, made concrete.
From base model to assistant
The core move is supervised fine-tuning (SFT). It is the same objective as pretraining - predict the next token, minimize cross-entropy - but the dataset changes. Instead of raw internet text, you feed the model a curated pile of (instruction, ideal response) demonstrations: thousands of prompts each paired with the kind of answer you wish the model would give. Crucially, the loss is counted only on the response tokens; the model is graded on how well it produces the answer, not on re-predicting the prompt.
Same brain, better manners
A base model and an instruct model can share the identical architecture and nearly identical weights. The base model isn't dumb - it already "knows" what gravity is. It just answers the wrong question, because "continue this text" and "respond to this request" are different tasks. SFT doesn't teach new facts so much as teach a habit: when you see an instruction, switch into answer-mode. That is why a few thousand good demonstrations can transform behaviour that took trillions of tokens of pretraining to build.
To make "where does the instruction end and the answer begin?" unambiguous, every example is wrapped in a chat template - special role markers the tokenizer inserts around each turn. At inference the model sees the same markers, so it knows it is being asked something:
<|user|>
Explain gravity to a child
<|assistant|>
Gravity is an invisible pull that draws things together ...See the difference - and what LoRA actually touches
Send the same instruction to a simulated base model and an instruction-tuned one, and watch how they diverge: the base model treats your prompt as a document to continue (echoed fragments are tinted red), while the instruct model answers it. Then flip the training mode. Full fine-tune lights up the entire weight matrix as trainable; LoRA freezes that backbone and lights up only two thin bars - the low-rank adapter - and the "% trained" readout collapses toward half a percent.
The weight matrix - what a fine-tune actually updates
LoRA (r = 4) - training 12,800 params (0.50% of the matrix); the 2,560,000 backbone weights stay frozen.
Grid is schematic; the count is for one real 1600×1600 weight matrix (GPT-2 XL scale).
The "aha": the giant stays frozen
Toggle Full fine-tune and the whole grid turns teal - every one of the 2.56 million weights gets a gradient, so you need optimizer state for all of them (roughly 2-3x the model in extra memory). Now switch back to LoRA: the backbone goes gray and only a thin L-shaped adapter lights up. Drag the rank slider - even at = 8 you are training about 1% of the matrix. Same downstream behaviour, a fraction of the compute, and you can ship a 20 MB adapter file instead of a fresh multi-gigabyte model.
LoRA: a low-rank patch on a frozen giant
LoRA (Low-Rank Adaptation) starts from a hypothesis: the change a fine-tune needs to make to a weight matrix is "low-rank" - it lives in a small number of directions, even though the matrix itself is huge. So instead of learning a full-size update , LoRA factors it into two skinny matrices and learns only those, leaving the pretrained untouched:
Freeze W_0, learn a rank-r correction
The frozen matrix still does the heavy lifting; the adapter adds a small steerable correction. Because and are thin, the trainable count is tiny compared to updating directly:
For a matrix at : that is params instead of - exactly 0.5%. LoRA is just the matrix factorization idea (a low-rank product approximating a bigger matrix) turned into a training trick. At inference you can even fold back into so there is zero added latency.
LoRA is the best-known member of a family called PEFT - parameter-efficient fine-tuning. The recipe is always the same shape: freeze the pretrained backbone, insert a small number of new trainable parameters, and train only those. In code with Hugging Face's peft library it is a handful of lines:
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM
# 1) load the BASE model - a raw next-token predictor
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3-8B")
# 2) freeze every backbone weight: no gradients flow into them
for p in model.parameters():
p.requires_grad = False
# 3) inject tiny low-rank B, A adapters into the attention projections
cfg = LoraConfig(r=8, lora_alpha=16, target_modules=["q_proj", "v_proj"])
model = get_peft_model(model, cfg) # only A and B now require_grad
model.print_trainable_parameters()
# trainable params: 3,407,872 || all params: 8,033,669,120 || trainable%: 0.0424Catastrophic forgetting - the reason not to just retrain everything
Full fine-tuning on a small, narrow dataset can overwrite the general skills pretraining built: teach it hard on medical Q&A and it may "forget" how to write a poem or do arithmetic. This is catastrophic forgetting - gradient descent on the new task drags weights away from the broad optimum without anything holding them back. Freezing the backbone (LoRA) is a natural guardrail: the original knowledge literally cannot move, so you specialise without clobbering the base. It also keeps the failure reversible - delete the adapter and you are back to the pristine model.
Check yourself
Given the prompt "Explain gravity to a child", a raw base model will most likely:
LoRA keeps fine-tuning cheap because it:
Recall: for a d x k weight matrix, why does a LoRA adapter of rank r train so few parameters?
Nail down the factored-update arithmetic. Try to state it, then check.
Lock it in
- Supervised fine-tuning (SFT) is the same next-token objective as pretraining, but on curated (instruction, response) pairs with loss only on the response.
- A chat template separates roles so the model learns when it is being asked versus when it should answer.
- LoRA freezes the pretrained backbone and trains only two thin matrices B, A whose product BA is a low-rank correction - often less than 1% of the parameters.
- Catastrophic forgetting is why full fine-tuning on a narrow task risks overwriting general knowledge; freezing the backbone is the guardrail.
Primary source
Work through the Hugging Face LLM Course for the full journey from a pretrained base model to a fine-tuned, instruction-following assistant, with runnable code for SFT and chat templates. Then read the original LoRA paper, Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models" (2021) - the low-rank hypothesis, the exact formulation, and the experiments showing it matches full fine-tuning at a fraction of the trainable parameters.
Ask your teacher
The whole lesson is two ideas you already know, reused. Fine-tuning is nothing exotic - it is just more gradient descent, only starting from pretrained weights and running on a smaller, task-specific dataset. And LoRA is a clever low-rank matrix factorization: approximate the update with a skinny product and you slash the parameters you have to learn. Want me to go deeper on how the instruction dataset is built and mixed, why lora_alpha scales the adapter, or how full SFT differs from the preference-based alignment that comes next? Ask and I will build the follow-up.