The identities the course actually derives, on one page, in the order the ladder builds them. Come back cold and try to say what each one means before you read the caption.
Regression and the first loss y ^ = w x + b
A linear model: a weighted input plus a bias. J ( w , b ) = n 1 i = 1 ∑ n ( y ^ i w x i + b − y i ) 2 Mean squared error: the average squared gap the fit tries to shrink. Derived in Linear regression , The cost function , and Loss in depth .
Classification z = w x + b , σ ( z ) = 1 + e − z 1 = p
The sigmoid squashes a raw score into a probability in (0, 1). L BCE = − [ y ln p + ( 1 − y ) ln ( 1 − p ) ]
Binary cross-entropy: the negative log probability of the true label. precision = T P + F P T P , recall = T P + F N T P , F 1 = 2 ⋅ precision + recall precision ⋅ recall
Scoring a classifier past accuracy: precision, recall, and their harmonic mean. Derived in Logistic regression and Evaluating a classifier .
Learning by gradient descent w := w − α d w dJ
The update rule: step against the gradient, scaled by the learning rate. ∂ a ∂ L = ∂ p ∂ L ⋅ ∂ u ∂ p ⋅ ∂ a ∂ u ( grad(parent) = grad(child) × local )
Backprop is the chain rule: a loss gradient factors along the path to a weight. ∂ z i ∂ L CE = p i − y i
Why softmax plus cross-entropy is the standard pairing: the gradient is just predicted minus true. Derived in Gradient descent , Backpropagation , and Loss in depth .
Optimizers v t = μ v t − 1 − η ∇ L ( θ t − 1 ) , θ t = θ t − 1 + v t
Momentum: carry a running velocity so updates keep rolling in a consistent direction. m t = β 1 m t − 1 + ( 1 − β 1 ) g t , s t = β 2 s t − 1 + ( 1 − β 2 ) g t 2 , θ t = θ t − 1 − η s ^ t + ϵ m ^ t Adam: per-parameter step sizes from running averages of the gradient and its square, bias-corrected. Derived in Optimizers .
Activations and softmax σ ′ ( x ) = σ ( x ) ( 1 − σ ( x ) ) , tanh ′ ( x ) = 1 − tanh 2 ( x ) , ReLU ( x ) = max ( 0 , x )
The classic activations and their derivatives; ReLU is the cheap default. softmax ( z ) i = j ∑ e z j e z i
Softmax: exponentiate and normalize a score vector into a probability distribution. Derived in Activation functions .
Language modeling p ( x 1 , … , x T ) = t = 1 ∏ T p ( x t x 1 , … , x t − 1 ) The chain rule of probability: a sequence factors into next-token predictions. L = − T 1 t = 1 ∑ T log p θ ( x t x < t ) The training loss: average next-token negative log-likelihood, which is cross-entropy. PPL = exp ( L )
Perplexity is exp of the loss: the model's effective number of equally likely next tokens. Derived in Language modeling .
Attention q i = x i W Q , k i = x i W K , v i = x i W V
Each token is projected into a query, a key, and a value. Attention ( Q , K , V ) = softmax ( d k Q K ⊤ ) V Scaled dot-product attention: relevance scores softmaxed, then used to blend the values. MultiHead ( Q , K , V ) = Concat ( head 1 , … , head h ) W O , d k = d model / h
Multi-head attention: run h attentions on split projections, concatenate, and mix. Derived in Self-attention and QKV and Multi-head attention .
The transformer block y = x + Sublayer ( x )
The residual connection: add the input back so gradients flow straight through. LN ( x ) = γ ⊙ σ 2 + ϵ x − μ + β , μ = d 1 i ∑ x i Layer normalization: standardize each token's vector, then apply learned gain and bias. FFN ( x ) = W 2 ϕ ( W 1 x + b 1 ) + b 2 , W 1 ∈ R 4 d × d
The feed-forward block: widen to 4d, apply a nonlinearity, project back. Derived in Residuals, LayerNorm, and the FFN .
Scale C ≈ 6 N D
The compute rule of thumb: FLOPs are about six times parameters times tokens. L ( N , D ) = E + N α A + D β B
The Chinchilla loss law: an irreducible floor plus terms that fall as power laws in parameters and data. D opt ≈ 20 N opt
Compute-optimal training: grow model and data together, roughly 20 tokens per parameter. Derived in Pretraining at scale and Scaling laws .
Alignment L RM = − log σ ( r ϕ ( y w ) − r ϕ ( y l ) )
The reward model loss: prefer the chosen response over the rejected one. π max E y ∼ π [ r ( y ) ] − β KL ( π ∥ π ref )
The RLHF objective: maximize reward while staying close to the reference model. L DPO = − log σ ( β log π ref ( y w ) π θ ( y w ) − β log π ref ( y l ) π θ ( y l ) )
DPO folds the reward and RL step into one preference loss, no separate reward model. Derived in RLHF and DPO .
Decoding p i = j ∑ exp ( z j / T ) exp ( z i / T )
Temperature scaling: divide the logits before softmax to sharpen or flatten the choice. cdf [ i ] = j = 0 ∑ i p ( j ) , m = min { i : cdf [ i ] ≥ p }
Top-p (nucleus): keep the shortest prefix of sorted tokens whose mass reaches p. Derived in Decoding and sampling , Sorting to top-k , and Prefix sums to top-p .
Inference cost attention cost = O ( n 2 )
The attention wall: cost grows with the square of sequence length. 2 n layers n d model
The KV cache trades memory for speed: it stores this many scalars, linear in context length. s = 2 b − 1 w m a x − w m i n , q = clamp ( ⌊ s w − w m i n ⌉ , 0 , 2 b − 1 )
Quantization maps a float range onto a small integer grid of 2^b levels. Derived in The Big-O attention wall , Memoization to the KV cache , and Bit tricks to quantization .