Reference shelf

Glossary

Every term in this spine, crisply defined and cross-linked.

Written by Claude Opus 4.8

Every term in this spine, defined crisply. Filter to find one fast, or skim it to see how the pieces connect.

Arithmetic series: A sum whose terms change by a fixed step; its total grows quadratically, as in 1 + 2 + ... + n = n(n+1)/2.
Base case: The starting case of an induction, usually P(1), verified directly. It knocks over the first domino.
Base rate: The prior prevalence of an outcome in the population. A rare base rate can make even an accurate test mostly wrong on the occasions it fires.
Basis: A minimal set of independent vectors whose span is the whole space, so every point has exactly one address.
Bayes' theorem: P(A | B) = P(B | A)P(A) / P(B): the rule that flips a known conditional into the one you want, updating a prior belief with evidence.
Bernoulli distribution: One trial with success probability p: the distribution of a single yes/no outcome.
Bijective: Both injective and surjective, a perfect pairing, which makes the function reversible.
Binomial coefficient: The count C(n,k) of ways to choose k of n items; also the coefficient of a^(n-k) b^k in the expansion of (a+b)^n.
Binomial distribution: The number of successes in n independent Bernoulli trials, P(X = k) = C(n,k) p^k (1-p)^(n-k), with mean np.
Binomial theorem: The expansion (a+b)^n = sum_k C(n,k) a^(n-k) b^k, whose coefficients are row n of Pascal's triangle.
Catastrophic cancellation: Subtracting two nearly equal numbers, whose matching leading digits annihilate and leave only noisy low bits, exploding the relative error.
Chain rule: The derivative of f(g(x)) is f'(g(x)) times g'(x). Rates along a chain of functions multiply.
Characteristic equation: det(A - lambda·I) = 0, the equation whose roots are the eigenvalues of A.
Closed form: A formula with no loop or sum left in it, letting you read off a total, and its Big-O, in one glance, like sum i = n(n+1)/2.
Codomain: The declared target set a function maps into, distinct from the image it actually reaches.
Column space: The span of a matrix's columns: every output the matrix can produce.
Combination: An unordered selection of k items from n, C(n,k) = n! / (k!(n-k)!), read 'n choose k'. Related to permutations by P(n,k) = C(n,k)·k!.
Conditional probability: P(A | B) = P(A and B) / P(B): the probability of A once you know B happened, zooming into the slice of the world where B holds.
Contrapositive: The statement (not q) => (not p), which always shares the truth value of p => q, so proving it proves the original.
Convergence: When an infinite series adds up to a finite total; otherwise it diverges to infinity. For a geometric series the fate hinges on whether |r| < 1.
Converse: The statement q => p, obtained by swapping the two sides of an implication. It can be false even when the original is true.
Cosine similarity: The dot product divided by both lengths, giving cos(theta) in [-1, 1]. Blind to magnitude, it scores pure direction.
Covariance matrix: A symmetric matrix whose entries measure how features vary together. Its eigenvectors are the directions of greatest spread in the data.
Cross-entropy: The average surprise of the truth p scored with a model's probabilities q, H(p,q) = -sum p(x) log q(x). Never smaller than the entropy H(p).
Derivative: f'(x): the slope of the tangent line at x, the limit of the secant slope as the gap shrinks to zero.
Determinant: The signed area-scaling factor of a matrix, det = ad - bc for a 2x2: the area of the image of the unit square, negative when orientation flips, zero when space collapses.
Diagonalization: Factoring a matrix as A = Q Lambda Q^-1, with eigenvectors in Q and eigenvalues on the diagonal of Lambda: in its own eigenbasis the matrix is pure per-axis scaling.
Dimension: The number of vectors in a basis: how many independent directions the space has.
Direct proof: A proof that assumes p and reasons forward, step by step, to q.
Directional derivative: The slope along a chosen direction u, equal to the gradient dotted with u.
Domain: The set of all legal inputs to a function.
Dot product: a·b: sum of matching components, equal to |a||b|cos(theta). Measures how much one vector points along another.
Eigenvalue: The factor lambda by which a matrix scales its eigenvector, Av = lambda·v. Negative flips it; |lambda| > 1 grows and |lambda| < 1 decays under repeated application.
Eigenvector: A nonzero direction a matrix only stretches, never rotates: Av points along v. One of the transformation's own axes.
Entropy: The average surprisal of a distribution, H(p) = -sum p(x) log p(x): the irreducible uncertainty of a source.
Event: Any subset of the sample space, to which a probability is assigned.
Evidence: The total probability of the observed data across all hypotheses: the marginal P(B) in the denominator of Bayes' theorem.
Expectation: The mass-weighted average of a random variable, E[X], its balance point: sum of x·p(x) when discrete, the integral of x·f(x) when continuous.
Exponent: The bits of a float that set its magnitude as a biased power of two. The gap between neighbouring floats doubles every time the exponent does.
Exponential: b to the x: multiply the base b by itself x times. Growth when b > 1, decay when 0 < b < 1.
Factorial: n! = n(n-1)...1, the number of ways to arrange n distinct items in a row, with 0! = 1 by convention.
Floating point: Scientific notation in binary: a sign, an exponent for magnitude, and a mantissa for the leading digits. Representable numbers are dense near zero and sparse at large magnitudes.
Function: A relation where every input is paired with exactly one output. A trained model is literally a function from inputs to outputs.
Gaussian: The symmetric bell curve N(mu, sigma^2), also called the normal distribution, pinned down entirely by its mean and standard deviation. About 68, 95, and 99.7 percent of its mass lies within 1, 2, and 3 standard deviations of the mean.
Geometric series: A sum whose terms change by a fixed ratio r. It converges to 1/(1-r) when |r| < 1 and diverges otherwise.
Gradient: The vector of all partial derivatives. It points in the direction of steepest ascent, perpendicular to the contour.
Gradient descent: Repeatedly stepping in the negative gradient direction, p := p - alpha·grad f, to find a minimum.
Harmonic series: The sum 1 + 1/2 + 1/3 + ... + 1/n, which grows like ln n. It diverges, but only as slowly as a logarithm.
Identity matrix: Ones on the diagonal, zeros elsewhere: the 'do nothing' transform, so IA = A.
Image: The subset of the codomain that some input actually maps to. A function is surjective when its image equals its codomain.
Implication: 'If p then q', written p => q, defined as false in only one case (p true, q false) and equal to (not p) or q.
Independence: Two events are independent when learning one changes nothing about the other: P(A | B) = P(A), equivalently P(A and B) = P(A)P(B).
Inductive hypothesis: The assumption that P(k) holds for some arbitrary k, borrowed to build P(k+1). Not the conclusion itself.
Inductive step: The proof that P(k) implies P(k+1) for every k, so each true case forces the next.
Injective: One-to-one: no two different inputs share the same output.
Joint probability: P(A and B), the probability that both events happen together: a single cell of a two-way table.
Kernel: The set of vectors a transformation crushes to the origin, also called the null space. It is nonzero exactly when the determinant is zero, and then the map is not invertible.
KL divergence: The gap between cross-entropy and entropy, D_KL(p || q) = H(p,q) - H(p) >= 0: the extra bits wasted by using q in place of the true p.
Law of large numbers: As the number of samples grows, the empirical histogram and sample mean converge to the true distribution and its expectation.
Learning rate: Alpha: the step size in gradient descent. Too small is slow, too large overshoots and can diverge.
Likelihood: How probable the observed evidence is under a hypothesis, P(B | A). Read as a function of the hypothesis with the data held fixed.
Linear combination: a·v + b·w: scale each vector by a number, then add. The basic move of a vector space.
Linear independence: No vector in a set is a linear combination of the others. In 2-D, the determinant of the pair is nonzero.
Linear transformation: A map of space that keeps grid lines straight, parallel, and evenly spaced and fixes the origin, obeying T(u+v) = T(u)+T(v) and T(cv) = cT(v). Its matrix columns are where the basis vectors land.
Log-likelihood: The logarithm of the likelihood, which turns the product of per-point probabilities into a sum without moving its maximum.
Log-sum-exp: The overflow-proof identity log sum e^(z_j) = m + log sum e^(z_j - m) with m = max z_j, which caps the largest exponentiated term at e^0 = 1.
Logarithm: The inverse of an exponential. log_b(x) is the power b must be raised to to reach x, and counts how many times you can halve.
Logical connective: An operator that builds a larger proposition from smaller ones: negation (not), conjunction (and), disjunction (or), and implication (if-then).
Mantissa: The fraction bits of a float that carry its precision, on top of an implicit leading 1. float32 gives about 7 decimal digits.
Marginal probability: A probability like P(A) obtained by summing the joint over everything you do not care about: a row or column total of the table.
Mathematical induction: A proof that certifies P(n) for all n from two facts: a base case and an inductive step. A finite argument covering infinitely many cases.
Matrix: A rectangle of numbers, m rows by n columns, encoding a linear transformation: each column is where one input axis lands.
Matrix multiplication: C = AB where C[i][j] is row i of A dotted with column j of B. It is function composition, not commutative.
Maximum likelihood estimation: Choosing the parameters that make the observed data most probable under the model. Training is MLE in disguise: MSE assumes Gaussian noise, cross-entropy assumes a categorical output.
Mean: The average of a set of values, their sum divided by the count. The balance point, or center of mass, of the data cloud.
NaN: 'Not a Number', the result of undefined operations like infinity minus infinity. It spreads through every later computation it touches.
Negative log-likelihood: The negated log-likelihood, the quantity minimized during training. Minimizing it is the same as maximizing likelihood.
Norm: The length of a vector, the square root of the vector dotted with itself.
Numerical stability: The property of an algorithm that avoids overflow, underflow, and cancellation, for instance by subtracting the max before softmax or working in log-space.
Orthogonal: Perpendicular: two nonzero vectors whose dot product is zero.
Overflow: Exceeding the largest finite value (about 3.4e38 for float32), which yields infinity and can poison later operations into NaN.
Partial derivative: The slope of a multi-input function when you vary one input and hold all the others fixed. Written with a curly d.
Pascal's triangle: The triangle of binomial coefficients where every entry is the sum of the two above it, C(n,k) = C(n-1,k-1) + C(n-1,k). Each row sums to 2^n.
Permutation: An ordered selection of k items from n, P(n,k) = n! / (n-k)!.
Perplexity: The exponential of the cross-entropy, 2^H(p,q): the effective number of equally likely choices the model is unsure among.
Pigeonhole principle: If n items go into m boxes with n > m, some box holds at least two, and more sharply at least ceil(n/m) of them.
Posterior: The updated probability of a hypothesis after incorporating evidence, P(A | B): the output of Bayes' theorem.
Power rule: The derivative of x to the n is n times x to the n minus one.
Principal component analysis: Compressing data by keeping the eigenvectors of its covariance matrix with the largest eigenvalues: the perpendicular directions of maximum variance.
Prior: The probability of a hypothesis before seeing evidence, P(A), such as the base rate of a disease in the population.
Probability density function: For a continuous random variable, f(x) >= 0 gives density, not probability. Probability is the area under it over an interval, and the whole integrates to 1.
Probability distribution: An assignment of a total of 1.0 of probability mass across the values a random variable can take.
Probability mass function: For a discrete random variable, p(x) = P(X = x): the honest probability of each value. The values sum to 1.
Proof by contradiction: A proof that assumes the claim is false, then derives an impossibility, forcing the assumption to have been wrong.
Proposition: A statement that is exactly true or false, with no maybe.
Quantifier: A symbol scoping a claim over a collection: the universal 'for all' (forall) and the existential 'there exists' (exists). Negation swaps the two.
Random variable: A function that maps each outcome to a number, such as the count of heads in three coin flips.
Recursion: A function defined in terms of itself on smaller inputs, with a base case that stops it. The executable mirror of an inductive proof.
Relation: Any set of ordered pairs: a chosen selection of links between elements of two sets, a subset of their Cartesian product.
Rule of product: When a task is a sequence of independent stages, the total number of ways is the product of the per-stage option counts.
Rule of sum: When possibilities split into non-overlapping cases, the total is the sum of the case counts.
Sample space: The set of all possible outcomes of an experiment, written Omega. An event is any subset of it.
Scalar: A single number, used to stretch or flip a vector.
Set: A collection of distinct things, written in curly braces. Order and repeats do not matter.
Set-builder notation: Describing a set by a rule its members satisfy, read as 'the set of x such that ...', rather than listing them.
Softmax: A function turning raw scores (logits) into a probability distribution, softmax(z)_i = e^(z_i) / sum_j e^(z_j). Every entry is non-negative and they sum to 1, so it is a PMF over a vocabulary.
Span: Every linear combination a set of vectors can build: the region they can reach.
Standard deviation: The square root of the variance, sigma: the spread of the data expressed back in its own units.
Strong induction: Induction whose step may assume the whole run of earlier cases P(1) through P(k), not just the immediate predecessor. Equally powerful as ordinary induction.
Summation: Sigma notation for adding a sequence of terms, a for-loop written in one line. sum_{i=1}^{n} a_i adds a_i as i walks from 1 to n.
Surjective: Onto: every element of the codomain is hit by some input.
Surprisal: The information content of an outcome, -log p, measured in bits. Rare events carry more surprise; a sure thing carries none.
Tangent line: The straight line that touches a curve at one point with the same slope: the local linear approximation.
Telescoping sum: A sum where each term splits into pieces that cancel their neighbours, collapsing the whole thing to just its first and last terms.
Truth table: A table listing the truth value of an expression for every combination of its inputs, defining exactly what each connective means.
Underflow: Falling below the smallest representable magnitude, so a value rounds to exactly 0. A long product of probabilities underflows, which is why training sums logs instead.
Uniform distribution: A distribution in which every outcome is equally likely.
Variance: The average squared distance of values from their mean, sigma^2. Squaring stops positive and negative gaps from cancelling.
Vector: An object you can add and scale. Pictured as an arrow from the origin, stored as a list of coordinates.
Well-ordering principle: Every non-empty set of positive integers has a smallest element: the foundation that makes induction valid.
Z-score: A value recast as (x - mu)/sigma, counting how many standard deviations it sits from its mean. Standardizing gives data mean 0 and standard deviation 1 without changing its shape.