Maximum likelihood estimation

You have data, and you have a model with some knobs. Maximum likelihood estimation asks one question and answers it precisely: which knob settings make the data you actually observed the least surprising? Turn that question into arithmetic and something remarkable falls out - the loss functions you have been minimizing all along, mean squared error and cross-entropy, are not arbitrary. They are the exact scores you get when you write down "make this data likely" and simplify. Training a model is maximum likelihood estimation wearing a disguise.

Likelihood: probability, read backwards

A probability model $p(x\mid \theta )$ normally answers a forward question: given the parameters $\theta$, how probable is an outcome $x$? Maximum likelihood flips it. Now the data is fixed - you already saw it - and $\theta$ is the unknown you get to choose. The same expression, read with data locked and parameters free, is called the likelihood. If the observations are independent, the likelihood of a whole dataset is the product of each point's probability:

Maximizing $\ell (\theta )$ is fitting the model: you are sliding the parameters until the model assigns as much total probability as possible to the exact points you observed. Everything else - gradient descent, loss functions, training - is machinery for climbing this one hill.

Steer the bell curve to fit the data

Here is the whole idea in one picture. Six data points sit fixed on the axis. Above them floats a Gaussian $\mathcal{N}(\mu ,{\sigma }^2)$ whose center $\mu$ and width $\sigma$ you control. Each point sends up a stem to the curve - the stem's height is that point's probability density $p(x_i)$. Multiply the six heights to get the likelihood; add their logs to get the log-likelihood. Drag the sliders: notice that pushing the peak over the data's center and tuning the width to match its spread makes every stem tall at once. Then hit Maximize and watch the curve snap to the exact place gradient descent would climb to - the MLE.

The MLE for a Gaussian has a closed form you can read off the data: $\hat{\mu }$ is just the mean of the points, and ${\hat{\sigma }}^2$ is their average squared distance from that mean. No search required - but gradient descent would find the very same summit by walking uphill on $\ell$. That equivalence is the bridge to every model too complex for a formula.

Where MSE and cross-entropy come from

Now the payoff. Suppose your model predicts ${\hat{y}}_i$ and you assume the real target is that prediction plus Gaussian noise: $y_i={\hat{y}}_i+{\epsilon }_i$ with ${\epsilon }_i\sim \mathcal{N}(0,{\sigma }^2)$. Write the log-likelihood of the targets and expand the Gaussian - the constants fall away and one term is left holding the parameters:

So the loss was never a design choice pulled from thin air. Pick a noise assumption - Gaussian for real-valued targets, categorical for labels - write "make the data likely," take the log, and the standard loss drops out of the algebra. Change the assumption and you change the loss; assume Laplace noise instead of Gaussian and MLE hands you mean absolute error. Loss functions are downstream of a probability story.

import numpy as np
data = np.array([2.0, 3.5, 4.0, 5.0, 5.5, 7.0])

def neg_log_likelihood(mu, sigma):
    z     = (data - mu) / sigma
    log_p = -0.5 * z**2 - np.log(sigma * np.sqrt(2 * np.pi))
    return -log_p.sum()          # NLL - the thing we minimize

mu_hat    = data.mean()             # 4.50  - MLE of the mean
sigma_hat = data.std()              # 1.58  - MLE of the std (ddof=0)
print(neg_log_likelihood(mu_hat, sigma_hat))  # 11.26 - the smallest possible NLL

Check yourself