Maximum Likelihood Estimation Explained

Suppose you flip a coin 10 times and see 7 heads. What is your best guess for the probability of heads? Most people say 0.7 without thinking, and maximum likelihood estimation (MLE) is the principle that justifies that instinct. It is also the hidden engine behind many of the loss functions used to train machine learning models, from linear regression to deep neural networks.

The Idea in One Sentence

Choose the parameter values that make the data you actually observed as probable as possible.

Formally, suppose data \( x_1, \dots, x_n \) are independent draws from a distribution with probability (or density) function \( p(x \mid \theta) \), where \( \theta \) is an unknown parameter. The likelihood function is

$$ L(\theta) = \prod_{i=1}^{n} p(x_i \mid \theta). $$

It is the same formula as the joint probability of the data, but read differently: the data are fixed and \( \theta \) varies. The maximum likelihood estimate is

$$ \hat{\theta}_{\text{MLE}} = \arg\max_{\theta} L(\theta). $$

Why We Use the Log-Likelihood

Products are awkward to differentiate, and multiplying many probabilities smaller than 1 quickly underflows on a computer. Since the logarithm is strictly increasing, maximising \( L(\theta) \) is equivalent to maximising

$$ \ell(\theta) = \ln L(\theta) = \sum_{i=1}^{n} \ln p(x_i \mid \theta). $$

The product becomes a sum, which is far easier to work with. The usual recipe is:

1. Write down the likelihood.
2. Take the natural log.
3. Differentiate with respect to each parameter and set the derivatives to zero.
4. Solve, then check it is a maximum (second derivative negative).

Worked Example 1: The Coin (Bernoulli)

Let \( x_i = 1 \) for heads and 0 for tails, with \( P(x_i = 1) = p \). In 10 flips we observe \( k = 7 \) heads and 3 tails.

Step 1. Likelihood. Each head contributes \( p \) and each tail contributes \( 1 - p \):

$$ L(p) = p^{7}(1 - p)^{3}. $$

Step 2. Log-likelihood.

$$ \ell(p) = 7 \ln p + 3 \ln(1 - p). $$

Step 3. Differentiate and set to zero.

$$ \frac{d\ell}{dp} = \frac{7}{p} - \frac{3}{1 - p} = 0 \;\Rightarrow\; 7(1 - p) = 3p \;\Rightarrow\; 7 = 10p \;\Rightarrow\; \hat{p} = 0.7 . $$

Step 4. Check it is a maximum.

$$ \frac{d^2\ell}{dp^2} = -\frac{7}{p^2} - \frac{3}{(1 - p)^2} < 0 \quad \text{for all } 0 < p < 1 . $$

Step 5. Sanity check with numbers. \( L(0.7) = 0.7^7 \times 0.3^3 \approx 0.08235 \times 0.027 \approx 0.002224 \), while \( L(0.5) = 0.5^{10} \approx 0.000977 \). The observed sequence is more than twice as likely under \( p = 0.7 \) as under a fair coin.

In general, with \( k \) heads in \( n \) flips, the same steps give \( \hat{p} = k/n \), the sample proportion.

Worked Example 2: The Normal Distribution

Five students' scores on a short quiz (out of 10) are 4, 6, 5, 7, 8. Model them as draws from \( N(\mu, \sigma^2) \) and find the MLEs of both parameters.

Step 1. Log-likelihood. The normal density is \( \frac{1}{\sqrt{2\pi\sigma^2}} \exp\!\left(-\frac{(x - \mu)^2}{2\sigma^2}\right) \), so

$$ \ell(\mu, \sigma^2) = -\frac{n}{2}\ln(2\pi) - \frac{n}{2}\ln\sigma^2 - \frac{1}{2\sigma^2}\sum_{i=1}^{n}(x_i - \mu)^2 . $$

Step 2. Derivative with respect to \( \mu \).

$$ \frac{\partial \ell}{\partial \mu} = \frac{1}{\sigma^2}\sum_{i=1}^{n}(x_i - \mu) = 0 \;\Rightarrow\; \hat{\mu} = \frac{1}{n}\sum x_i = \bar{x}. $$

Here \( \hat{\mu} = (4 + 6 + 5 + 7 + 8)/5 = 30/5 = 6 \).

Step 3. Derivative with respect to \( \sigma^2 \).

$$ \frac{\partial \ell}{\partial \sigma^2} = -\frac{n}{2\sigma^2} + \frac{1}{2\sigma^4}\sum (x_i - \mu)^2 = 0 \;\Rightarrow\; \hat{\sigma}^2 = \frac{1}{n}\sum (x_i - \hat{\mu})^2 . $$

Deviations from 6 are \( -2, 0, -1, 1, 2 \); squared they are \( 4, 0, 1, 1, 4 \), summing to 10. So \( \hat{\sigma}^2 = 10/5 = 2 \) and \( \hat{\sigma} \approx 1.414 \).

Step 4. Note on bias. The usual sample variance divides by \( n - 1 \), giving \( 10/4 = 2.5 \). The MLE divides by \( n \) and is slightly biased downwards, though the difference vanishes as \( n \) grows. This is a good reminder that "maximum likelihood" does not automatically mean "unbiased".

A Quick Third Example: Poisson Counts

A call centre records the number of calls in five one-minute intervals: 2, 3, 1, 4, 0. For a Poisson model \( p(x \mid \lambda) = e^{-\lambda}\lambda^x/x! \), the log-likelihood is \( \ell(\lambda) = -n\lambda + (\sum x_i)\ln\lambda - \sum \ln x_i! \). Setting \( d\ell/d\lambda = -n + (\sum x_i)/\lambda = 0 \) gives \( \hat{\lambda} = \bar{x} = 10/5 = 2 \) calls per minute.

The Connection to Machine Learning Losses

Many common loss functions are simply negative log-likelihoods.

Linear regression and squared error. If \( y_i = \mathbf{w}^\top\mathbf{x}_i + \varepsilon_i \) with Gaussian noise \( \varepsilon_i \sim N(0, \sigma^2) \), then the log-likelihood is a constant minus \( \frac{1}{2\sigma^2}\sum (y_i - \mathbf{w}^\top\mathbf{x}_i)^2 \). Maximising it is exactly minimising the sum of squared errors. Least squares is MLE under Gaussian noise.

Logistic regression and cross-entropy. For a binary label with predicted probability \( \hat{p}_i \), the Bernoulli log-likelihood is

$$ \ell = \sum_{i=1}^{n} \left[ y_i \ln \hat{p}_i + (1 - y_i)\ln(1 - \hat{p}_i) \right]. $$

Its negative is the binary cross-entropy loss used to train classifiers. For example, if the true label is 1 and the model predicts \( \hat{p} = 0.8 \), the loss is \( -\ln 0.8 \approx 0.223 \); if it predicts \( 0.25 \), the loss is \( -\ln 0.25 \approx 1.386 \). Confident wrong predictions are punished heavily.

When the equations cannot be solved by hand, as in neural networks, we maximise the log-likelihood numerically with gradient-based optimizers.

Properties That Make MLE Popular

Under standard regularity conditions and for large samples, MLEs are:

• Consistent: they converge to the true parameter as \( n \to \infty \).
• Asymptotically normal: their sampling distribution is approximately normal, which allows confidence intervals.
• Asymptotically efficient: no other consistent estimator has a smaller variance in the limit.
• Invariant: if \( \hat{\theta} \) is the MLE of \( \theta \), then \( g(\hat{\theta}) \) is the MLE of \( g(\theta) \). For instance, \( \hat{\sigma} = \sqrt{2} \) above.

A weakness is overfitting with small samples; adding a prior and maximising the posterior (MAP estimation) leads to regularised methods such as ridge regression.

In practice, R's glm(), Python's statsmodels, Stata and SPSS all fit models such as logistic and Poisson regression by maximum likelihood, and report the log-likelihood in their output.

Did You Know?

• Ronald Fisher developed and named the method of maximum likelihood between about 1912 and 1922, though earlier mathematicians such as Gauss used related ideas.
• The likelihood is not a probability distribution over \( \theta \): it need not integrate to 1.
• Comparing log-likelihoods of nested models is the basis of the likelihood ratio test and of criteria such as AIC and BIC.

Key Takeaways

• MLE chooses the parameters under which the observed data are most probable.
• Work with the log-likelihood: it turns products into sums and avoids numerical underflow.
• Bernoulli, Poisson and normal MLEs give the sample proportion, sample mean, and variance divided by \( n \).
• Squared error and cross-entropy losses are negative log-likelihoods under Gaussian and Bernoulli models.
• MLEs are consistent and efficient for large samples but can be biased for small ones.

Stuck on a likelihood derivation for your statistics or machine learning course? Amathify's experts and PhD scholars offer step-by-step solutions; send your question on WhatsApp or through the Ask a Question form.