The Bias–Variance Tradeoff

Every machine learning model makes mistakes, but not all mistakes have the same cause. Some models are wrong because they are too rigid to capture the pattern; others are wrong because they react too strongly to the particular data they happened to see. The bias–variance tradeoff gives us a precise mathematical language for these two failures and explains why the best model usually sits somewhere in the middle.

The Setting

Assume the data are generated by

$$ y = f(x) + \varepsilon, \qquad \mathbb{E}[\varepsilon] = 0, \quad \operatorname{Var}(\varepsilon) = \sigma^2 . $$

Here \( f \) is the unknown true function and \( \varepsilon \) is random noise. We collect a training set \( D \), fit a model and obtain a prediction function \( \hat{f}_D \). The crucial idea is that \( D \) is random: had we sampled a different training set, we would have obtained a different \( \hat{f}_D \). So for a fixed input \( x_0 \), the prediction \( \hat{f}_D(x_0) \) is itself a random variable.

The Decomposition

We want the expected squared error at \( x_0 \), averaging over both the noise in a new observation and the randomness of the training set. Write \( \bar{f}(x_0) = \mathbb{E}_D[\hat{f}_D(x_0)] \) for the average prediction. Then

$$ \mathbb{E}\left[(y_0 - \hat{f}_D(x_0))^2\right] = \underbrace{\left(\bar{f}(x_0) - f(x_0)\right)^2}_{\text{Bias}^2} + \underbrace{\mathbb{E}_D\left[(\hat{f}_D(x_0) - \bar{f}(x_0))^2\right]}_{\text{Variance}} + \underbrace{\sigma^2}_{\text{Irreducible noise}} . $$

Where the Formula Comes From

Write \( y_0 - \hat{f}_D = (f - \bar{f}) + (\bar{f} - \hat{f}_D) + \varepsilon \), dropping the argument \( x_0 \) for brevity. Square it and take expectations:

1. The noise \( \varepsilon \) is independent of \( D \) and has mean zero, so every cross term containing \( \varepsilon \) vanishes, leaving \( \sigma^2 \).
2. The term \( f - \bar{f} \) is a constant, and \( \mathbb{E}_D[\bar{f} - \hat{f}_D] = 0 \) by the definition of \( \bar{f} \), so the cross term between them is zero as well.
3. What remains is \( (f - \bar{f})^2 + \mathbb{E}_D[(\hat{f}_D - \bar{f})^2] + \sigma^2 \).

Each term has a clear meaning:

• Bias measures how far the average prediction is from the truth. It reflects the model's built-in assumptions.
• Variance measures how much predictions wobble from one training set to the next.
• Noise is the floor that no model can beat, because \( \varepsilon \) is genuinely unpredictable.

The Dartboard Picture

Imagine four archers, each shooting many arrows, where each arrow corresponds to a model trained on a different dataset and the bullseye is \( f(x_0) \):

• Low bias, low variance: arrows clustered tightly on the bullseye. The ideal.
• High bias, low variance: arrows tightly clustered but off-centre. A consistent but wrong model, such as a straight line fitted to a curve.
• Low bias, high variance: arrows centred on the bullseye on average but scattered everywhere. A very flexible model that chases noise.
• High bias, high variance: scattered and off-centre. The worst of both.

Worked Example 1: Comparing Two Models at One Point

Suppose the true value at a test point is \( f(x_0) = 10 \) and the noise variance is \( \sigma^2 = 1 \). We train each model on four independent training sets and record its prediction at \( x_0 \).

• Model A (flexible): predictions 8, 9, 11, 12.
• Model B (rigid): predictions 7, 7.5, 7, 7.5.

Model A.

Mean prediction: \( (8 + 9 + 11 + 12)/4 = 40/4 = 10 \). Bias \( = 10 - 10 = 0 \), so bias\(^2 = 0\).

Variance: deviations from 10 are \( -2, -1, 1, 2 \), with squares \( 4, 1, 1, 4 \). Average \( = 10/4 = 2.5 \).

Expected error \( = 0 + 2.5 + 1 = 3.5 \).

Model B.

Mean prediction: \( (7 + 7.5 + 7 + 7.5)/4 = 29/4 = 7.25 \). Bias \( = 7.25 - 10 = -2.75 \), so bias\(^2 = 7.5625 \).

Variance: deviations from 7.25 are \( -0.25, 0.25, -0.25, 0.25 \), squares all \( 0.0625 \). Average \( = 0.0625 \).

Expected error \( = 7.5625 + 0.0625 + 1 = 8.625 \).

Model B is far more stable, but its systematic error dominates. Here the flexible model wins. With noisier data, the variance of Model A would grow and the verdict could flip.

Worked Example 2: Shrinking an Estimator Can Lower Error

Accepting some bias can reduce total error. Suppose we estimate a population mean \( \mu = 4 \) from \( n = 9 \) observations with variance \( 9 \). The sample mean \( \bar{x} \) is unbiased with \( \operatorname{Var}(\bar{x}) = 9/9 = 1 \).

Consider the shrunk estimator \( \hat{\mu}_c = c\,\bar{x} \) for a constant \( 0 < c \le 1 \).

Step 1. Bias. \( \mathbb{E}[c\bar{x}] = c\mu \), so bias \( = (c - 1)\mu = 4(c - 1) \) and bias\(^2 = 16(c - 1)^2 \).

Step 2. Variance. \( \operatorname{Var}(c\bar{x}) = c^2 \operatorname{Var}(\bar{x}) = c^2 \).

Step 3. Mean squared error.

$$ \text{MSE}(c) = 16(c - 1)^2 + c^2 . $$

Step 4. Minimise. Differentiate and set to zero: \( 32(c - 1) + 2c = 0 \Rightarrow 34c = 32 \Rightarrow c = 16/17 \approx 0.941 \).

Step 5. Compare. At \( c = 1 \), MSE \( = 0 + 1 = 1 \). At \( c = 16/17 \):

$$ \text{MSE} = 16\left(\tfrac{1}{17}\right)^2 + \left(\tfrac{16}{17}\right)^2 = \frac{16 + 256}{289} = \frac{272}{289} = \frac{16}{17} \approx 0.941 . $$

By introducing a small bias we cut the error by about 6%. This is precisely the logic behind ridge regression and other shrinkage methods. (In practice \( \mu \) is unknown, so \( c \) must be chosen from data, but the principle holds.)

Complexity and the U-Shaped Curve

As model complexity increases (higher polynomial degree, deeper trees, smaller \( k \) in k-nearest neighbours):

• Bias generally decreases, because the model can represent more shapes.
• Variance generally increases, because the model has more freedom to fit noise.

Total test error is their sum plus the noise floor, so it typically falls, reaches a minimum, then rises again. Training error, by contrast, keeps falling, which is why training error alone is a misleading guide.

For k-nearest neighbours the effect is especially transparent. With \( k = 1 \) the prediction copies one noisy neighbour, giving low bias but high variance. With very large \( k \) the prediction averages over distant points, giving low variance but high bias.

Practical Levers

Bagging methods such as random forests reduce variance by averaging many high-variance trees, while boosting mainly reduces bias by adding weak learners one at a time.

Did You Know?

• Very large neural networks sometimes show "double descent": test error rises near the point where the model can just fit the training data perfectly, then falls again as the model grows even larger.
• The noise term \( \sigma^2 \) sets a hard limit: even the true function \( f \) itself has expected squared error \( \sigma^2 \).
• Averaging \( m \) independent predictors each with variance \( v \) gives variance \( v/m \), which is why ensembles work so well.

Key Takeaways

• Expected prediction error equals bias squared plus variance plus irreducible noise.
• Bias comes from wrong assumptions; variance comes from sensitivity to the particular training set.
• Increasing complexity lowers bias but raises variance, producing a U-shaped test-error curve.
• A deliberately biased estimator, such as a shrunk mean, can have lower total error than an unbiased one.
• Diagnose with training versus validation error, then choose remedies that target the dominant component.

Need help with a machine learning assignment or a derivation like this one? Amathify's experts and PhD scholars provide step-by-step solutions; reach us on WhatsApp or through the Ask a Question form.