Loss Functions in Machine Learning: MSE, Cross-Entropy and More

Every machine learning model learns by minimising a number, and that number is the loss. The loss function decides what "wrong" means, how harshly different mistakes are punished, and even what the model's predictions represent. Choosing it well is as important as choosing the model itself, so in this article we work through the most common losses with real calculations.

What a Loss Function Does

For a single example with true value \(y\) and prediction \(\hat{y}\), a loss \(\ell(y, \hat{y})\) returns a non-negative number that is zero (or near zero) for a perfect prediction. The model's overall objective, often called the cost, averages this over \(n\) training examples:

$$ J(\theta) = \frac{1}{n}\sum_{i=1}^{n} \ell\bigl(y_i, \hat{y}_i(\theta)\bigr) $$

Training adjusts the parameters \(\theta\) to make \(J\) as small as possible, usually with gradient descent. That means a good loss should be:

• meaningful for the task (numbers versus categories),
• differentiable, or at least nearly so, so gradients exist,
• ideally convex for simple models, so there is a single minimum.

Regression Losses

Mean Squared Error (MSE)

$$ \text{MSE} = \frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2 $$

Squaring makes all errors positive and penalises large errors much more than small ones: an error of 4 costs 16 times an error of 1. The gradient with respect to a prediction is \(-2(y_i - \hat{y}_i)/n\), which is proportional to the error, so learning slows smoothly near the optimum. Minimising MSE makes the model predict the conditional mean of \(y\). Its weakness is sensitivity to outliers. The root mean squared error, \(\text{RMSE} = \sqrt{\text{MSE}}\), is often reported because it has the same units as \(y\).

Mean Absolute Error (MAE)

$$ \text{MAE} = \frac{1}{n}\sum_{i=1}^{n}\lvert y_i - \hat{y}_i\rvert $$

MAE grows linearly with the error, so outliers have less influence. Minimising it makes the model predict the conditional median. Its drawback is the sharp corner at zero error, where the derivative is undefined and the gradient does not shrink as you approach the optimum.

Huber Loss

Huber loss combines the best of both. For a threshold \(\delta\) and error \(a = y - \hat{y}\):

$$ L_\delta(a) = \begin{cases} \tfrac{1}{2}a^2 & \text{if } |a| \le \delta \\[4pt] \delta\left(|a| - \tfrac{1}{2}\delta\right) & \text{if } |a| > \delta \end{cases} $$

It is quadratic (smooth) for small errors and linear (robust) for large ones, and the two pieces join smoothly at \(|a| = \delta\).

Worked Example 1: Comparing Regression Losses

A model predicting house prices (in millions of PKR) gives these results:

• True values \(y\): 3, 5, 2.5, 7
• Predictions \(\hat{y}\): 2.5, 5, 4, 8

Step 1: Errors \(a = y - \hat{y}\): 0.5, 0, −1.5, −1.

Step 2: MSE. Squared errors are 0.25, 0, 2.25, 1, totalling 3.5.

$$ \text{MSE} = \frac{3.5}{4} = 0.875, \qquad \text{RMSE} = \sqrt{0.875} \approx 0.935 $$

Step 3: MAE. Absolute errors are 0.5, 0, 1.5, 1, totalling 3.

$$ \text{MAE} = \frac{3}{4} = 0.75 $$

Step 4: Huber loss with \(\delta = 1\).

• \(a = 0.5\): \(|a| \le 1\), so \(\tfrac12(0.25) = 0.125\)
• \(a = 0\): 0
• \(a = -1.5\): \(|a| > 1\), so \(1(1.5 - 0.5) = 1.0\)
• \(a = -1\): \(|a| \le 1\), so \(\tfrac12(1) = 0.5\)

$$ \text{Huber} = \frac{0.125 + 0 + 1.0 + 0.5}{4} = 0.40625 $$

The third house, with the largest error, contributes 2.25 of the 3.5 total to MSE (64%) but only 1.5 of 3 to MAE (50%). That is the outlier sensitivity of MSE in action. Under Huber loss, it contributes 1.0 of 1.625, about 62%, but this share stays bounded as errors grow because the large-error branch is linear.

Classification Losses

Binary Cross-Entropy

For a label \(y \in \{0,1\}\) and predicted probability \(\hat{p} = P(y=1)\):

$$ \text{BCE} = -\frac{1}{n}\sum_{i=1}^{n}\bigl[y_i\ln\hat{p}_i + (1-y_i)\ln(1-\hat{p}_i)\bigr] $$

Only one term is active for each example: \(-\ln\hat{p}\) when \(y = 1\) and \(-\ln(1-\hat{p})\) when \(y = 0\). Minimising BCE is the same as maximising the likelihood of the data under a Bernoulli model, which is why it pairs naturally with the sigmoid output of logistic regression and neural networks.

Categorical Cross-Entropy

With \(K\) classes, one-hot label vector \(\mathbf{y}\) and softmax probabilities \(\hat{\mathbf{p}}\):

$$ \text{CCE} = -\sum_{k=1}^{K} y_k \ln \hat{p}_k $$

Since only the true class has \(y_k = 1\), this is simply \(-\ln(\text{probability given to the correct class})\).

Worked Example 2: Cross-Entropy by Hand

Part (a): Binary. A spam filter produces these outputs on four emails:

• Email 1: \(y = 1\), \(\hat{p} = 0.9\), loss \(= -\ln 0.9 = 0.1054\)
• Email 2: \(y = 0\), \(\hat{p} = 0.2\), loss \(= -\ln(1 - 0.2) = -\ln 0.8 = 0.2231\)
• Email 3: \(y = 1\), \(\hat{p} = 0.6\), loss \(= -\ln 0.6 = 0.5108\)
• Email 4: \(y = 0\), \(\hat{p} = 0.4\), loss \(= -\ln 0.6 = 0.5108\)

$$ \text{BCE} = \frac{0.1054 + 0.2231 + 0.5108 + 0.5108}{4} = \frac{1.3501}{4} \approx 0.3375 $$

The confident, correct prediction on Email 1 costs very little; the hesitant ones on Emails 3 and 4 cost about five times more.

Part (b): Categorical. An image classifier outputs probabilities 0.7 (cat), 0.2 (dog) and 0.1 (bird).

• If the image really is a cat: loss \(= -\ln 0.7 \approx 0.357\).
• If it is actually a bird: loss \(= -\ln 0.1 \approx 2.303\).

Being confidently wrong is penalised more than six times as heavily. As \(\hat{p} \to 0\) for the true class, the loss goes to infinity, which strongly pushes the model away from certainty in wrong answers.

Why Not MSE for Classification?

With a sigmoid output, the gradient of MSE contains the factor \(\sigma'(z)\), which is tiny when the model is confidently wrong, so learning stalls. With cross-entropy that factor cancels and the gradient becomes simply \(\hat{p} - y\), large exactly when the error is large.

Hinge Loss

Support vector machines use labels \(y \in \{-1, +1\}\) and a raw score \(f(x)\):

$$ \ell_{\text{hinge}} = \max\bigl(0,\; 1 - y f(x)\bigr) $$

A point classified correctly with a margin (\(yf(x) \ge 1\)) costs nothing. For example, \(y = 1\), \(f = 0.4\) gives \(1 - 0.4 = 0.6\): correct side, but too close to the boundary. And \(y = -1\), \(f = 0.6\) gives \(1 + 0.6 = 1.6\): wrong side, larger penalty.

Choosing a Loss: A Quick Guide

Did You Know?

• Cross-entropy comes from information theory: it measures the average number of nats (or bits, with base-2 logs) needed to encode outcomes using the model's probabilities instead of the true ones.
• Peter Huber introduced his loss in 1964 as part of the field of robust statistics.
• Libraries compute cross-entropy from raw scores ("logits") rather than probabilities to avoid taking the logarithm of numbers that round to zero.

Key Takeaways

• MSE punishes large errors heavily and targets the mean; MAE is robust and targets the median.
• Huber loss is quadratic for small errors and linear for large ones.
• Cross-entropy is the negative log-probability of the correct answer and is the standard loss for classification.
• Hinge loss rewards correct classification with a margin and underlies support vector machines.

Struggling with loss-function derivations, gradient calculations or a machine learning assignment? Amathify's experts and PhD scholars provide step-by-step solutions; contact us on WhatsApp or use the Ask a Question form.