Probability and Statistics in Machine Learning

Machine learning is often described as "learning from data", but data is noisy, incomplete and uncertain. Probability gives us the language to describe that uncertainty, and statistics gives us the tools to draw reliable conclusions from samples. This article walks through the core probability and statistics ideas that every machine learning student needs, with worked examples at each stage.

Why Probability Matters

A spam filter never knows for certain that an email is spam; it estimates a probability. A medical model does not say "this patient has the disease"; it says "there is an 83% chance". Thinking in probabilities lets a model express confidence, combine evidence, and make decisions that balance different kinds of mistakes.

Three basic rules underpin everything:

• Complement: \( P(\text{not } A) = 1 - P(A) \).
• Product rule: \( P(A \cap B) = P(A \mid B)\,P(B) \).
• Total probability: \( P(A) = P(A \mid B)P(B) + P(A \mid \text{not } B)P(\text{not } B) \).

Bayes' Theorem: Updating Beliefs

Bayes' theorem, published after the death of the English minister Thomas Bayes in 1763 and developed further by Pierre-Simon Laplace, tells us how to reverse a conditional probability:

$$ P(H \mid E) = \frac{P(E \mid H)\,P(H)}{P(E)} $$

Here \( P(H) \) is the prior belief in hypothesis \( H \), \( P(E \mid H) \) is the likelihood of the evidence, and \( P(H \mid E) \) is the posterior, the updated belief.

Worked Example 1: A Spam Filter

Suppose 30% of emails are spam. The word "free" appears in 60% of spam emails but only 5% of genuine ("ham") emails. An email contains "free". What is the probability it is spam?

Step 1. List the probabilities. \( P(S) = 0.3 \), \( P(H) = 0.7 \), \( P(F \mid S) = 0.6 \), \( P(F \mid H) = 0.05 \).

Step 2. Total probability of seeing "free".

$$ P(F) = 0.6(0.3) + 0.05(0.7) = 0.18 + 0.035 = 0.215 $$

Step 3. Apply Bayes.

$$ P(S \mid F) = \frac{0.18}{0.215} \approx 0.837 $$

Step 4. Interpret. One word lifted the spam probability from 30% to about 84%. A Naive Bayes classifier repeats this for every word, assuming the words are independent given the class, and multiplies the likelihoods together. Despite that "naive" assumption, it works remarkably well for text.

Random Variables and Distributions

A random variable assigns a number to each outcome. Its distribution tells us how likely each value is. In machine learning you will constantly meet:

• Bernoulli: a single yes/no outcome with probability \( p \) (a click, a fraud flag).
• Binomial: the number of successes in \( n \) independent trials.
• Normal (Gaussian): the bell curve, with mean \( \mu \) and standard deviation \( \sigma \). Measurement errors and many natural features are roughly normal.

The two most important summaries are the mean (expected value) and the variance:

$$ \mu = \frac{1}{n}\sum x_i, \qquad \sigma^2 = \frac{1}{n}\sum (x_i - \mu)^2 $$

Statistics for Preparing Data

Many algorithms, such as gradient descent, k-nearest neighbours and support vector machines, behave badly when features have very different scales (for example, age in years beside income in rupees). The fix is standardisation, converting each value to a z-score:

$$ z = \frac{x - \mu}{\sigma} $$

Worked Example 2: Standardising a Feature

A feature has values \( 2, 4, 4, 4, 5, 5, 7, 9 \). Standardise the value 9.

Step 1. Mean. Sum \( = 2 + 4 + 4 + 4 + 5 + 5 + 7 + 9 = 40 \), so \( \mu = 40/8 = 5 \).

Step 2. Squared deviations. \( (2-5)^2 = 9 \), three values of \( (4-5)^2 = 1 \), two values of \( (5-5)^2 = 0 \), \( (7-5)^2 = 4 \), \( (9-5)^2 = 16 \).

Step 3. Variance. Total \( = 9 + 3 + 0 + 4 + 16 = 32 \), so \( \sigma^2 = 32/8 = 4 \) and \( \sigma = 2 \).

Step 4. z-score. \( z = (9 - 5)/2 = 2 \). The value 9 is two standard deviations above the mean. After standardisation, every feature has mean 0 and standard deviation 1, so no feature dominates just because of its units.

Maximum Likelihood: How Models Learn Parameters

How does a model choose its parameters? A central principle is maximum likelihood estimation (MLE): pick the parameter values that make the observed data most probable.

Worked Example 3: Estimating a Click Rate

An advert is shown 10 times and clicked 7 times. Model each view as Bernoulli with click probability \( p \). Find the MLE of \( p \).

Step 1. Likelihood. \( L(p) = p^7 (1 - p)^3 \).

Step 2. Take logs (easier to differentiate, same maximum):

$$ \ell(p) = 7\ln p + 3\ln(1 - p) $$

Step 3. Differentiate and set to zero.

$$ \frac{7}{p} - \frac{3}{1 - p} = 0 \;\Rightarrow\; 7(1 - p) = 3p \;\Rightarrow\; p = 0.7 $$

Step 4. Interpret. The MLE is simply the observed proportion, 7 out of 10. Remarkably, minimising cross-entropy loss in logistic regression and neural networks is exactly maximising a log-likelihood like this one.

Evaluating Models: Beyond Accuracy

Statistics also tells us whether a model is any good. Consider a fraud detector tested on 1000 transactions:

• Accuracy \( = \dfrac{40 + 930}{1000} = 0.97 \)
• Precision \( = \dfrac{TP}{TP + FP} = \dfrac{40}{50} = 0.80 \)
• Recall \( = \dfrac{TP}{TP + FN} = \dfrac{40}{60} \approx 0.667 \)

The F1 score is the harmonic mean of precision and recall:

$$ F_1 = \frac{2(0.80)(0.667)}{0.80 + 0.667} \approx 0.727 $$

Accuracy looks excellent at 97%, but the model misses a third of real fraud cases. When classes are unbalanced, precision and recall tell the true story; a model that always says "normal" would still score 94% accuracy here.

Overfitting and the Bias-Variance Trade-off

A model trained on a sample can memorise noise rather than learn the pattern. Statisticians describe its error in two parts: bias (being systematically wrong because the model is too simple) and variance (being unstable because it is too sensitive to the particular sample). Techniques such as cross-validation, regularisation and collecting more data all aim to find the balance that generalises best to new data.

Did You Know?

• Laplace used Bayesian reasoning in the 1780s to show that slightly more boys than girls are born in Paris, one of the earliest uses of statistics on real data.
• The term "regression" comes from Francis Galton, who in the 1880s noticed that children of very tall parents tend to be less extreme, "regressing" towards the average height.

Key Takeaways

• Bayes' theorem updates a prior belief into a posterior using evidence, and underpins Naive Bayes classifiers.
• Means, variances and z-scores are essential for preparing features.
• Maximum likelihood estimation chooses parameters that make the data most probable, and connects directly to cross-entropy loss.
• Precision, recall and F1 are more honest than accuracy on unbalanced data.
• Good models balance bias and variance to generalise beyond the training data.

For step-by-step solutions in probability, statistics or a machine learning assignment, get help from Amathify's subject experts and PhD scholars via WhatsApp or the Ask a Question form.