Bayesian Statistics for Beginners

Bayesian statistics is a way of reasoning about uncertainty that mirrors how sensible people think: start with what you believe, look at the evidence, and update your belief. It has moved from a niche topic to the heart of modern data science, spam filtering, medical research and machine learning. This guide explains the core ideas from scratch, with fully worked examples.

Two Ways to Think About Probability

There are two main schools of statistics:

• Frequentist: probability is the long-run frequency of an event in repeated trials. A coin has probability 0.5 of heads because, over thousands of tosses, about half are heads. Unknown quantities (like the true bias of a coin) are fixed numbers, not random.
• Bayesian: probability measures a degree of belief, given the information available. Unknown quantities can be described by probability distributions, which are updated as data arrives.

Both approaches are useful. The Bayesian view is especially natural when you have prior knowledge, small samples, or need to update conclusions continuously.

A Short History

The idea goes back to Thomas Bayes (around 1701–1761), an English Presbyterian minister from Tunbridge Wells. His essay on "the doctrine of chances" was found after his death by his friend Richard Price, who edited it and had it published by the Royal Society in 1763. The French mathematician Pierre-Simon Laplace independently developed the idea in 1774 and applied it far more widely, for example to estimate the proportion of boys among births in Paris. For much of the twentieth century Bayesian methods were controversial, but cheap computing power from the 1990s onwards made them practical and popular.

Bayes' Theorem

For a hypothesis \( H \) and evidence \( E \):

$$ P(H \mid E) = \frac{P(E \mid H)\,P(H)}{P(E)} $$

Each part has a name:

• \( P(H) \) is the prior: your belief before seeing the evidence.
• \( P(E \mid H) \) is the likelihood: how probable the evidence is if \( H \) is true.
• \( P(H \mid E) \) is the posterior: your updated belief after seeing the evidence.
• \( P(E) \) is the total probability of the evidence, which makes everything add to 1. With two possibilities,

$$ P(E) = P(E \mid H)P(H) + P(E \mid \text{not } H)P(\text{not } H) $$

A memorable summary is: posterior is proportional to likelihood times prior.

Worked Example 1: The Medical Test Puzzle

Problem: A disease affects 1% of a population. A test detects it 95% of the time when present (sensitivity), but gives a false positive 5% of the time when absent. A randomly chosen person tests positive. What is the probability they have the disease?

Step 1 (prior). \( P(D) = 0.01 \) and \( P(\text{not } D) = 0.99 \).

Step 2 (likelihoods). \( P(+ \mid D) = 0.95 \) and \( P(+ \mid \text{not } D) = 0.05 \).

Step 3 (total probability of a positive).

$$ P(+) = 0.95 \times 0.01 + 0.05 \times 0.99 = 0.0095 + 0.0495 = 0.059 $$

Step 4 (posterior).

$$ P(D \mid +) = \frac{0.0095}{0.059} \approx 0.161 $$

Step 5 (interpret). Only about 16%, not 95%! Picture 10,000 people: 100 have the disease and 95 of them test positive, but 9,900 are healthy and 495 of them also test positive. Of the 590 positives, only 95 are genuinely ill, and \( 95/590 \approx 0.161 \). The low prior (the disease is rare) dominates. Doctors call ignoring this the base-rate fallacy.

Updating again

Suppose the person takes a second, independent test and it is also positive. The posterior from the first test becomes the new prior, \( P(D) = 0.161 \):

$$ P(D \mid ++) = \frac{0.95 \times 0.161}{0.95 \times 0.161 + 0.05 \times 0.839} = \frac{0.1530}{0.1530 + 0.0420} \approx 0.785 $$

After two positives the probability jumps to about 78%. This step-by-step updating is the essence of Bayesian thinking: today's posterior is tomorrow's prior.

From Single Numbers to Whole Distributions

In real problems the unknown is often a quantity like a success rate \( \theta \) between 0 and 1. A Bayesian puts a probability distribution on \( \theta \) itself. A very convenient choice for proportions is the Beta distribution, \( \text{Beta}(a, b) \), whose mean is

$$ \frac{a}{a + b} $$

You can think of \( a \) and \( b \) as "imaginary prior successes and failures". The magic is that if the prior is \( \text{Beta}(a, b) \) and you then observe \( k \) successes in \( n \) trials, the posterior is simply

$$ \text{Beta}(a + k,\ b + n - k) $$

This is called a conjugate prior: the update is just addition.

Worked Example 2: Estimating a Free-Throw Success Rate

Problem: A new basketball player's free-throw success rate \( \theta \) is unknown. Before watching, a coach believes \( \theta \) is probably around 50% but is unsure, and uses the prior \( \text{Beta}(2, 2) \). The player then scores 7 out of 10 shots. Find the posterior and its mean.

Step 1 (prior mean).

$$ \frac{2}{2 + 2} = 0.5 $$

Step 2 (update). With \( k = 7 \) successes and \( n - k = 3 \) misses:

$$ \text{Beta}(2 + 7,\ 2 + 3) = \text{Beta}(9, 5) $$

Step 3 (posterior mean).

$$ \frac{9}{9 + 5} = \frac{9}{14} \approx 0.643 $$

Step 4 (compare). The simple frequentist estimate is \( 7/10 = 0.70 \). The Bayesian estimate, 0.643, is pulled slightly towards the prior belief of 0.5, because 10 shots is not much evidence. With 1,000 shots the data would completely dominate and the two estimates would almost agree.

Step 5 (uncertainty). From the \( \text{Beta}(9, 5) \) distribution, a 95% credible interval for \( \theta \) runs from about 0.39 to 0.86. A Bayesian can say directly: "there is a 95% probability the true rate lies in this range", a statement the frequentist confidence interval does not strictly allow.

Where Bayesian Statistics Is Used

1. Spam filters estimate the probability that an email is spam given the words it contains.
2. Medicine uses Bayesian methods to design adaptive clinical trials that update as patient results come in.
3. Search and rescue: the US Coast Guard uses Bayesian search theory to update where a lost vessel is likely to be.
4. Machine learning: Bayesian networks, Gaussian processes and probabilistic programming all rest on Bayes' theorem.
5. Sports and elections: forecasters combine polls and past results as priors and update daily.

Did You Know?

• Bayesian search methods helped locate the wreck of the submarine USS Scorpion in 1968 and the wreckage of Air France Flight 447 in 2011.
• Alan Turing used Bayesian reasoning, measured in units he called "bans", to help break German Enigma codes during the Second World War.
• The name "Bayesian" for this approach only became common in the 1950s, about two centuries after Bayes died.

Key Takeaways

• Bayesian statistics treats probability as a degree of belief that is updated by evidence.
• Bayes' theorem: posterior \( \propto \) likelihood \( \times \) prior.
• Low prior probabilities matter: a positive result from a 95%-accurate test for a 1% disease means only about a 16% chance of having it.
• Conjugate priors such as the Beta distribution make updating as simple as adding counts.
• Credible intervals give direct probability statements about unknown quantities.

Need help with probability, statistics projects or data analysis assignments? Amathify's experts and PhD scholars offer step-by-step solutions, reach us via WhatsApp or the Ask a Question form.