Naive Bayes Classifiers Explained

Naive Bayes is one of the oldest and fastest classifiers in machine learning, and it still powers spam filters, sentiment analysis and document sorting today. It rests on Bayes' theorem plus one bold simplifying assumption, and that assumption is what makes it both "naive" and remarkably efficient. Let us build it from the ground up and classify some emails by hand.

Bayes' Theorem

For a class \(C\) and observed evidence \(\mathbf{x}\), Bayes' theorem says

$$ P(C \mid \mathbf{x}) = \frac{P(\mathbf{x} \mid C)\,P(C)}{P(\mathbf{x})} $$

Each piece has a name:

• \(P(C)\) is the prior: how common the class is before we see any evidence.
• \(P(\mathbf{x} \mid C)\) is the likelihood: how probable the evidence is if the class is true.
• \(P(\mathbf{x})\) is the evidence: the overall probability of what we observed.
• \(P(C \mid \mathbf{x})\) is the posterior: our updated belief after seeing the evidence.

To classify, we pick the class with the highest posterior. Since \(P(\mathbf{x})\) is the same for every class, we can ignore it when comparing and simply maximise \(P(\mathbf{x} \mid C)P(C)\).

The "Naive" Assumption

The difficulty is the likelihood. If \(\mathbf{x}\) contains \(d\) features (say, which of 10,000 words appear in an email), estimating the joint probability \(P(x_1, x_2, \dots, x_d \mid C)\) directly would need astronomically many examples. Naive Bayes assumes that the features are conditionally independent given the class:

$$ P(x_1, x_2, \dots, x_d \mid C) = \prod_{j=1}^{d} P(x_j \mid C) $$

So the classifier becomes

$$ \hat{C} = \arg\max_{C}\; P(C)\prod_{j=1}^{d}P(x_j \mid C) $$

The assumption is usually false. In real emails, "free" and "offer" tend to appear together. Yet Naive Bayes often classifies well anyway, because it only needs to rank the classes correctly, not estimate the probabilities exactly.

Worked Example 1: Is This Email Spam?

From past data, a company knows that 40% of its emails are spam. It has also estimated:

• \(P(\text{"free"} \mid \text{spam}) = 0.5\) and \(P(\text{"free"} \mid \text{ham}) = 0.05\)
• \(P(\text{"win"} \mid \text{spam}) = 0.3\) and \(P(\text{"win"} \mid \text{ham}) = 0.02\)

A new email contains both "free" and "win". Is it spam?

Step 1: Priors. \(P(\text{spam}) = 0.4\) and \(P(\text{ham}) = 0.6\).

Step 2: Unnormalised score for spam.

$$ P(\text{spam})\,P(\text{free} \mid \text{spam})\,P(\text{win} \mid \text{spam}) = 0.4 \times 0.5 \times 0.3 = 0.06 $$

Step 3: Unnormalised score for ham.

$$ 0.6 \times 0.05 \times 0.02 = 0.0006 $$

Step 4: Normalise to get the posterior.

$$ P(\text{spam} \mid \text{free, win}) = \frac{0.06}{0.06 + 0.0006} = \frac{0.06}{0.0606} \approx 0.990 $$

The email is spam with about 99% probability. Notice how two moderately suspicious words multiply together to give overwhelming evidence, even though spam was not the more common class a priori.

Training from Word Counts

In practice the probabilities are estimated from data. In the multinomial Naive Bayes model, used for text, we count how often each word appears in each class:

$$ P(w \mid C) = \frac{\text{count}(w, C)}{\text{total words in } C} $$

The Zero-Frequency Problem

If a word never appeared in spam during training, this formula gives \(P(w \mid \text{spam}) = 0\), and a single such word would force the whole spam probability to zero, no matter how much other evidence there is. The fix is Laplace (add-one) smoothing:

$$ P(w \mid C) = \frac{\text{count}(w, C) + 1}{\text{total words in } C + V} $$

where \(V\) is the vocabulary size. Every word gets a small non-zero probability.

Worked Example 2: Laplace Smoothing in Action

A tiny training set has 4 spam emails and 6 ham emails, with a vocabulary of \(V = 6\) words. The word counts are:

Classify the message "free project".

Step 1: Priors. \(P(\text{spam}) = 4/10 = 0.4\) and \(P(\text{ham}) = 6/10 = 0.6\).

Step 2: Notice the problem. Without smoothing, \(P(\text{project} \mid \text{spam}) = 0/20 = 0\), so the spam score would be exactly zero. One word would veto everything.

Step 3: Smoothed likelihoods. Spam denominators are \(20 + 6 = 26\); ham denominators are \(30 + 6 = 36\).

$$ P(\text{free} \mid \text{spam}) = \frac{6 + 1}{26} = \frac{7}{26} \approx 0.2692, \qquad P(\text{project} \mid \text{spam}) = \frac{0 + 1}{26} \approx 0.0385 $$

$$ P(\text{free} \mid \text{ham}) = \frac{1 + 1}{36} \approx 0.0556, \qquad P(\text{project} \mid \text{ham}) = \frac{9 + 1}{36} \approx 0.2778 $$

Step 4: Scores.

$$ \text{spam: } 0.4 \times \frac{7}{26} \times \frac{1}{26} \approx 0.004142, \qquad \text{ham: } 0.6 \times \frac{2}{36} \times \frac{10}{36} \approx 0.009259 $$

Step 5: Posterior.

$$ P(\text{spam} \mid \text{free, project}) = \frac{0.004142}{0.004142 + 0.009259} \approx 0.309 $$

The message is classified as ham, with about 69% confidence. "Free" points towards spam, but "project" points even more strongly towards a genuine work email. For comparison, the message "win money" would give a spam posterior of about 0.953 with the same model.

Working in Log Space

With thousands of words, multiplying many small probabilities produces numbers too small for a computer to store (underflow). Since the logarithm is increasing, we can compare sums of logs instead:

$$ \ln P(C) + \sum_{j}\ln P(x_j \mid C) $$

For Example 2, the spam log-score is \(\ln 0.4 + \ln(7/26) + \ln(1/26) \approx -5.487\), and the ham log-score is \(\ln 0.6 + \ln(2/36) + \ln(10/36) \approx -4.682\). Ham has the larger (less negative) value, giving the same decision.

Other Flavours of Naive Bayes

• Multinomial NB: word counts, for text classification.
• Bernoulli NB: binary features recording only whether each word is present or absent.
• Gaussian NB: continuous features such as height or blood pressure, assuming each follows a normal distribution within each class:

$$ P(x_j \mid C) = \frac{1}{\sqrt{2\pi\sigma_C^2}}\exp\!\left(-\frac{(x_j - \mu_C)^2}{2\sigma_C^2}\right) $$

where \(\mu_C\) and \(\sigma_C^2\) are the mean and variance of that feature in class \(C\).

All three are available in Python's scikit-learn (MultinomialNB, BernoulliNB, GaussianNB) and in R through packages such as e1071.

Strengths and Weaknesses

Strengths:

1. Training is a single pass of counting, so it is extremely fast.
2. It works well with small datasets and very many features.
3. It is easy to explain and to update when new data arrives.

Weaknesses:

1. The independence assumption ignores correlations between features.
2. Its probability estimates are often overconfident, even when its classifications are right.
3. Strongly correlated features get "double counted".

Did You Know?

• Bayes' theorem comes from an essay by the Reverend Thomas Bayes, published in 1763, two years after his death, by his friend Richard Price.
• Paul Graham's 2002 essay "A Plan for Spam" popularised Bayesian filtering and influenced many early email spam filters.
• Naive Bayes is often a strong baseline for text classification: a model that cannot beat it is probably not worth its extra complexity.

Key Takeaways

• Naive Bayes chooses the class that maximises \(P(C)\prod_j P(x_j \mid C)\).
• It assumes features are independent given the class, which is rarely true but often good enough.
• Laplace smoothing prevents unseen words from forcing probabilities to zero.
• Summing log-probabilities avoids numerical underflow without changing the decision.

If Bayes' theorem problems, probability assignments or a text classification project are giving you trouble, Amathify's experts and PhD scholars offer step-by-step solutions. Reach us on WhatsApp or via the Ask a Question form.