Evaluating Models: Precision, Recall, F1 and ROC Curves

A model that detects a rare disease could score 99% accuracy simply by declaring every patient healthy, and it would be completely useless. Accuracy hides the kinds of mistakes a classifier makes, and in real applications those mistakes rarely cost the same. This article builds the standard evaluation toolkit from the confusion matrix up to ROC curves, with every number calculated by hand.

The Confusion Matrix

For a binary classifier, every prediction falls into one of four boxes:

A false positive is a false alarm, such as a genuine bank transaction flagged as fraud. A false negative is a miss, such as a fraudulent transaction that slips through. Almost every metric is a ratio built from these four counts.

The Core Metrics

Accuracy is the fraction of all predictions that are correct:

$$ \text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}. $$

Precision asks: of everything the model flagged as positive, how much really was positive?

$$ \text{Precision} = \frac{TP}{TP + FP}. $$

Recall (also called sensitivity or true positive rate) asks: of all the real positives, how many did the model find?

$$ \text{Recall} = \frac{TP}{TP + FN}. $$

Specificity is the recall of the negative class, \( TN/(TN + FP) \), and the false positive rate is \( \text{FPR} = FP/(FP + TN) = 1 - \text{specificity} \).

The F1 score combines precision \( P \) and recall \( R \) using the harmonic mean:

$$ F_1 = \frac{2PR}{P + R}. $$

The harmonic mean is dominated by the smaller value, so a model cannot achieve a high F1 by being excellent at one metric and terrible at the other. If you care more about recall, the general \( F_\beta \) score weights recall \( \beta \) times as heavily:

$$ F_\beta = \frac{(1 + \beta^2) P R}{\beta^2 P + R}. $$

Worked Example 1: A Fraud Detector

A bank tests a fraud model on 1000 transactions, of which 60 are actually fraudulent. The model flags 50 transactions, and 40 of those are genuinely fraud.

Step 1. Fill the confusion matrix.

• \( TP = 40 \) (flagged and fraudulent)
• \( FP = 50 - 40 = 10 \) (flagged but genuine)
• \( FN = 60 - 40 = 20 \) (fraud that was missed)
• \( TN = 1000 - 40 - 10 - 20 = 930 \)

Step 2. Accuracy. \( (40 + 930)/1000 = 970/1000 = 0.97 \), or 97%.

Step 3. Precision. \( 40/(40 + 10) = 40/50 = 0.80 \).

Step 4. Recall. \( 40/(40 + 20) = 40/60 \approx 0.667 \).

Step 5. F1.

$$ F_1 = \frac{2(0.8)(0.6667)}{0.8 + 0.6667} = \frac{1.0667}{1.4667} \approx 0.727 . $$

Step 6. Compare with a lazy baseline. A model that labels everything "genuine" has \( TP = 0 \), \( TN = 940 \), \( FN = 60 \), so its accuracy is \( 940/1000 = 94\% \). That sounds impressive, yet its recall is 0. The 3 percentage-point accuracy gap hides the difference between catching two-thirds of the fraud and catching none.

Interpretation. When this model raises an alarm it is right 80% of the time, but it lets one fraud in three through. Whether that is acceptable depends on the cost of each error.

Thresholds and the Precision–Recall Tradeoff

Most classifiers output a score or probability, and we choose a threshold: predict positive if the score is at least \( \tau \).

• Lowering \( \tau \) flags more cases: recall goes up, but precision usually goes down.
• Raising \( \tau \) flags fewer cases: precision goes up, but recall goes down.

The right threshold depends on the application. A cancer screening test should favour recall, because a missed case is very costly and follow-up tests can filter false alarms. A spam filter should favour precision, because losing a genuine email is worse than seeing one spam message.

ROC Curves and AUC

The receiver operating characteristic (ROC) curve shows performance across every possible threshold at once. For each threshold we plot the true positive rate (recall) on the vertical axis against the false positive rate on the horizontal axis.

• A perfect classifier passes through the top-left corner \( (0, 1) \).
• Random guessing gives the diagonal line from \( (0, 0) \) to \( (1, 1) \).

The area under the curve (AUC) summarises it as one number between 0 and 1. It has a lovely probabilistic meaning: AUC is the probability that a randomly chosen positive example receives a higher score than a randomly chosen negative one.

Worked Example 2: Building an ROC Curve

A model scores eight test cases, four positive (P) and four negative (N). Sorted by score: 0.95 P, 0.85 P, 0.70 N, 0.60 P, 0.45 N, 0.30 P, 0.20 N, 0.10 N.

Step 1. Sweep the threshold down through each score. Each time we pass a positive, TPR rises by \( 1/4 \); each time we pass a negative, FPR rises by \( 1/4 \).

Together with the starting point \( (0, 0) \), these points trace a staircase.

Step 2. Area by rectangles. The curve only moves right when FPR increases, so the area is the sum of width times height for each horizontal step:

• FPR from 0 to 0.25 at TPR 0.50: \( 0.25 \times 0.50 = 0.125 \)
• FPR from 0.25 to 0.50 at TPR 0.75: \( 0.25 \times 0.75 = 0.1875 \)
• FPR from 0.50 to 0.75 at TPR 1.00: \( 0.25 \)
• FPR from 0.75 to 1.00 at TPR 1.00: \( 0.25 \)

AUC \( = 0.125 + 0.1875 + 0.25 + 0.25 = 0.8125 \).

Step 3. Check with the probability interpretation. There are \( 4 \times 4 = 16 \) positive–negative pairs. Count pairs where the positive outscores the negative: the positives at 0.95 and 0.85 beat all 4 negatives (8 pairs), the positive at 0.60 beats 3 negatives, and the positive at 0.30 beats 2. Total \( 8 + 3 + 2 = 13 \), and \( 13/16 = 0.8125 \). The two methods agree.

Choosing the Right Metric

• Balanced classes, equal costs: accuracy is fine.
• Rare positive class: report precision, recall and F1; also consider the precision–recall curve, which is more sensitive than ROC when positives are scarce.
• Ranking quality across thresholds: ROC AUC.
• Multi-class problems: compute metrics per class, then take the macro average (each class counts equally) or the micro average (each prediction counts equally).

In Python, scikit-learn provides precision_score, recall_score, f1_score and roc_auc_score; R users often use the pROC or yardstick packages.

Did You Know?

• ROC curves come from signal detection work on radar during the Second World War, where operators had to tell enemy aircraft from noise.
• An AUC below 0.5 means the model is worse than random, but simply flipping its predictions would give an AUC above 0.5.
• The F1 score is the same as the Dice coefficient, widely used to evaluate medical image segmentation.

Key Takeaways

• The confusion matrix (TP, FP, FN, TN) underlies every classification metric.
• Precision measures how trustworthy positive predictions are; recall measures how many positives are found.
• F1 is the harmonic mean of precision and recall and punishes imbalance between them.
• ROC curves plot TPR against FPR across all thresholds, and AUC equals the probability that a positive outscores a negative.
• On imbalanced data, accuracy alone is misleading; always report precision and recall.

Need help evaluating a classifier for your project or thesis? Amathify's experts and PhD scholars provide step-by-step solutions and assignment help via WhatsApp or the Ask a Question form.