Principal Component Analysis (PCA) Step by Step

Real datasets often have dozens or hundreds of correlated variables, which makes them hard to visualise and slow to model. Principal Component Analysis (PCA) finds a small number of new directions that capture most of the variation in the data, letting you compress, plot and denoise it with minimal loss of information. Underneath, PCA is a beautiful piece of linear algebra built entirely on covariance matrices and eigenvectors.

The Big Idea

Imagine a cloud of points stretched out like a tilted ellipse. Most of the spread lies along the long axis of the ellipse; very little lies across it. If we rotated our coordinate axes to line up with the ellipse, we could describe each point mostly by one number (its position along the long axis) and throw away the second number with little loss.

PCA does exactly this. It finds:

• the first principal component (PC1): the direction along which the data has maximum variance,
• the second principal component (PC2): the direction of maximum remaining variance, perpendicular to PC1,
• and so on, for as many dimensions as the data has.

Why Eigenvectors?

Let the centred data matrix be \(X\) (\(n\) rows, \(p\) columns) and the sample covariance matrix be

$$ C = \frac{1}{n-1}X^{T}X $$

If we project the data onto a unit vector \(\mathbf{v}\), the variance of the projected values is \(\mathbf{v}^{T}C\,\mathbf{v}\). We want to maximise this subject to \(\mathbf{v}^{T}\mathbf{v} = 1\). Using a Lagrange multiplier \(\lambda\) and setting the derivative to zero gives

$$ C\mathbf{v} = \lambda\mathbf{v} $$

So the best directions are eigenvectors of the covariance matrix, and the variance along each one equals its eigenvalue \(\lambda\), since \(\mathbf{v}^{T}C\mathbf{v} = \lambda\mathbf{v}^{T}\mathbf{v} = \lambda\). Because \(C\) is symmetric, its eigenvectors are automatically perpendicular, which is why principal components are orthogonal.

The PCA Algorithm

1. Centre the data: subtract each variable's mean.
2. Standardise if needed: divide by standard deviations when variables have different units.
3. Compute the covariance matrix \(C\).
4. Find eigenvalues and eigenvectors of \(C\) and sort them from largest to smallest eigenvalue.
5. Choose \(k\) components that explain enough variance.
6. Project: the scores are \(Z = XV_k\), where \(V_k\) holds the top \(k\) eigenvectors as columns.

Worked Example 1: PCA on Five Points

Five students have scores on two short tests, \(x_1\) and \(x_2\): (2, 1), (3, 4), (4, 3), (5, 6), (6, 6).

Step 1: Centre the data. The means are \(\bar{x}_1 = 20/5 = 4\) and \(\bar{x}_2 = 20/5 = 4\). Subtracting gives the centred points

$$ (-2,-3),\; (-1, 0),\; (0,-1),\; (1, 2),\; (2, 2) $$

Step 2: Covariance matrix (dividing by \(n - 1 = 4\)).

$$ \text{Var}(x_1) = \frac{4 + 1 + 0 + 1 + 4}{4} = 2.5, \qquad \text{Var}(x_2) = \frac{9 + 0 + 1 + 4 + 4}{4} = 4.5 $$

$$ \text{Cov}(x_1, x_2) = \frac{(-2)(-3) + (-1)(0) + (0)(-1) + (1)(2) + (2)(2)}{4} = \frac{12}{4} = 3 $$

$$ C = \begin{pmatrix} 2.5 & 3 \\ 3 & 4.5 \end{pmatrix} $$

Step 3: Eigenvalues. Solve \(\det(C - \lambda I) = 0\):

$$ (2.5 - \lambda)(4.5 - \lambda) - 9 = 0 \quad\Longrightarrow\quad \lambda^2 - 7\lambda + 2.25 = 0 $$

$$ \lambda = \frac{7 \pm \sqrt{49 - 9}}{2} = \frac{7 \pm \sqrt{40}}{2} = 3.5 \pm 3.1623 $$

So \(\lambda_1 \approx 6.6623\) and \(\lambda_2 \approx 0.3377\). As a check, they add to the trace \(2.5 + 4.5 = 7\) and multiply to the determinant \(11.25 - 9 = 2.25\).

Step 4: Eigenvector for \(\lambda_1\). From the first row of \((C - \lambda_1 I)\mathbf{v} = 0\):

$$ (2.5 - 6.6623)v_1 + 3v_2 = 0 \quad\Longrightarrow\quad v_2 = 1.3874\,v_1 $$

Normalising \((1, 1.3874)\) by its length \(\sqrt{1 + 1.9249} \approx 1.7103\) gives

$$ \mathbf{v}_1 \approx (0.5847,\; 0.8112) $$

PC2 is perpendicular: \(\mathbf{v}_2 \approx (-0.8112,\; 0.5847)\).

Step 5: Explained variance.

$$ \frac{\lambda_1}{\lambda_1 + \lambda_2} = \frac{6.6623}{7} \approx 0.952 $$

PC1 alone captures about 95.2% of the total variance. We can reduce the data from two dimensions to one and keep almost everything.

Step 6: Project onto PC1. Each score is the dot product of a centred point with \(\mathbf{v}_1\). For the first student:

$$ z = (-2)(0.5847) + (-3)(0.8112) = -1.1694 - 2.4337 \approx -3.603 $$

The full set of PC1 scores is −3.603, −0.585, −0.811, 2.207 and 2.792. PC1 here is roughly an "overall ability" score, weighting both tests positively.

Worked Example 2: Reconstruction Error

A different centred dataset has covariance matrix

$$ C = \begin{pmatrix} 5 & 2 \\ 2 & 2 \end{pmatrix} $$

Find the principal components, the variance explained by PC1, and the reconstruction of the centred point \((1, 3)\) from PC1 alone.

Step 1: Eigenvalues. \(\lambda^2 - 7\lambda + (10 - 4) = 0\), so \(\lambda^2 - 7\lambda + 6 = 0\), giving \((\lambda - 6)(\lambda - 1) = 0\). Thus \(\lambda_1 = 6\) and \(\lambda_2 = 1\).

Step 2: Eigenvectors. For \(\lambda = 6\): \(-v_1 + 2v_2 = 0\), so \(\mathbf{v}_1 = (2, 1)/\sqrt{5}\). For \(\lambda = 1\): \(4v_1 + 2v_2 = 0\), so \(\mathbf{v}_2 = (1, -2)/\sqrt{5}\).

Step 3: Explained variance. \(6/7 \approx 85.7\%\).

Step 4: Project and reconstruct.

$$ z_1 = \frac{(1)(2) + (3)(1)}{\sqrt{5}} = \frac{5}{\sqrt{5}} = \sqrt{5} $$

$$ \hat{\mathbf{x}} = z_1\mathbf{v}_1 = \sqrt{5}\cdot\frac{(2, 1)}{\sqrt{5}} = (2, 1) $$

Step 5: Error. The difference is \((1, 3) - (2, 1) = (-1, 2)\), with squared length 5. This equals \(z_2^2\), where \(z_2 = (1 - 6)/\sqrt{5} = -\sqrt{5}\). The information lost is exactly the part of the point lying along the discarded component.

Interpreting Loadings

The entries of each eigenvector are called loadings, and they tell you how much each original variable contributes to a component. In Example 1, both loadings of PC1 are positive (0.585 and 0.811), so a high PC1 score means a student did well on both tests; the second test carries slightly more weight because it varies more. PC2 has loadings of opposite sign, so it contrasts the two tests, separating students who did relatively better on one than the other. In a thesis, researchers often name components after the variables with the largest absolute loadings.

Practical Tips

• Always centre; otherwise PC1 may simply point towards the mean.
• Standardise when variables use different units (for example rupees and kilograms), which is equivalent to running PCA on the correlation matrix.
• Use a scree plot of eigenvalues, or keep enough components for 80–95% of the variance.
• Principal components are combinations of all variables, so they can be harder to interpret than the originals.
• In software, PCA is available as prcomp() in R, PCA in Python's scikit-learn, and through the factor analysis menus of SPSS. These usually compute it via the singular value decomposition, which is more numerically stable.

Did You Know?

• Karl Pearson described the idea of PCA in 1901 as finding "lines and planes of closest fit"; Harold Hotelling developed and named it in 1933.
• The "eigenfaces" method of the early 1990s recognised faces by applying PCA to thousands of face images.
• Maximising variance along a direction is equivalent to minimising the squared perpendicular distances of points to that line.

Key Takeaways

• PCA finds orthogonal directions of maximum variance: the eigenvectors of the covariance matrix.
• Each eigenvalue equals the variance captured by its component; their sum equals the total variance.
• Projecting onto the top \(k\) components reduces dimensions while keeping most of the information.
• The reconstruction error equals the variance along the components you discard.

Working on a PCA assignment or using it in your thesis? Amathify's experts and PhD scholars offer step-by-step solutions and analysis support; reach us on WhatsApp or via the Ask a Question form.