Singular Value Decomposition and Its Applications

If you could learn only one matrix factorisation, it should be the singular value decomposition (SVD). It works for every matrix, square or rectangular, and it reveals the hidden structure inside data: which directions matter, which are noise, and how to compress information with the least possible error. Image compression, recommender systems, search engines and PCA all rely on it.

What the SVD Says

Any real \(m \times n\) matrix \(A\) can be written as

$$ A = U\Sigma V^{T} $$

where:

• \(U\) is an \(m \times m\) orthogonal matrix whose columns \(\mathbf{u}_i\) are the left singular vectors,
• \(\Sigma\) is an \(m \times n\) diagonal matrix holding the singular values \(\sigma_1 \ge \sigma_2 \ge \dots \ge 0\),
• \(V\) is an \(n \times n\) orthogonal matrix whose columns \(\mathbf{v}_i\) are the right singular vectors.

Equivalently, \(A\) is a sum of simple rank-one pieces, ordered by importance:

$$ A = \sigma_1\mathbf{u}_1\mathbf{v}_1^{T} + \sigma_2\mathbf{u}_2\mathbf{v}_2^{T} + \dots + \sigma_r\mathbf{u}_r\mathbf{v}_r^{T} $$

where \(r\) is the rank of \(A\).

The Geometry: Rotate, Stretch, Rotate

Think of \(A\) as a transformation acting on vectors. The SVD says every linear transformation does three things in sequence:

1. \(V^{T}\) rotates (or reflects) the input space.
2. \(\Sigma\) stretches along the coordinate axes by the factors \(\sigma_i\).
3. \(U\) rotates the result into the output space.

A unit circle is therefore always mapped to an ellipse (or its higher-dimensional cousin). The singular values are the lengths of the ellipse's semi-axes, the \(\mathbf{u}_i\) point along those axes, and the \(\mathbf{v}_i\) are the input directions that land on them. In particular, \(\sigma_1\) is the greatest factor by which \(A\) can stretch any vector.

How to Compute the SVD by Hand

The connection to eigenvalues provides the recipe. Since \(A^{T}A = V\Sigma^{T}\Sigma V^{T}\):

1. Compute \(A^{T}A\), a symmetric matrix.
2. Its eigenvalues are \(\sigma_i^2\); its unit eigenvectors are the \(\mathbf{v}_i\).
3. Take \(\sigma_i = \sqrt{\lambda_i}\).
4. Find each left singular vector from \(\mathbf{u}_i = A\mathbf{v}_i / \sigma_i\).

Worked Example 1: SVD of a 2×2 Matrix

Find the SVD of

$$ A = \begin{pmatrix} 3 & 0 \\ 4 & 5 \end{pmatrix} $$

Step 1: Form \(A^{T}A\).

$$ A^{T}A = \begin{pmatrix} 3 & 4 \\ 0 & 5 \end{pmatrix}\begin{pmatrix} 3 & 0 \\ 4 & 5 \end{pmatrix} = \begin{pmatrix} 25 & 20 \\ 20 & 25 \end{pmatrix} $$

Step 2: Eigenvalues.

$$ (25 - \lambda)^2 - 400 = 0 \quad\Longrightarrow\quad 25 - \lambda = \pm 20 \quad\Longrightarrow\quad \lambda_1 = 45,\; \lambda_2 = 5 $$

Step 3: Singular values.

$$ \sigma_1 = \sqrt{45} = 3\sqrt{5} \approx 6.708, \qquad \sigma_2 = \sqrt{5} \approx 2.236 $$

Check: \(\sigma_1\sigma_2 = 15\), which equals \(|\det A| = |15 - 0| = 15\).

Step 4: Right singular vectors. For \(\lambda = 45\): \(-20v_1 + 20v_2 = 0\), so \(\mathbf{v}_1 = \tfrac{1}{\sqrt{2}}(1, 1)\). For \(\lambda = 5\): \(20v_1 + 20v_2 = 0\), so \(\mathbf{v}_2 = \tfrac{1}{\sqrt{2}}(1, -1)\).

Step 5: Left singular vectors.

$$ A\mathbf{v}_1 = \frac{1}{\sqrt{2}}\begin{pmatrix} 3 \\ 9 \end{pmatrix}, \qquad \mathbf{u}_1 = \frac{A\mathbf{v}_1}{3\sqrt{5}} = \frac{1}{\sqrt{10}}\begin{pmatrix} 1 \\ 3 \end{pmatrix} $$

$$ A\mathbf{v}_2 = \frac{1}{\sqrt{2}}\begin{pmatrix} 3 \\ -1 \end{pmatrix}, \qquad \mathbf{u}_2 = \frac{A\mathbf{v}_2}{\sqrt{5}} = \frac{1}{\sqrt{10}}\begin{pmatrix} 3 \\ -1 \end{pmatrix} $$

Both \(\mathbf{u}\) vectors have length 1 and their dot product is \(3 - 3 = 0\), so \(U\) is orthogonal, as required.

Step 6: Assemble.

$$ A = \underbrace{\frac{1}{\sqrt{10}}\begin{pmatrix} 1 & 3 \\ 3 & -1 \end{pmatrix}}_{U}\underbrace{\begin{pmatrix} 3\sqrt{5} & 0 \\ 0 & \sqrt{5} \end{pmatrix}}_{\Sigma}\underbrace{\frac{1}{\sqrt{2}}\begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix}}_{V^{T}} $$

Geometrically, \(A\) stretches the direction \((1,1)\) by a factor of about 6.7 and the perpendicular direction \((1,-1)\) by about 2.2.

Low-Rank Approximation

The most useful property of the SVD is the Eckart–Young theorem: keeping only the first \(k\) terms,

$$ A_k = \sum_{i=1}^{k}\sigma_i\mathbf{u}_i\mathbf{v}_i^{T} $$

gives the best possible rank-\(k\) approximation to \(A\), and the error in the Frobenius norm is

$$ \lVert A - A_k\rVert_F = \sqrt{\sigma_{k+1}^2 + \dots + \sigma_r^2} $$

Worked Example 2: Compressing a Matrix

Part (a): Best rank-1 approximation of \(A\) above.

$$ A_1 = \sigma_1\mathbf{u}_1\mathbf{v}_1^{T} = 3\sqrt{5}\cdot\frac{1}{\sqrt{10}}\begin{pmatrix} 1 \\ 3 \end{pmatrix}\cdot\frac{1}{\sqrt{2}}\begin{pmatrix} 1 & 1 \end{pmatrix} $$

The constant is \(3\sqrt{5}/\sqrt{20} = 3/2\), so

$$ A_1 = \frac{3}{2}\begin{pmatrix} 1 & 1 \\ 3 & 3 \end{pmatrix} = \begin{pmatrix} 1.5 & 1.5 \\ 4.5 & 4.5 \end{pmatrix} $$

The error matrix is

$$ A - A_1 = \begin{pmatrix} 1.5 & -1.5 \\ -0.5 & 0.5 \end{pmatrix} $$

Its squared Frobenius norm is \(2.25 + 2.25 + 0.25 + 0.25 = 5 = \sigma_2^2\), exactly as Eckart–Young predicts. The fraction of the matrix's "energy" kept is \(\sigma_1^2/(\sigma_1^2 + \sigma_2^2) = 45/50 = 90\%\).

Part (b): Image storage. A greyscale image is a \(1000 \times 800\) matrix of pixel intensities, requiring \(800{,}000\) numbers. A rank-50 approximation stores 50 singular values, 50 left vectors of length 1000 and 50 right vectors of length 800:

$$ 50 \times (1000 + 800 + 1) = 90{,}050 \text{ numbers} $$

That is about \(90{,}050/800{,}000 \approx 11.3\%\) of the original storage. For natural photographs the singular values decay quickly, so such an approximation often looks very close to the original.

Applications in Machine Learning and Beyond

Principal Component Analysis

If \(X\) is a centred data matrix with SVD \(X = U\Sigma V^{T}\), then the columns of \(V\) are exactly the principal components, and the variance along component \(i\) is \(\sigma_i^2/(n-1)\). Libraries compute PCA this way because it avoids forming \(X^{T}X\), which can lose numerical precision.

Recommender Systems

A matrix of users by movies, filled with ratings, tends to be approximately low-rank: a few hidden factors (taste for action, romance, comedy) explain most preferences. Truncated SVD and its variants uncover these latent factors, an approach made famous during the Netflix Prize competition of 2006–2009.

Noise Reduction

Measured data, such as sensor readings or survey responses, is usually a low-rank signal plus random noise. The signal concentrates in a few large singular values, while noise spreads thinly across many small ones. Plotting the singular values in decreasing order typically shows a sharp drop followed by a long flat tail; truncating at the drop keeps the structure and discards much of the noise. The same reasoning is used to clean up images and to denoise gene-expression data in bioinformatics.

Latent Semantic Analysis

Applying SVD to a term–document matrix groups words that appear in similar contexts, so a search for "car" can also find documents about "automobiles".

Solving Least Squares and the Pseudo-inverse

The Moore–Penrose pseudo-inverse is \(A^{+} = V\Sigma^{+}U^{T}\), where \(\Sigma^{+}\) inverts each non-zero singular value. It gives the least-squares solution \(\mathbf{x} = A^{+}\mathbf{b}\) even when \(A\) is rectangular or singular. The ratio \(\sigma_1/\sigma_r\), called the condition number, warns when a system is numerically unstable.

Did You Know?

• The SVD was discovered independently by Eugenio Beltrami (1873) and Camille Jordan (1874); the efficient algorithms used today grew from work by Gene Golub, William Kahan and Christian Reinsch around 1965–1970.
• The largest singular value \(\sigma_1\) equals the matrix 2-norm, the maximum stretching factor of \(A\).
• For a symmetric positive semi-definite matrix, the SVD and the eigendecomposition are the same thing.

Key Takeaways

• Every matrix factorises as \(A = U\Sigma V^{T}\): a rotation, a stretch and another rotation.
• Singular values are square roots of the eigenvalues of \(A^{T}A\).
• Truncating the SVD gives the best low-rank approximation, with error set by the discarded singular values.
• SVD powers PCA, compression, recommender systems, search and stable least-squares solutions.

Need help computing an SVD, proving a linear algebra result or completing a data science assignment? Amathify's experts and PhD scholars provide step-by-step solutions; contact us on WhatsApp or use the Ask a Question form.