Mathematics in Computer Vision

When your phone unlocks by recognising your face, or a car on the motorway reads a speed-limit sign, a computer is doing something remarkable: turning a grid of numbers into meaning. Computer vision is the science of that transformation, and underneath every impressive demo sits a surprisingly clean set of mathematical ideas. In this article we walk through the core maths, from matrices and convolution to camera geometry and the metrics researchers use to judge their models.

An Image Is Just a Matrix

A greyscale image of width \( W \) and height \( H \) is stored as a matrix \( I \) with \( H \) rows and \( W \) columns. Each entry \( I(x, y) \) is a brightness value, usually an integer from 0 (black) to 255 (white). A colour image is three such matrices stacked together, one each for red, green and blue, which mathematicians call a tensor of shape \( H \times W \times 3 \).

This simple fact means that almost every operation in vision is a piece of linear algebra or calculus applied to arrays:

• Brightening an image adds a constant to every entry.
• Rotating or scaling it applies a geometric transformation matrix to the pixel coordinates.
• Blurring, sharpening and edge detection are convolutions.
• Deep learning models are long chains of matrix multiplications and simple non-linear functions.

Convolution and Edge Detection

Edges are where brightness changes quickly, so finding them means estimating a derivative. Because the image is discrete, we approximate the partial derivative \( \partial I / \partial x \) with a small grid of weights called a kernel. The famous Sobel kernels are

$$ G_x = \begin{pmatrix} -1 & 0 & 1 \\ -2 & 0 & 2 \\ -1 & 0 & 1 \end{pmatrix}, \qquad G_y = \begin{pmatrix} -1 & -2 & -1 \\ 0 & 0 & 0 \\ 1 & 2 & 1 \end{pmatrix}. $$

To apply a kernel at a pixel, place it over the \( 3 \times 3 \) neighbourhood, multiply matching entries and add them up. (Strictly, this is cross-correlation; true convolution flips the kernel first, but deep learning libraries use the unflipped version and still call it convolution.)

Worked Example 1: Detecting a Vertical Edge

Suppose a \( 3 \times 3 \) patch of an image has a dark region on the left and a bright stripe on the right:

$$ P = \begin{pmatrix} 10 & 10 & 80 \\ 10 & 10 & 80 \\ 10 & 10 & 80 \end{pmatrix}. $$

Step 1: Horizontal gradient. Multiply \( P \) entry by entry with \( G_x \) and sum. The middle column is multiplied by 0, so only the outer columns matter:

• Top row: \( (-1)(10) + (1)(80) = 70 \)
• Middle row: \( (-2)(10) + (2)(80) = 140 \)
• Bottom row: \( (-1)(10) + (1)(80) = 70 \)

So \( g_x = 70 + 140 + 70 = 280 \).

Step 2: Vertical gradient. With \( G_y \), the top row gives \( -(10) - 2(10) - 80 = -110 \), the middle row gives 0, and the bottom row gives \( 10 + 2(10) + 80 = 110 \). So \( g_y = 0 \).

Step 3: Magnitude and direction.

$$ |\nabla I| = \sqrt{g_x^2 + g_y^2} = \sqrt{280^2 + 0^2} = 280, \qquad \theta = \tan^{-1}\!\left(\frac{g_y}{g_x}\right) = 0^\circ. $$

A large magnitude tells us there is a strong edge, and a gradient direction of \( 0^\circ \) (pointing left to right) tells us the edge itself runs vertically. That is exactly what we see in the patch.

From Hand-Made Kernels to Learned Kernels

A convolutional neural network (CNN) uses the same operation, but instead of a human choosing the weights, the network learns thousands of kernels from data by gradient descent. Early layers usually learn edge and colour detectors that look strikingly like Sobel filters, while deeper layers combine them into detectors for textures, eyes, wheels and letters. Vision transformers, a newer family of models, replace convolution with attention, which is again built from matrix products and the softmax function.

Camera Geometry: From 3D World to 2D Pixels

To measure distances, build 3D maps or guide a robot arm, we need to know how a point in the world lands on the image. The simplest model is the pinhole camera. A point with camera coordinates \( (X, Y, Z) \), where \( Z \) is the distance in front of the lens, projects to pixel coordinates

$$ u = f\,\frac{X}{Z} + c_x, \qquad v = f\,\frac{Y}{Z} + c_y, $$

where \( f \) is the focal length measured in pixels and \( (c_x, c_y) \) is the principal point, roughly the image centre. In homogeneous coordinates this becomes a single matrix equation with the camera intrinsic matrix \( K \):

$$ \begin{pmatrix} u \\ v \\ 1 \end{pmatrix} \sim \begin{pmatrix} f & 0 & c_x \\ 0 & f & c_y \\ 0 & 0 & 1 \end{pmatrix} \begin{pmatrix} X \\ Y \\ Z \end{pmatrix}. $$

The symbol \( \sim \) means "equal up to a scale factor": we divide by the third component at the end.

Worked Example 2: Projecting a Point

A camera has \( f = 800 \) pixels and principal point \( (320, 240) \) on a \( 640 \times 480 \) image. A traffic cone is at \( (X, Y, Z) = (0.5, -0.2, 4) \) metres.

Step 1: Multiply by \( K \).

$$ K\begin{pmatrix} 0.5 \\ -0.2 \\ 4 \end{pmatrix} = \begin{pmatrix} 800(0.5) + 320(4) \\ 800(-0.2) + 240(4) \\ 4 \end{pmatrix} = \begin{pmatrix} 1680 \\ 800 \\ 4 \end{pmatrix}. $$

Step 2: Divide by the third component. \( u = 1680/4 = 420 \) and \( v = 800/4 = 200 \).

Step 3: Check with the direct formula. \( u = 800 \times 0.5/4 + 320 = 100 + 320 = 420 \) and \( v = 800 \times (-0.2)/4 + 240 = -40 + 240 = 200 \). The cone appears at pixel \( (420, 200) \), right of centre and slightly above it (image \( v \) grows downward), which matches its position to the right and above the camera axis.

Notice that if the cone moved to \( Z = 8 \) m, its offset from the centre would halve. This division by depth is why distant objects look smaller and why a single image cannot tell you depth on its own. Stereo vision fixes this with two cameras: depth is recovered from the disparity \( d \) between the two images using \( Z = fB/d \), where \( B \) is the distance between the cameras.

Measuring Success: Intersection over Union

Object detectors draw bounding boxes around cars, people or tumours. To score a predicted box \( A \) against the true box \( B \), researchers use Intersection over Union:

$$ \text{IoU} = \frac{\text{area}(A \cap B)}{\text{area}(A \cup B)}. $$

Worked Example 3: A Quick IoU Calculation

Let box \( A \) run from \( (0, 0) \) to \( (4, 4) \) and box \( B \) from \( (2, 2) \) to \( (6, 6) \).

1. Each box has area \( 4 \times 4 = 16 \).
2. The overlap runs from \( (2, 2) \) to \( (4, 4) \), so its area is \( 2 \times 2 = 4 \).
3. The union is \( 16 + 16 - 4 = 28 \).
4. Therefore \( \text{IoU} = 4/28 = 1/7 \approx 0.143 \).

Many benchmarks only count a detection as correct when IoU is at least 0.5, so this prediction would be rejected despite clearly overlapping the target.

Other Mathematical Tools in Vision

• Optimisation: training networks minimises a loss function with stochastic gradient descent and its variants.
• Probability: classifiers output probabilities via softmax, and Bayesian methods quantify uncertainty in medical imaging.
• Linear algebra: the singular value decomposition underpins image compression, camera calibration and estimating the fundamental matrix between two views.
• Geometry and topology: 3D reconstruction, point clouds and neural radiance fields rely on projective geometry and ray integrals.
• Fourier analysis: the convolution theorem says convolution in space is multiplication in frequency, which speeds up large filters.

Did You Know?

• The Sobel operator was presented by Irwin Sobel and Gary Feldman around 1968 at the Stanford Artificial Intelligence Laboratory, long before deep learning existed.
• In 2012, a CNN called AlexNet dramatically cut the error rate in the ImageNet competition, sparking the modern deep learning boom in vision.
• A single \( 1920 \times 1080 \) colour image contains more than six million numbers, which is why efficient linear algebra on GPUs is so important.

Key Takeaways

• Digital images are matrices (or tensors), so vision is built on linear algebra and calculus.
• Convolution with small kernels estimates derivatives and detects edges; CNNs learn these kernels from data.
• The pinhole model \( u = fX/Z + c_x \) explains perspective and why depth needs stereo or extra cues.
• Metrics such as IoU turn "how good is this box?" into a precise number between 0 and 1.
• Research frontiers combine optimisation, probability and geometry in models such as vision transformers and 3D reconstruction.

If you are working on a computer vision project, thesis or assignment and want the mathematics explained step by step, Amathify's subject experts and PhD scholars are ready to help via WhatsApp or the Ask a Question form.