Convolutional Neural Networks: The Mathematics of Images

To a computer, a photograph is nothing more than a grid of numbers. Convolutional neural networks (CNNs) are the models that turned those grids into face recognition, medical image diagnosis and self-driving perception. Their power comes from a few simple mathematical operations repeated many times, and once you can compute them by hand the whole architecture becomes much less mysterious.

Images as Tensors

A greyscale image of height \( H \) and width \( W \) is a matrix \( I \in \mathbb{R}^{H \times W} \), where each entry is a brightness, often scaled from 0 (black) to 1 or 255 (white). A colour image has three channels (red, green, blue), so it is a tensor of shape \( H \times W \times 3 \). A modest \( 224 \times 224 \) colour image therefore contains \( 224 \times 224 \times 3 = 150{,}528 \) numbers.

If we fed this into an ordinary fully connected layer with just 1000 neurons, we would need over 150 million weights in that single layer. Worse, such a layer ignores the fact that nearby pixels are related and that a cat in the top-left corner is still a cat in the bottom-right. CNNs fix both problems.

The Convolution Operation

A CNN layer slides a small grid of weights, called a kernel or filter, across the image. At each position it multiplies overlapping entries and adds them up. For a kernel \( K \) of size \( k \times k \), the output (also called a feature map) is

$$ S(i, j) = \sum_{m=0}^{k-1} \sum_{n=0}^{k-1} I(i + m,\, j + n)\, K(m, n) + b . $$

Strictly speaking this is cross-correlation; true mathematical convolution flips the kernel first. Deep learning libraries use the unflipped version but still call it convolution, and since the kernel weights are learned, the distinction does not matter in practice.

Two ideas make convolution efficient:

• Local connectivity: each output depends only on a small patch of input.
• Weight sharing: the same kernel is used at every position, so a feature detector learned in one place works everywhere.

Worked Example 1: Detecting a Vertical Edge

Consider a \( 4 \times 4 \) greyscale image with a bright region on the left and a dark column on the right, and a vertical-edge kernel:

$$ I = \begin{pmatrix} 10 & 10 & 10 & 0 \\ 10 & 10 & 10 & 0 \\ 10 & 10 & 10 & 0 \\ 10 & 10 & 10 & 0 \end{pmatrix}, \qquad K = \begin{pmatrix} 1 & 0 & -1 \\ 1 & 0 & -1 \\ 1 & 0 & -1 \end{pmatrix}. $$

With no padding and stride 1, a \( 3 \times 3 \) kernel fits in \( 2 \times 2 \) positions. Take bias \( b = 0 \).

Position (0, 0) covers columns 1 to 3 of rows 1 to 3. Each row contributes \( 10(1) + 10(0) + 10(-1) = 0 \). Total \( S(0,0) = 0 \).

Position (0, 1) covers columns 2 to 4. Each row contributes \( 10(1) + 10(0) + 0(-1) = 10 \). Total over three rows \( S(0,1) = 30 \).

Positions (1, 0) and (1, 1) cover rows 2 to 4. The rows are identical, so \( S(1,0) = 0 \) and \( S(1,1) = 30 \).

$$ S = \begin{pmatrix} 0 & 30 \\ 0 & 30 \end{pmatrix}. $$

The output is zero over the flat bright area and large exactly where brightness drops from left to right. The kernel has acted as an edge detector. In a trained CNN, nobody designs such kernels by hand; gradient descent discovers edge detectors, texture detectors and much more on its own.

Padding, Stride and Output Size

Two settings control the size of the output:

• Padding \( p \): adding \( p \) rings of zeros around the input so the kernel can be centred on border pixels.
• Stride \( s \): how many pixels the kernel jumps between positions.

For an input of width \( W \), kernel width \( k \), padding \( p \) and stride \( s \), the output width is

$$ W_{\text{out}} = \left\lfloor \frac{W - k + 2p}{s} \right\rfloor + 1 . $$

Check with Example 1: \( W = 4, k = 3, p = 0, s = 1 \) gives \( (4 - 3 + 0)/1 + 1 = 2 \), matching the \( 2 \times 2 \) result.

Multiple Channels and Multiple Filters

When the input has \( C_{\text{in}} \) channels, each filter has shape \( k \times k \times C_{\text{in}} \) and sums over all channels, producing one feature map. A layer with \( C_{\text{out}} \) filters produces \( C_{\text{out}} \) feature maps stacked into a new tensor. The number of learnable parameters is

$$ \text{Parameters} = C_{\text{out}} \times \left(k \times k \times C_{\text{in}} + 1\right), $$

where the \( +1 \) is each filter's bias.

Worked Example 2: Sizing a Convolutional Layer

An input image has shape \( 32 \times 32 \times 3 \). A convolutional layer uses 16 filters of size \( 5 \times 5 \), padding 2 and stride 1. Find the output shape and the parameter count, then compare with a fully connected layer.

Step 1. Output width. \( W_{\text{out}} = (32 - 5 + 2 \times 2)/1 + 1 = 31 + 1 = 32 \). The height is the same, so the spatial size is preserved.

Step 2. Output shape. One feature map per filter: \( 32 \times 32 \times 16 \).

Step 3. Parameters. Each filter has \( 5 \times 5 \times 3 = 75 \) weights plus 1 bias, giving 76. With 16 filters: \( 16 \times 76 = 1216 \) parameters.

Step 4. Fully connected comparison. The input has \( 32 \times 32 \times 3 = 3072 \) values and the output has \( 32 \times 32 \times 16 = 16{,}384 \) values. A dense layer connecting them needs \( 3072 \times 16{,}384 = 50{,}331{,}648 \) weights, plus 16,384 biases.

The convolutional layer uses roughly 41,000 times fewer parameters while still producing an output of the same size. That is the payoff of local connectivity and weight sharing.

Non-Linearity and Pooling

After convolution, each value passes through an activation function, most commonly ReLU, \( \text{ReLU}(z) = \max(0, z) \). Without it, stacking layers would collapse into one big linear operation.

Pooling then reduces the spatial size. In \( 2 \times 2 \) max pooling with stride 2, each non-overlapping block is replaced by its largest value. For example, the block

$$ \begin{pmatrix} 1 & 5 \\ 3 & 2 \end{pmatrix} \;\longrightarrow\; 5 . $$

This halves the width and height, cuts computation by a factor of four in later layers and makes the network tolerant to small shifts of the object.

Building Up a Hierarchy

A typical CNN repeats the pattern convolution, ReLU, pooling several times and ends with one or more fully connected layers and a softmax that outputs class probabilities. Each layer sees a larger region of the original image, its receptive field. Two stacked \( 3 \times 3 \) convolutions have a \( 5 \times 5 \) receptive field, three have \( 7 \times 7 \). Early layers respond to edges and colours, middle layers to textures and parts like eyes or wheels, and deep layers to whole objects.

Training uses backpropagation. A neat fact is that the gradient of the loss with respect to a kernel is itself a convolution, of the input with the gradient of the output, so the same efficient operation powers both the forward and backward passes.

Did You Know?

• Yann LeCun and colleagues used CNNs around 1989 to read handwritten postcode digits, and a later version, LeNet-5, was used to read cheques in the 1990s.
• In 2012, AlexNet, a deep CNN trained on graphics cards, sharply cut the error rate in the ImageNet competition and sparked the modern deep learning boom.
• A \( 1 \times 1 \) convolution sounds useless, but it mixes information across channels and is widely used to shrink or expand the channel count cheaply.

Key Takeaways

• Images are tensors of numbers; a convolution slides a small kernel over them, multiplying and summing.
• Weight sharing and local connectivity let CNNs use thousands of parameters where dense layers would need millions.
• Output width follows \( \lfloor (W - k + 2p)/s \rfloor + 1 \), and parameters equal \( C_{\text{out}}(k^2 C_{\text{in}} + 1) \).
• ReLU adds non-linearity, and pooling shrinks feature maps while adding robustness to small shifts.
• Stacked layers build a hierarchy from edges to textures to whole objects.

Working on a deep learning assignment or a CNN calculation? Amathify's experts and PhD scholars can walk you through it step by step; message us on WhatsApp or use the Ask a Question form.