Backpropagation and the Chain Rule

Every time a neural network learns to recognise a face or translate a sentence, it relies on one idea from first-year calculus: the chain rule. Backpropagation is simply an efficient, organised way of applying the chain rule to compute how much each weight contributed to the error. Once you see it worked through with real numbers, the mystery disappears.

The Problem Backpropagation Solves

A neural network is a long chain of functions: each layer multiplies its inputs by weights, adds a bias and applies an activation function, then passes the result to the next layer. At the end, a loss function \(L\) measures how wrong the output is. To improve the network by gradient descent, we need the partial derivative of \(L\) with respect to every weight \(w\):

$$ w \leftarrow w - \alpha \frac{\partial L}{\partial w} $$

A modern network can have billions of weights. Estimating each derivative separately by nudging one weight at a time would require billions of forward passes per step. Backpropagation computes all the derivatives in roughly the cost of one forward pass plus one backward pass.

The Chain Rule, Revisited

If \(y = f(u)\) and \(u = g(x)\), then

$$ \frac{dy}{dx} = \frac{dy}{du}\cdot\frac{du}{dx} $$

For longer chains, we just keep multiplying local derivatives:

$$ \frac{dL}{dx} = \frac{dL}{du_3}\cdot\frac{du_3}{du_2}\cdot\frac{du_2}{du_1}\cdot\frac{du_1}{dx} $$

When a variable affects the output through several paths, the multivariable chain rule says we add the contributions from each path. Backpropagation is nothing more than evaluating these products from the output backwards, reusing each intermediate result so nothing is computed twice.

Computational Graphs

It helps to draw a calculation as a graph where each node is a simple operation. Two passes happen on this graph:

1. Forward pass: compute the value at every node, from inputs to output.
2. Backward pass: starting with \(\partial L/\partial L = 1\) at the output, move backwards. At each node, multiply the incoming gradient (the "upstream" gradient) by the node's local derivative and pass the result to its inputs.

Simple rules emerge. An addition node passes the upstream gradient unchanged to both inputs. A multiplication node \(q = ab\) sends \(\text{upstream} \times b\) to \(a\) and \(\text{upstream} \times a\) to \(b\).

Worked Example 1: A Small Computational Graph

Let \(f = (ab + c)^2\) with \(a = 2\), \(b = 3\), \(c = -4\). Find \(\partial f/\partial a\), \(\partial f/\partial b\) and \(\partial f/\partial c\).

Step 1: Forward pass. Introduce intermediate variables:

$$ p = ab = 6, \qquad q = p + c = 2, \qquad f = q^2 = 4 $$

Step 2: Start the backward pass. The local derivative of the last node is

$$ \frac{\partial f}{\partial q} = 2q = 4 $$

Step 3: Through the addition node. Since \(q = p + c\), both local derivatives are 1:

$$ \frac{\partial f}{\partial p} = 4 \times 1 = 4, \qquad \frac{\partial f}{\partial c} = 4 \times 1 = 4 $$

Step 4: Through the multiplication node. Since \(p = ab\), we have \(\partial p/\partial a = b\) and \(\partial p/\partial b = a\):

$$ \frac{\partial f}{\partial a} = 4 \times 3 = 12, \qquad \frac{\partial f}{\partial b} = 4 \times 2 = 8 $$

Step 5: Check directly. Expanding, \(\partial f/\partial a = 2(ab+c)\cdot b = 2(2)(3) = 12\). It matches. The graph method gave every derivative in one sweep.

Backpropagation in a Neural Network

Consider a tiny network with one input, one hidden neuron using a sigmoid activation, and one linear output neuron:

$$ z_1 = w_1 x + b_1, \qquad h = \sigma(z_1), \qquad \hat{y} = w_2 h + b_2, \qquad L = \tfrac{1}{2}(\hat{y} - y)^2 $$

Applying the chain rule backwards gives

$$ \frac{\partial L}{\partial \hat{y}} = \hat{y} - y, \qquad \frac{\partial L}{\partial w_2} = (\hat{y} - y)\,h, \qquad \frac{\partial L}{\partial h} = (\hat{y} - y)\,w_2 $$

$$ \frac{\partial L}{\partial z_1} = \frac{\partial L}{\partial h}\,\sigma(z_1)\bigl(1 - \sigma(z_1)\bigr), \qquad \frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial z_1}\,x $$

The quantity \(\partial L/\partial z\) at each layer is often written \(\delta\) and called the error signal. Notice how \(\partial L/\partial h\) is computed once and reused for both \(w_1\) and \(b_1\). In a deep network this reuse is what makes backpropagation efficient.

Worked Example 2: One Full Training Step

Take \(x = 1\), target \(y = 1\), and initial parameters \(w_1 = 0.5\), \(b_1 = 0\), \(w_2 = 2\), \(b_2 = -1\). Use learning rate \(\alpha = 0.5\).

Step 1: Forward pass.

$$ z_1 = 0.5(1) + 0 = 0.5, \qquad h = \sigma(0.5) = \frac{1}{1 + e^{-0.5}} \approx 0.6225 $$

$$ \hat{y} = 2(0.6225) - 1 = 0.2449, \qquad L = \tfrac{1}{2}(0.2449 - 1)^2 \approx 0.2851 $$

Step 2: Output layer gradients.

$$ \frac{\partial L}{\partial \hat{y}} = 0.2449 - 1 = -0.7551 $$

$$ \frac{\partial L}{\partial w_2} = -0.7551 \times 0.6225 \approx -0.4700, \qquad \frac{\partial L}{\partial b_2} = -0.7551 $$

Step 3: Pass the error back to the hidden neuron.

$$ \frac{\partial L}{\partial h} = -0.7551 \times 2 = -1.5102 $$

Step 4: Through the sigmoid. The local derivative is \(h(1 - h) = 0.6225 \times 0.3775 \approx 0.2350\):

$$ \frac{\partial L}{\partial z_1} = -1.5102 \times 0.2350 \approx -0.3549 $$

Step 5: Hidden layer gradients.

$$ \frac{\partial L}{\partial w_1} = -0.3549 \times 1 = -0.3549, \qquad \frac{\partial L}{\partial b_1} = -0.3549 $$

Step 6: Update all parameters.

• \(w_2 = 2 - 0.5(-0.4700) = 2.2350\)
• \(b_2 = -1 - 0.5(-0.7551) = -0.6225\)
• \(w_1 = 0.5 - 0.5(-0.3549) = 0.6774\)
• \(b_1 = 0 - 0.5(-0.3549) = 0.1774\)

Step 7: Verify improvement. The new forward pass gives \(z_1 = 0.8549\), \(h \approx 0.7016\) and \(\hat{y} = 2.2350(0.7016) - 0.6225 \approx 0.9456\). The loss falls from 0.2851 to about 0.0015. A numerical check, nudging \(w_1\) by \(\pm 10^{-6}\) and measuring the change in \(L\), gives \(-0.3549\), confirming the backpropagated gradient.

Backpropagation in Matrix Form

Real layers contain many neurons, so the same steps are written with vectors and matrices. For layer \(l\) with weight matrix \(W^{(l)}\), pre-activation \(\mathbf{z}^{(l)} = W^{(l)}\mathbf{a}^{(l-1)} + \mathbf{b}^{(l)}\) and activation \(\mathbf{a}^{(l)} = g(\mathbf{z}^{(l)})\), the error signals obey

$$ \boldsymbol{\delta}^{(l)} = \bigl(W^{(l+1)}\bigr)^{T}\boldsymbol{\delta}^{(l+1)} \odot g'\bigl(\mathbf{z}^{(l)}\bigr), \qquad \frac{\partial L}{\partial W^{(l)}} = \boldsymbol{\delta}^{(l)}\bigl(\mathbf{a}^{(l-1)}\bigr)^{T} $$

where \(\odot\) means element-wise multiplication. The transpose of the weight matrix carries the error backwards along the same connections that carried the signal forwards. This is exactly what our single-neuron calculation did, with each matrix shrunk to a single number, and it is why GPUs, which excel at matrix multiplication, train networks so quickly.

Vanishing and Exploding Gradients

Because gradients are products of many local derivatives, their size can change dramatically with depth. The sigmoid's derivative is at most 0.25, so ten sigmoid layers can shrink a gradient by a factor of up to \(0.25^{10} \approx 10^{-6}\). Early layers then barely learn. This vanishing gradient problem is why modern networks prefer the ReLU activation, whose derivative is 1 for positive inputs, along with careful weight initialisation and residual connections. The opposite problem, exploding gradients, is controlled by gradient clipping.

Did You Know?

• The core idea of reverse-mode differentiation was described by Seppo Linnainmaa in his 1970 master's thesis; backpropagation was popularised for neural networks by Rumelhart, Hinton and Williams in 1986.
• Frameworks such as PyTorch and TensorFlow build the computational graph automatically and run backpropagation for you through "automatic differentiation".
• The backward pass typically costs only about two to three times as much as the forward pass, regardless of how many parameters there are.

Key Takeaways

• Backpropagation is the chain rule applied systematically from the loss back to every weight.
• A forward pass stores intermediate values; a backward pass multiplies local derivatives using those values.
• Gradients for each layer reuse the error signal from the layer above, which makes the method efficient.
• Long chains of small derivatives cause vanishing gradients, motivating ReLU and residual connections.

If backpropagation derivations or neural network assignments are giving you trouble, Amathify's experts and PhD scholars can provide step-by-step solutions. Message us on WhatsApp or submit your problem through the Ask a Question form.