The Calculus Behind Gradient Descent

Almost every modern machine learning model, from a simple line of best fit to a large language model, is trained by one surprisingly simple idea: walk downhill. That idea is called gradient descent, and it rests entirely on the derivatives you meet in FSc and A Level calculus. In this article we unpack the calculus behind it and run the algorithm by hand so you can see every number change.

The Problem: Minimising a Loss Function

When a model makes predictions, we measure how wrong it is using a loss function \( L \). The model's adjustable numbers (its parameters or weights) are inputs to \( L \). Training means finding the parameter values that make \( L \) as small as possible.

For a simple function like \( f(x) = x^2 - 4x + 5 \), you could set \( f'(x) = 0 \) and solve. But a neural network may have millions of parameters and a loss function too complicated to solve exactly. Gradient descent instead finds the minimum step by step.

The Derivative Tells You Which Way Is Downhill

Recall that the derivative \( f'(x) \) is the slope of the curve at \( x \).

• If \( f'(x) > 0 \), the function increases as \( x \) increases, so to go down we should decrease \( x \).
• If \( f'(x) < 0 \), we should increase \( x \).

In both cases, moving in the direction of \( -f'(x) \) lowers the function. This gives the update rule:

$$ x_{\text{new}} = x_{\text{old}} - \eta\, f'(x_{\text{old}}) $$

The small positive number \( \eta \) (eta) is the learning rate. It controls the step size.

Worked Example 1: One Variable

Minimise \( f(x) = x^2 - 4x + 5 \) starting from \( x_0 = 0 \) with \( \eta = 0.1 \).

Step 1. Differentiate. \( f'(x) = 2x - 4 \).

Step 2. First update. \( f'(0) = -4 \), so

$$ x_1 = 0 - 0.1(-4) = 0.4 $$

Step 3. Second update. \( f'(0.4) = 0.8 - 4 = -3.2 \), so \( x_2 = 0.4 - 0.1(-3.2) = 0.72 \).

Step 4. Third update. \( f'(0.72) = 1.44 - 4 = -2.56 \), so \( x_3 = 0.72 + 0.256 = 0.976 \).

Step 5. Check against the exact answer. Setting \( 2x - 4 = 0 \) gives the true minimum at \( x = 2 \). The distances from 2 are \( 2,\ 1.6,\ 1.28,\ 1.024 \): each step shrinks the error by a factor of \( 0.8 \). Indeed,

$$ x_{n+1} - 2 = (x_n - 2) - 0.1(2x_n - 4) = 0.8(x_n - 2) $$

so the algorithm steadily closes in on \( x = 2 \).

Many Variables: Partial Derivatives and the Gradient

Real models depend on many parameters. For a function \( f(x, y) \), the partial derivative \( \frac{\partial f}{\partial x} \) is the slope in the \( x \)-direction while holding \( y \) fixed. Collecting all partial derivatives into a vector gives the gradient:

$$ \nabla f = \left( \frac{\partial f}{\partial x},\ \frac{\partial f}{\partial y} \right) $$

A key theorem from multivariable calculus says the gradient points in the direction of steepest increase, and its length is that steepest rate. So \( -\nabla f \) is the direction of steepest descent, and the update becomes:

$$ \mathbf{w}_{\text{new}} = \mathbf{w}_{\text{old}} - \eta\, \nabla f(\mathbf{w}_{\text{old}}) $$

Worked Example 2: Two Variables

Minimise \( f(x, y) = x^2 + 3y^2 \) from \( (2, 1) \) with \( \eta = 0.1 \).

Step 1. Gradient. \( \frac{\partial f}{\partial x} = 2x \), \( \frac{\partial f}{\partial y} = 6y \), so \( \nabla f = (2x, 6y) \).

Step 2. First step. At \( (2, 1) \): \( \nabla f = (4, 6) \).

$$ (x_1, y_1) = (2 - 0.4,\ 1 - 0.6) = (1.6,\ 0.4) $$

The function value falls from \( f = 4 + 3 = 7 \) to \( f = 2.56 + 0.48 = 3.04 \).

Step 3. Second step. At \( (1.6, 0.4) \): \( \nabla f = (3.2, 2.4) \).

$$ (x_2, y_2) = (1.6 - 0.32,\ 0.4 - 0.24) = (1.28,\ 0.16) $$

Now \( f = 1.6384 + 0.0768 = 1.7152 \).

Step 4. Interpret. The \( y \)-coordinate shrinks faster (by a factor \( 0.4 \) each step) than \( x \) (factor \( 0.8 \)) because the surface is steeper in the \( y \)-direction. Both head towards the true minimum at \( (0, 0) \).

Gradient Descent in Machine Learning

Worked Example 3: Fitting a Line Through the Origin

Suppose we model data \( (1, 2), (2, 4), (3, 5) \) with \( \hat{y} = wx \) and use the mean squared error:

$$ L(w) = \frac{1}{3}\sum_{i=1}^{3} (w x_i - y_i)^2 $$

Step 1. Differentiate using the chain rule.

$$ L'(w) = \frac{2}{3}\sum x_i (w x_i - y_i) = \frac{2}{3}\left( w\sum x_i^2 - \sum x_i y_i \right) $$

Here \( \sum x_i^2 = 1 + 4 + 9 = 14 \) and \( \sum x_i y_i = 2 + 8 + 15 = 25 \), so \( L'(w) = \frac{2}{3}(14w - 25) \).

Step 2. Iterate from \( w_0 = 0 \) with \( \eta = 0.05 \).

• \( L'(0) = -16.667 \), so \( w_1 = 0 + 0.05(16.667) = 0.833 \).
• \( L'(0.833) = \frac{2}{3}(11.667 - 25) = -8.889 \), so \( w_2 = 0.833 + 0.444 = 1.278 \).
• \( L'(1.278) = \frac{2}{3}(17.889 - 25) = -4.741 \), so \( w_3 = 1.278 + 0.237 = 1.515 \).

Step 3. Check. Setting \( L'(w) = 0 \) gives \( w = 25/14 \approx 1.786 \). Our iterates \( 0.833, 1.278, 1.515 \) are climbing steadily towards it. A neural network does exactly this, except the chain rule (called backpropagation) computes millions of partial derivatives at once.

Choosing the Learning Rate

The learning rate is the most important setting in gradient descent.

• Too small: progress is painfully slow.
• Too large: the steps overshoot. For \( f(x) = x^2 \) with \( \eta = 1.1 \), the update is \( x_{\text{new}} = x - 2.2x = -1.2x \), so \( x \) flips sign and grows. The algorithm diverges.

For \( f(x) = x^2 \), the update \( x_{\text{new}} = (1 - 2\eta)x \) converges only when \( |1 - 2\eta| < 1 \), that is, \( 0 < \eta < 1 \). In general, steeper curvature (a larger second derivative) demands a smaller learning rate.

Practical Variants

• Stochastic gradient descent (SGD): estimates the gradient from a small random batch of data rather than the whole dataset, making each step much cheaper.
• Momentum: adds a fraction of the previous step, helping the algorithm roll through flat regions and narrow valleys.
• Adam: adapts the learning rate for each parameter separately and is a popular default in deep learning.

Remember that gradient descent only guarantees a point where the gradient is (nearly) zero. On non-convex functions this may be a local minimum or a saddle point rather than the global minimum.

Did You Know?

• The method of steepest descent is usually credited to the French mathematician Augustin-Louis Cauchy, who described it in 1847, more than a century before computers.
• In very high dimensions, saddle points are far more common than poor local minima, which is one reason gradient descent works better on huge neural networks than early researchers expected.

Key Takeaways

• The derivative gives the slope, and moving against it lowers the function.
• The update rule is \( \mathbf{w} \leftarrow \mathbf{w} - \eta \nabla L \).
• The gradient collects partial derivatives and points uphill most steeply.
• The learning rate must be small enough to avoid divergence but large enough to make progress.
• Backpropagation is simply the chain rule applied to compute gradients efficiently.

Stuck on derivatives, partial derivatives or a machine learning assignment? Amathify's subject experts and PhD scholars can give you step-by-step solutions; reach us on WhatsApp or through the Ask a Question form.