How Mathematics Is Driving Artificial Intelligence Research

Headlines about artificial intelligence focus on chatbots, image generators and self-driving cars, but every one of these systems is, underneath, a very large mathematical object. The breakthroughs of the last decade came as much from mathematical ideas as from faster computers. For students wondering whether algebra and calculus still matter in the age of AI, the honest answer is that they matter more than ever.

Linear Algebra: The Language of Data

Every piece of data an AI system touches, whether a word, an image or a sound, is first turned into a vector of numbers. A neural network layer is essentially a matrix multiplication followed by a simple non-linear function:

$$ \mathbf{h} = \sigma(W\mathbf{x} + \mathbf{b}). $$

Large language models perform enormous numbers of such multiplications for every word they generate. This is why graphics processors, designed for fast matrix arithmetic, became the workhorses of AI.

Linear algebra also explains meaning. Word embeddings place words as points in a high-dimensional space, and the angle between vectors, measured by cosine similarity \( \cos\theta = \mathbf{a}\cdot\mathbf{b}/(\lVert\mathbf{a}\rVert\lVert\mathbf{b}\rVert) \), captures how related two words are. Research on compressing models uses low-rank matrix factorization: a \( 4096 \times 4096 \) weight matrix has about 16.8 million entries, but if it is approximated as the product of a \( 4096 \times 16 \) and a \( 16 \times 4096 \) matrix, only \( 2 \times 4096 \times 16 = 131{,}072 \) numbers need to be trained. That idea underlies popular fine-tuning methods such as LoRA.

Calculus: How Machines Learn

Learning means adjusting weights to reduce a loss function, and the direction of adjustment comes from derivatives. The algorithm that computes them efficiently, backpropagation, is the chain rule applied systematically from the output back to the input. It was popularised for neural networks by David Rumelhart, Geoffrey Hinton and Ronald Williams in 1986, although related ideas appeared earlier.

Worked Example 1: Backpropagation Through One Neuron

A neuron receives input \( x = 2 \), has weight \( w = 0.5 \) and bias \( b = -0.5 \), and uses the sigmoid activation \( \sigma(z) = 1/(1 + e^{-z}) \). The target output is \( t = 1 \) and the loss is \( L = (\hat{y} - t)^2 \).

Step 1. Forward pass.

• \( z = wx + b = 0.5(2) - 0.5 = 0.5 \)
• \( \hat{y} = \sigma(0.5) = 1/(1 + e^{-0.5}) = 1/(1 + 0.6065) \approx 0.6225 \)
• \( L = (0.6225 - 1)^2 = (-0.3775)^2 \approx 0.1425 \)

Step 2. Chain rule. The loss depends on \( w \) through \( \hat{y} \) and \( z \):

$$ \frac{\partial L}{\partial w} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z} \cdot \frac{\partial z}{\partial w}. $$

Step 3. Each factor.

• \( \partial L/\partial \hat{y} = 2(\hat{y} - t) = 2(-0.3775) = -0.7551 \)
• \( \partial \hat{y}/\partial z = \sigma(z)(1 - \sigma(z)) = 0.6225 \times 0.3775 \approx 0.2350 \)
• \( \partial z/\partial w = x = 2 \)

Step 4. Multiply. \( \partial L/\partial w \approx -0.7551 \times 0.2350 \times 2 \approx -0.355 \). Similarly \( \partial L/\partial b \approx -0.177 \), since \( \partial z/\partial b = 1 \).

Step 5. Update and sanity check. With learning rate 0.5, \( w \leftarrow 0.5 + 0.5(0.355) \approx 0.677 \). The gradient is negative, so increasing \( w \) lowers the loss, which makes sense: a larger \( w \) raises \( z \), pushing \( \hat{y} \) towards the target of 1. A network with billions of weights does exactly this, using the same rule, for every weight at once.

Research on why deep networks train at all, and why gradients sometimes vanish or explode as they pass through many layers, is fundamentally a study of products of derivatives. Innovations such as residual connections and careful weight initialisation came directly from analysing that mathematics.

Probability and Statistics: Handling Uncertainty

AI systems rarely know the answer for certain, so they work with probabilities. A language model outputs a probability distribution over the next word, computed with the softmax function. Classifiers are trained by maximising likelihood. Generative image models, such as diffusion models, learn to reverse a process that gradually adds Gaussian noise to images, a construction built on stochastic processes. Bayesian methods quantify how confident a model should be, which is vital in medicine and finance where an overconfident wrong answer is dangerous.

Information Theory: Measuring Surprise

Claude Shannon's 1948 theory of information gives AI its most common loss function. The information content of an event with probability \( p \) is \( -\log_2 p \) bits, and the cross-entropy loss averages this over the true answers.

Worked Example 2: Cross-Entropy and Scaling

Part (a). Surprise in bits. A model predicts the correct next word with probability 0.8. The loss for that word is \( -\log_2 0.8 \approx 0.322 \) bits. If it had given the correct word only 0.25, the loss would be \( -\log_2 0.25 = 2 \) bits. A better model is, quite literally, less surprised by real text.

Part (b). Scaling laws. Researchers studying language models, notably Kaplan and colleagues in 2020, observed that test loss falls roughly as a power law in the number of parameters \( N \):

$$ L(N) \approx \left(\frac{N_c}{N}\right)^{\alpha}, \qquad \alpha \approx 0.076 . $$

What does doubling the model size buy? Taking the ratio,

$$ \frac{L(2N)}{L(N)} = \left(\frac{1}{2}\right)^{0.076} = 2^{-0.076} \approx 0.949 . $$

So doubling parameters cuts the loss by only about 5%. Multiplying \( N \) by 10 gives \( 10^{-0.076} \approx 0.839 \), a reduction of about 16%. Sanity check: a small exponent means slow but steady improvement, which matches the observation that each big gain has required training models many times larger. These simple power laws now guide how labs decide model size, data size and compute budgets, and later work refined the exponents and the best balance between them.

Optimization: Finding Good Solutions

Training a network is a high-dimensional optimization problem with millions or billions of variables. Methods such as stochastic gradient descent with momentum and Adam, along with learning-rate schedules, are products of optimization theory. A major research question is why such simple methods find good solutions in landscapes full of saddle points and local minima. Studies of the geometry of loss surfaces, including how "flat" a minimum is, try to explain why some trained models generalise better than others.

Geometry, Symmetry and Beyond

Several newer research directions are openly mathematical:

• Geometric deep learning builds networks that respect symmetries, such as rotations of a molecule, using ideas from group theory.
• Graph neural networks use graph theory and spectral methods to learn from social networks, road maps and chemical structures.
• Optimal transport provides ways to measure distance between probability distributions, used in generative models.
• AI for mathematics runs the other way: systems are now helping mathematicians search for proofs, check formal arguments and discover patterns, while models like AlphaFold applied deep learning to predicting protein structures.

What This Means for Students

If you want to work in AI, the core mathematical toolkit is clear:

1. Linear algebra: vectors, matrices, eigenvalues and decompositions.
2. Calculus: partial derivatives, the chain rule and gradients.
3. Probability and statistics: distributions, expectation, likelihood and Bayes' theorem.
4. Optimization: gradient methods and convexity.

The topics you meet in FSc, A Level and the first years of university are not old-fashioned; they are the foundations on which the most advanced systems are built.

Did You Know?

• The perceptron, an early neural network, was demonstrated by Frank Rosenblatt in 1958, but deep learning only took off decades later when data, computing power and better mathematics came together.
• The sigmoid's derivative is never larger than 0.25, which helps explain why gradients shrank in early deep networks and why ReLU became popular.
• Training a large modern model can involve on the order of \( 10^{24} \) or more arithmetic operations.

Key Takeaways

• Linear algebra represents data and computes every layer; low-rank ideas make large models cheaper to adapt.
• Calculus, through the chain rule in backpropagation, is how networks learn.
• Probability and information theory supply the outputs and loss functions, such as softmax and cross-entropy.
• Empirical scaling laws are power laws: doubling parameters lowers loss by only about 5% when \( \alpha \approx 0.076 \).
• Strong foundations in school and university mathematics are the best preparation for AI research.

Want to master the maths behind AI, or need help with an assignment in calculus, linear algebra or statistics? Amathify's experts and PhD scholars offer step-by-step solutions via WhatsApp or the Ask a Question form.