The Mathematics of Neural Networks

Neural networks power face recognition, translation apps and chatbots, yet underneath the hype they are built from three familiar pieces of mathematics: weighted sums, simple non-linear functions and the chain rule. If you are comfortable with FSc-level matrices and derivatives, you can follow exactly how a network turns inputs into predictions and how it learns from its mistakes. Let us build one from the ground up.

The Artificial Neuron

A single artificial neuron takes inputs \( x_1, x_2, \ldots, x_n \), multiplies each by a weight \( w_i \), adds a bias \( b \), and passes the result through an activation function \( \sigma \):

$$ z = w_1 x_1 + w_2 x_2 + \cdots + w_n x_n + b = \mathbf{w}\cdot\mathbf{x} + b $$

$$ a = \sigma(z) $$

The weights decide how much each input matters, and the bias shifts the threshold at which the neuron "fires". This idea dates back to Warren McCulloch and Walter Pitts in 1943, and Frank Rosenblatt built a learning version, the perceptron, around 1958.

Activation Functions

Without an activation function, stacking neurons would only ever produce a linear function, because a composition of linear maps is linear. Non-linear activations let networks model curves and complex patterns. Common choices are:

• Sigmoid: \( \sigma(z) = \dfrac{1}{1 + e^{-z}} \), output between 0 and 1, useful for probabilities.
• ReLU: \( \text{ReLU}(z) = \max(0, z) \), simple and fast, the default in most deep networks.
• tanh: output between \(-1\) and \(1\).

The sigmoid has a convenient derivative, which matters for learning:

$$ \sigma'(z) = \sigma(z)\big(1 - \sigma(z)\big) $$

Worked Example 1: One Neuron, Forward Pass

Inputs \( \mathbf{x} = (2, 3) \), weights \( \mathbf{w} = (0.4, -0.2) \), bias \( b = 0.1 \), sigmoid activation. Find the output.

Step 1. Weighted sum.

$$ z = 0.4(2) + (-0.2)(3) + 0.1 = 0.8 - 0.6 + 0.1 = 0.3 $$

Step 2. Activation. \( e^{-0.3} \approx 0.7408 \), so

$$ a = \frac{1}{1 + 0.7408} = \frac{1}{1.7408} \approx 0.5744 $$

The neuron outputs about 0.574, which we could read as "57% confident the answer is yes".

Layers and Matrices

A layer is a group of neurons that all receive the same inputs. Instead of computing each neuron separately, we stack the weights into a matrix \( W \), one row per neuron, and the biases into a vector \( \mathbf{b} \):

$$ \mathbf{h} = \sigma(W\mathbf{x} + \mathbf{b}) $$

A deep network simply repeats this: the output of one layer becomes the input to the next. This is why neural networks are so well suited to GPUs, which are built for fast matrix multiplication.

Worked Example 2: A Two-Layer Forward Pass

Input \( \mathbf{x} = (1, 2) \). The hidden layer has two ReLU neurons:

$$ W_1 = \begin{pmatrix} 1 & -1 \\ 2 & 1 \end{pmatrix}, \qquad \mathbf{b}_1 = \begin{pmatrix} 0 \\ -1 \end{pmatrix} $$

The output neuron is linear with weights \( (0.5, -1) \) and bias \( 0.2 \).

Step 1. Hidden pre-activations.

$$ W_1\mathbf{x} + \mathbf{b}_1 = \begin{pmatrix} 1(1) + (-1)(2) + 0 \\ 2(1) + 1(2) - 1 \end{pmatrix} = \begin{pmatrix} -1 \\ 3 \end{pmatrix} $$

Step 2. Apply ReLU. \( \max(0, -1) = 0 \) and \( \max(0, 3) = 3 \), so \( \mathbf{h} = (0, 3) \). The first neuron is "switched off" for this input.

Step 3. Output.

$$ \hat{y} = 0.5(0) + (-1)(3) + 0.2 = -2.8 $$

Check: with ReLU, a negative pre-activation always gives exactly zero, so only the second hidden neuron influences this prediction. Changing the first row of \( W_1 \) slightly would not change \( \hat{y} \) at all, which is exactly what backpropagation will discover (its gradient is zero).

Measuring Error: The Loss Function

To learn, the network compares its output \( a \) with the true target \( y \). A simple choice is the squared error

$$ L = \tfrac{1}{2}(a - y)^2 $$

For classification, cross-entropy loss \( L = -\big[y \ln a + (1 - y)\ln(1 - a)\big] \) is usually preferred. Training means adjusting all weights to make \( L \) small, using gradient descent:

$$ w \leftarrow w - \eta \frac{\partial L}{\partial w} $$

Backpropagation: The Chain Rule at Scale

The loss depends on a weight only indirectly: \( w \) affects \( z \), which affects \( a \), which affects \( L \). The chain rule links them:

$$ \frac{\partial L}{\partial w_i} = \frac{\partial L}{\partial a}\cdot\frac{\partial a}{\partial z}\cdot\frac{\partial z}{\partial w_i} = (a - y)\cdot \sigma'(z) \cdot x_i $$

Backpropagation, popularised by David Rumelhart, Geoffrey Hinton and Ronald Williams in 1986, applies this rule layer by layer from the output backwards, reusing intermediate results so that every gradient in the network is found in a single backward sweep.

Worked Example 3: One Training Step

Use the neuron from Example 1 with target \( y = 1 \) and learning rate \( \eta = 0.5 \).

Step 1. Error signal. \( a - y = 0.5744 - 1 = -0.4256 \).

Step 2. Sigmoid derivative. \( \sigma'(z) = 0.5744(1 - 0.5744) = 0.5744 \times 0.4256 \approx 0.2445 \).

Step 3. Combine. Define \( \delta = (a - y)\sigma'(z) = -0.4256 \times 0.2445 \approx -0.1040 \).

Step 4. Gradients. Multiply by each input (and by 1 for the bias):

• \( \partial L/\partial w_1 = \delta \cdot 2 = -0.2081 \)
• \( \partial L/\partial w_2 = \delta \cdot 3 = -0.3121 \)
• \( \partial L/\partial b = \delta = -0.1040 \)

Step 5. Update.

• \( w_1 = 0.4 - 0.5(-0.2081) \approx 0.5040 \)
• \( w_2 = -0.2 - 0.5(-0.3121) \approx -0.0440 \)
• \( b = 0.1 - 0.5(-0.1040) = 0.1520 \)

Step 6. Check the improvement. New \( z = 0.5040(2) - 0.0440(3) + 0.1520 = 1.028 \), so \( a = \sigma(1.028) \approx 0.7365 \). The output rose from 0.574 to 0.737, closer to the target of 1. The loss fell from \( \tfrac12(0.4256)^2 \approx 0.0906 \) to \( \tfrac12(0.2635)^2 \approx 0.0347 \).

Why Depth Helps

A single neuron draws a straight boundary (a line or plane) between classes, so it cannot solve the famous XOR problem, where the "true" points sit on opposite corners of a square. Adding just one hidden layer fixes this. The universal approximation theorem (Cybenko, 1989, and Hornik and colleagues around the same time) shows that one hidden layer with enough neurons can approximate any continuous function on a bounded region as closely as desired. In practice, deeper networks learn such functions with far fewer neurons.

Did You Know?

• Marvin Minsky and Seymour Papert's 1969 book Perceptrons highlighted the XOR limitation and contributed to a long slowdown in neural network research.
• Large modern language models contain hundreds of billions of weights, yet each one is updated by the same chain-rule calculation you just did by hand.

Key Takeaways

• A neuron computes a weighted sum plus bias, then applies a non-linear activation.
• A layer is a matrix multiplication followed by an activation, \( \mathbf{h} = \sigma(W\mathbf{x} + \mathbf{b}) \).
• Training minimises a loss function using gradient descent.
• Backpropagation is the chain rule applied efficiently from the output layer backwards.
• Hidden layers with non-linear activations let networks model complex, curved patterns.

If you want step-by-step help with neural network maths or a machine learning assignment, Amathify's subject experts and PhD scholars are here for you; contact us on WhatsApp or through the Ask a Question form.