Support Vector Machines: The Geometry of Classification

Many lines can separate two groups of points, but which one is best? Support vector machines (SVMs) answer with a geometric principle: choose the boundary that stays as far as possible from both classes. This simple idea of maximising the margin leads to an elegant optimisation problem, a sparse solution that depends on only a few critical points, and, through kernels, the ability to draw curved boundaries.

Hyperplanes and Distances

In two dimensions a linear classifier is a line; in higher dimensions it is a hyperplane:

$$ \mathbf{w}\cdot\mathbf{x} + b = 0 $$

The vector \(\mathbf{w}\) is perpendicular (normal) to the hyperplane, and \(b\) shifts it away from the origin. We label the classes \(y = +1\) and \(y = -1\), and classify a new point by the sign of \(f(\mathbf{x}) = \mathbf{w}\cdot\mathbf{x} + b\).

The perpendicular distance from a point \(\mathbf{x}_0\) to the hyperplane is

$$ d = \frac{\lvert\mathbf{w}\cdot\mathbf{x}_0 + b\rvert}{\lVert\mathbf{w}\rVert} $$

This formula is the heart of the SVM.

The Maximum-Margin Idea

Scaling \(\mathbf{w}\) and \(b\) by the same constant does not move the hyperplane, so we are free to fix the scale. SVMs choose it so that the closest points on each side satisfy \(|\mathbf{w}\cdot\mathbf{x} + b| = 1\). Then every training point must obey

$$ y_i(\mathbf{w}\cdot\mathbf{x}_i + b) \ge 1 $$

The two parallel hyperplanes \(\mathbf{w}\cdot\mathbf{x} + b = \pm 1\) form the edges of a "street" with the decision boundary down its centre. Using the distance formula, each edge is \(1/\lVert\mathbf{w}\rVert\) from the centre, so the full margin width is

$$ \text{margin} = \frac{2}{\lVert\mathbf{w}\rVert} $$

Maximising the margin is the same as minimising \(\lVert\mathbf{w}\rVert\), or more conveniently:

$$ \min_{\mathbf{w}, b}\; \tfrac{1}{2}\lVert\mathbf{w}\rVert^2 \quad\text{subject to}\quad y_i(\mathbf{w}\cdot\mathbf{x}_i + b) \ge 1 \text{ for all } i $$

This is a convex quadratic programme, so it has a single global solution.

Support Vectors and the Dual

Solving with Lagrange multipliers \(\alpha_i \ge 0\) shows that

$$ \mathbf{w} = \sum_i \alpha_i y_i\mathbf{x}_i, \qquad \sum_i \alpha_i y_i = 0 $$

and that \(\alpha_i > 0\) only for points lying exactly on the margin. These are the support vectors. Every other point has \(\alpha_i = 0\) and could be deleted without changing the boundary. The dual problem also involves the data only through dot products \(\mathbf{x}_i\cdot\mathbf{x}_j\), which opens the door to kernels.

Worked Example 1: A Hard-Margin SVM by Geometry

Class \(+1\): (3, 3), (4, 2), (5, 4). Class \(-1\): (1, 1), (0, 1), (1, 0).

Step 1: Find the closest pair. The negative point nearest the positive class is (1, 1), and the nearest positive point is (3, 3), at distance \(\sqrt{2^2 + 2^2} = \sqrt{8} \approx 2.828\). The point (4, 2) also lies on the line \(x_1 + x_2 = 6\), the same as (3, 3), so it is equally close in the direction that matters.

Step 2: Direction of \(\mathbf{w}\). The optimal boundary is perpendicular to the segment joining (1, 1) and (3, 3), so \(\mathbf{w} = (a, a)\) for some \(a > 0\).

Step 3: Use the margin conditions.

$$ \mathbf{w}\cdot(3,3) + b = 1 \;\Rightarrow\; 6a + b = 1, \qquad \mathbf{w}\cdot(1,1) + b = -1 \;\Rightarrow\; 2a + b = -1 $$

Subtracting gives \(4a = 2\), so \(a = 0.5\) and \(b = -2\). The classifier is

$$ f(\mathbf{x}) = 0.5x_1 + 0.5x_2 - 2 $$

and the boundary is the line \(x_1 + x_2 = 4\).

Step 4: Check every constraint.

• (3, 3): \(f = 1\); (4, 2): \(f = 1\); (5, 4): \(f = 2.5\) — all \(\ge 1\)
• (1, 1): \(f = -1\); (0, 1) and (1, 0): \(f = -1.5\) — all \(\le -1\)

Step 5: Margin. \(\lVert\mathbf{w}\rVert = \sqrt{0.25 + 0.25} = \sqrt{0.5} \approx 0.7071\), so

$$ \text{margin} = \frac{2}{0.7071} \approx 2.828 $$

This equals the distance between (1, 1) and (3, 3), as it should.

Step 6: Multipliers. Writing \(\mathbf{w} = -\alpha_1(1,1) + \alpha_2(3,3) + \alpha_3(4,2)\) with \(\alpha_1 = \alpha_2 + \alpha_3\) gives \(0.5 = 2\alpha_2 + 3\alpha_3\) and \(0.5 = 2\alpha_2 + \alpha_3\). Hence \(\alpha_3 = 0\) and \(\alpha_1 = \alpha_2 = 0.25\). Interestingly, (4, 2) sits on the margin but is not needed to define the boundary.

Step 7: Classify a new point. For (2, 2.5), \(f = 1 + 1.25 - 2 = 0.25 > 0\), so it is labelled \(+1\), although it lies inside the margin, signalling low confidence.

Soft Margins and Hinge Loss

Real data is rarely perfectly separable. The soft-margin SVM allows violations at a price:

$$ \min_{\mathbf{w}, b}\; \tfrac{1}{2}\lVert\mathbf{w}\rVert^2 + C\sum_{i}\max\bigl(0,\; 1 - y_i f(\mathbf{x}_i)\bigr) $$

The second term is the hinge loss. A point safely beyond its margin costs nothing; a point inside the margin or on the wrong side pays in proportion to how far it intrudes. The constant \(C\) controls the trade-off: large \(C\) insists on fitting the training data, small \(C\) prefers a wider margin and tolerates mistakes.

Worked Example 2: Evaluating the Soft-Margin Objective

Keep \(\mathbf{w} = (0.5, 0.5)\), \(b = -2\) and \(C = 1\), but add two noisy points: (2.5, 2.5) labelled \(-1\) and (1.5, 2) labelled \(+1\).

Step 1: Scores.

$$ f(2.5, 2.5) = 1.25 + 1.25 - 2 = 0.5, \qquad f(1.5, 2) = 0.75 + 1 - 2 = -0.25 $$

Step 2: Hinge losses.

• (2.5, 2.5), \(y = -1\): \(\max(0, 1 - (-1)(0.5)) = 1.5\) (wrong side)
• (1.5, 2), \(y = +1\): \(\max(0, 1 - (1)(-0.25)) = 1.25\) (wrong side)
• The six original points all have \(y f \ge 1\), so their losses are 0.

Step 3: Regularisation term. \(\tfrac{1}{2}\lVert\mathbf{w}\rVert^2 = \tfrac{1}{2}(0.5) = 0.25\).

Step 4: Total objective.

$$ 0.25 + 1 \times (1.5 + 1.25) = 3.0 $$

An optimiser would now adjust \(\mathbf{w}\) and \(b\) to lower this total, balancing a wide margin against the two misclassified points.

The Kernel Trick

When classes are separated by a curve, we can map the data into a higher-dimensional space where a flat hyperplane works. Because the SVM needs only dot products, we never build the mapping explicitly; we use a kernel \(K(\mathbf{x}, \mathbf{z}) = \phi(\mathbf{x})\cdot\phi(\mathbf{z})\).

For example, with \(K(\mathbf{x}, \mathbf{z}) = (\mathbf{x}\cdot\mathbf{z})^2\) and \(\mathbf{x} = (1, 2)\), \(\mathbf{z} = (3, 1)\): the dot product is \(3 + 2 = 5\), so \(K = 25\). The hidden mapping is \(\phi(\mathbf{x}) = (x_1^2, \sqrt{2}x_1x_2, x_2^2)\), giving \(\phi(\mathbf{x}) = (1, 2\sqrt{2}, 4)\) and \(\phi(\mathbf{z}) = (9, 3\sqrt{2}, 1)\). Their dot product is \(9 + 12 + 4 = 25\). Same answer, without leaving two dimensions. The popular radial basis function kernel, \(K = e^{-\gamma\lVert\mathbf{x} - \mathbf{z}\rVert^2}\), corresponds to an infinite-dimensional feature space.

Did You Know?

• The original maximum-margin classifier was proposed by Vladimir Vapnik and Alexey Chervonenkis in the 1960s; the kernel version appeared in 1992 and the soft margin in 1995 (Cortes and Vapnik).
• Before deep learning took over, SVMs were among the best performers for handwritten digit recognition and text classification.
• Because only support vectors matter, a trained SVM can be stored compactly even for large datasets.

Key Takeaways

• An SVM picks the hyperplane with the widest margin, \(2/\lVert\mathbf{w}\rVert\), between the classes.
• Only support vectors, the points on the margin, have non-zero multipliers and define the boundary.
• Soft margins use hinge loss and a constant \(C\) to tolerate noisy or overlapping data.
• Kernels let SVMs learn curved boundaries by computing dot products in hidden feature spaces.

If SVM derivations, Lagrange multipliers or a classification project have you stuck, Amathify's experts and PhD scholars can provide step-by-step solutions. Reach us on WhatsApp or through the Ask a Question form.