Regularization: Ridge, Lasso and Overfitting

A model that scores perfectly on its training data but fails on new data has not learned anything useful; it has memorised. Regularization is the mathematical fix: we add a penalty to the loss function so that the model prefers simpler explanations. In this article we build ridge and lasso regression from the ground up and work through the numbers by hand.

What Overfitting Looks Like

Suppose you have 10 data points that follow a roughly straight line with some noise. A straight line fits them reasonably well. A polynomial of degree 9 can pass through every single point, giving zero training error. Yet between the points it swings wildly, and on fresh data its predictions are terrible.

The degree-9 polynomial has too much freedom. Its coefficients become very large positive and negative numbers that cancel each other at the training points but explode elsewhere. That observation is the key insight behind regularization: overfitted models tend to have large weights. If we punish large weights, we discourage overfitting.

Typical warning signs of overfitting include:

• Training error far lower than validation error.
• Coefficients with huge magnitudes or unstable signs.
• Predictions that change dramatically when one training point is removed.

The Ordinary Least Squares Starting Point

In linear regression we predict \( \hat{y} = \mathbf{w}^\top \mathbf{x} \) and choose \( \mathbf{w} \) to minimise the sum of squared errors:

$$ J_{\text{OLS}}(\mathbf{w}) = \sum_{i=1}^{n} \left(y_i - \mathbf{w}^\top \mathbf{x}_i\right)^2 = \lVert \mathbf{y} - X\mathbf{w} \rVert^2 . $$

Setting the gradient to zero gives the normal equations and the familiar solution

$$ \mathbf{w}_{\text{OLS}} = (X^\top X)^{-1} X^\top \mathbf{y}. $$

When features are highly correlated, or when there are more features than data points, \( X^\top X \) is nearly singular. Its inverse then contains enormous numbers, and the weights blow up. This is exactly the unstable behaviour we want to control.

Ridge Regression (L2 Penalty)

Ridge regression, introduced by Arthur Hoerl and Robert Kennard in 1970, adds the squared length of the weight vector to the loss:

$$ J_{\text{ridge}}(\mathbf{w}) = \lVert \mathbf{y} - X\mathbf{w} \rVert^2 + \lambda \lVert \mathbf{w} \rVert_2^2, \qquad \lVert \mathbf{w} \rVert_2^2 = \sum_j w_j^2 . $$

The number \( \lambda \ge 0 \) is the regularization strength. Differentiating and setting the gradient to zero:

$$ -2X^\top(\mathbf{y} - X\mathbf{w}) + 2\lambda \mathbf{w} = 0 \;\Rightarrow\; \mathbf{w}_{\text{ridge}} = (X^\top X + \lambda I)^{-1} X^\top \mathbf{y}. $$

Adding \( \lambda I \) to the matrix guarantees it is invertible for any \( \lambda > 0 \), which is why ridge is so stable. The weights shrink smoothly towards zero as \( \lambda \) grows, but they almost never become exactly zero.

Worked Example 1: Ridge with One Feature

Take three data points with no intercept: \( x = (1, 2, 3) \) and \( y = (1, 2, 2) \). With a single weight \( w \), the ridge formula reduces to

$$ w_{\text{ridge}} = \frac{\sum x_i y_i}{\sum x_i^2 + \lambda}. $$

Step 1. Compute the sums: \( \sum x_i y_i = 1(1) + 2(2) + 3(2) = 1 + 4 + 6 = 11 \) and \( \sum x_i^2 = 1 + 4 + 9 = 14 \).

Step 2. Ordinary least squares (\( \lambda = 0 \)): \( w = 11/14 \approx 0.786 \).

Step 3. Ridge with \( \lambda = 2 \): \( w = 11/(14 + 2) = 11/16 = 0.6875 \).

Step 4. Ridge with \( \lambda = 14 \): \( w = 11/28 \approx 0.393 \).

Notice the pattern: as \( \lambda \) increases, the weight shrinks, but it is never pushed to exactly zero. We trade a little bias for a big reduction in variance.

Lasso Regression (L1 Penalty)

Lasso, proposed by Robert Tibshirani in 1996, uses absolute values instead of squares:

$$ J_{\text{lasso}}(\mathbf{w}) = \frac{1}{2}\lVert \mathbf{y} - X\mathbf{w} \rVert^2 + \lambda \lVert \mathbf{w} \rVert_1, \qquad \lVert \mathbf{w} \rVert_1 = \sum_j |w_j| . $$

(The factor \( \tfrac{1}{2} \) is a common convention that tidies up the derivative.) Because \( |w| \) has a sharp corner at zero, lasso can set weights to exactly zero. It therefore performs automatic feature selection, which is extremely useful when you have hundreds of candidate predictors and suspect only a few matter.

There is no single closed-form formula for lasso in general, but in one dimension the solution is given by the soft-thresholding operator. Let \( a = \sum x_i y_i \) and \( b = \sum x_i^2 \). Then

$$ w_{\text{lasso}} = \frac{\operatorname{sign}(a)\,\max\left(|a| - \lambda,\, 0\right)}{b}. $$

Worked Example 2: Lasso Soft-Thresholding

Use the same data, so \( a = 11 \) and \( b = 14 \).

Step 1. For \( \lambda = 4 \): \( |a| - \lambda = 11 - 4 = 7 > 0 \), so \( w = 7/14 = 0.5 \).

Step 2. For \( \lambda = 8 \): \( 11 - 8 = 3 \), so \( w = 3/14 \approx 0.214 \).

Step 3. For \( \lambda = 12 \): \( 11 - 12 = -1 < 0 \), so the maximum is 0 and \( w = 0 \).

Step 4. Check the boundary: the weight becomes zero as soon as \( \lambda \ge |a| = 11 \). Above that point the feature is dropped from the model entirely.

Compare this with ridge, where even \( \lambda = 1000 \) gives \( w = 11/1014 \approx 0.011 \): small, but not zero.

The Geometry: Why Lasso Gives Zeros

Regularized regression can be rewritten as a constrained problem: minimise the squared error subject to \( \lVert \mathbf{w} \rVert \le t \). In two dimensions:

• The ridge constraint \( w_1^2 + w_2^2 \le t^2 \) is a circle.
• The lasso constraint \( |w_1| + |w_2| \le t \) is a diamond with corners on the axes.

The contours of the squared-error loss are ellipses centred on the OLS solution. The solution is the first point where an expanding ellipse touches the constraint region. A circle has no corners, so the touching point is usually somewhere with both weights non-zero. A diamond's corners stick out, and the ellipse very often hits a corner first, where one coordinate is exactly zero. That picture explains sparsity better than any formula.

Choosing Lambda

The value of \( \lambda \) is a hyperparameter, not something learned from training error (training error always prefers \( \lambda = 0 \)). The standard approach is k-fold cross-validation:

1. Pick a grid of values, for example \( \lambda \in \{0.001, 0.01, 0.1, 1, 10, 100\} \).
2. For each value, train on \( k-1 \) folds and measure error on the held-out fold, rotating through all folds.
3. Choose the \( \lambda \) with the lowest average validation error, or the largest \( \lambda \) within one standard error of it for an even simpler model.

Always standardise features (mean 0, standard deviation 1) before applying ridge or lasso. Otherwise a feature measured in rupees and one measured in lakhs of rupees are penalised very differently, which makes no sense.

Elastic Net and Deep Learning

Elastic net combines both penalties, \( \lambda_1 \lVert \mathbf{w} \rVert_1 + \lambda_2 \lVert \mathbf{w} \rVert_2^2 \), keeping lasso's sparsity while handling groups of correlated features more gracefully. In neural networks, the L2 penalty is usually called weight decay, because the gradient step \( w \leftarrow w - \eta(\nabla L + 2\lambda w) \) multiplies each weight by \( (1 - 2\eta\lambda) \) before the usual update, decaying it a little every iteration. Dropout and early stopping are other regularizers built on the same philosophy.

Did You Know?

• Ridge regression has a Bayesian interpretation: it is exactly the most probable weight vector when each weight has a Gaussian prior centred at zero. Lasso corresponds to a Laplace prior.
• The name "lasso" stands for "least absolute shrinkage and selection operator".
• Ridge regression was first motivated by chemical engineering data with strongly correlated variables, long before machine learning was a field.

Key Takeaways

• Overfitting shows up as large, unstable weights; regularization penalises them.
• Ridge adds \( \lambda \sum w_j^2 \), shrinks weights smoothly and always has a closed-form solution.
• Lasso adds \( \lambda \sum |w_j| \) and can set weights exactly to zero, performing feature selection.
• The diamond-shaped L1 constraint has corners on the axes, which is why lasso produces sparse models.
• Choose \( \lambda \) by cross-validation and standardise features first.

Stuck on a regression assignment or a regularization derivation? Send your question to Amathify's subject experts and PhD scholars on WhatsApp or through the Ask a Question form for clear, step-by-step solutions.