Study note · 21 October 2026 · 2 min

Ridge and LASSO Regression

Shrinkage methods to improve predictions.

Ridge regression is a shrinkage method: it pulls coefficients towards zero, but generally does not set them exactly to zero. The predictors remain in the model. This is useful when predictors are highly correlated, because shrinkage can reduce the variance of the estimated coefficients. Ridge itself does not perform variable selection.

Ridge: the L2 penalty

Ridge uses λ as a tuning parameter to control how much shrinkage is applied. It minimises the sum of squared errors (SSE) plus λ times the sum of the squared predictor coefficients—an L2 penalty.

SSE + λ ∑j βj2Ridge: larger λ means more shrinkage.

LASSO: the L1 penalty

Unlike ridge, LASSO (Least Absolute Shrinkage and Selection Operator) can shrink some coefficients exactly to zero. This makes it useful when we want a simpler model with fewer predictors. Its penalty is the sum of the absolute values of the predictor coefficients, rather than their squares.

SSE + λ ∑j |βj|LASSO: some coefficients may become exactly zero.
A detail worth remembering

The intercept is usually kept in both models but is not penalised. Predictors are usually standardised before fitting so that their measurement scales do not determine how strongly they are penalised. Choose λ using validation data or cross-validation.

In short: ridge keeps the predictors and shrinks their coefficients; LASSO can also remove predictors by setting coefficients to zero. A zero LASSO coefficient is a modelling choice, not proof that a variable is scientifically irrelevant.

← Back to all writing Study note · October 2026