In a linear regression model, each predictor occupies a column of the design matrix X. Two columns are orthogonal when their dot product is zero. Geometrically, we can imagine the columns as vectors meeting at a right angle: neither points in the direction of the other.
If all nonzero columns in X are mutually orthogonal, they are also linearly independent. Each predictor therefore contributes a distinct direction of information to the model.
Why does orthogonality matter?
Simpler calculations
The cross-product matrix has zero off-diagonal entries, so the normal equations separate.
Clearer effects
Each coefficient can be estimated without sharing explanatory direction with another predictor.
Stable estimates
The variance inflation associated with correlated predictors is removed.
Ordinary least squares estimates the coefficients using
A diagonal XTX is easy to invert: take the reciprocal of each diagonal entry. Under the usual homoskedastic regression assumptions, the corresponding coefficient estimates are also uncorrelated. Adding or removing one orthogonal predictor does not change the estimated coefficients of the others.
Orthogonality and correlation are related—but not identical
A zero dot product is not automatically the same as zero correlation. Correlation is calculated after subtracting each variable’s mean. Therefore, orthogonal columns have zero sample correlation when the predictor columns are centred. Orthogonality between a predictor and the intercept column also means that the predictor itself has mean zero.
Centring or standardising predictors does not generally make correlated variables orthogonal. Centring changes their means, and standardising changes their scales, but neither removes the underlying linear association.
How can we obtain orthogonal columns?
Experimental design. Balanced factorial experiments can be deliberately coded so that main-effect and interaction columns are orthogonal. This is one reason carefully designed experiments make effects easier to separate.
Orthogonal transformations. Principal component analysis transforms correlated variables into mutually orthogonal components. The interpretation changes, however, because each component is a combination of the original variables.
Orthogonal polynomials. In polynomial regression, raw terms such as x, x2 and x3 can be strongly correlated. R’s poly() function can construct orthogonal polynomial columns for more stable estimation.
A small R example
The two predictor columns shown above come from a balanced two-factor design. We can check their orthogonality using crossprod(), which calculates XTX.
x1 <- c(-1, -1, 1, 1)
x2 <- c(-1, 1, -1, 1)
X <- cbind(
Intercept = 1,
x1 = x1,
x2 = x2
)
crossprod(X)
# Intercept x1 x2
# Intercept 4 0 0
# x1 0 4 0
# x2 0 0 4All off-diagonal entries are zero. The intercept, x1 and x2 are mutually orthogonal, while each diagonal entry is the squared length of its column.
Orthogonality gives every predictor its own direction in the model. That makes the mathematics cleaner and the contribution of each term easier to see.