Hat Matrix in Linear Regression: Geometric Interpretation and Leverage Points

1. Definition of the Hat Matrix

In linear regression, the hat matrix H is defined as:

where X is the design matrix of predictors.

The hat matrix relates the observed response values y to the fitted values ŷ:

2. Geometric Interpretation

The hat matrix has several important geometric interpretations:

2.1 Projection Matrix

H is a projection matrix that projects the response vector y onto the column space of X. Geometrically, this means:

• H projects y onto the hyperplane spanned by the columns of X.
• The fitted values ŷ are the orthogonal projections of y onto this hyperplane.
• The residuals e = y - ŷ are orthogonal to the column space of X.

2.2 Properties of H

• H is symmetric: H' = H
• H is idempotent: H² = H
• The trace of H equals the rank of X: tr(H) = rank(X)

2.3 Geometric Meaning of h_ii

The diagonal elements h_ii of H have a special interpretation:

• 0 ≤ h_ii ≤ 1 for all i
• h_ii represents the amount of leverage exerted by the i-th observation on its own fitted value.
• Geometrically, h_ii is the squared distance of the i-th point from the centroid of the X space, relative to the total squared distance of all points from the centroid.

3. Leverage Points

Leverage points are observations that have a large influence on the regression model due to their extreme values in the predictor space.

3.1 Identification of Leverage Points

Leverage points are identified using the diagonal elements h_ii of the hat matrix:

• An observation i is considered a leverage point if h_ii > 2p/n, where p is the number of predictors and n is the number of observations.
• This cutoff is based on the fact that the average h_ii is p/n.

3.2 Geometric Interpretation of Leverage

Geometrically, leverage points are observations that:

• Lie far from the center of the predictor space.
• Have unusual combinations of predictor values.
• Exert a strong influence on the slope of the regression line or hyperplane.

3.3 Impact of Leverage Points

• High leverage points can significantly affect the regression coefficients.
• They may or may not be influential points, depending on whether their y-values are also extreme.
• Points with high leverage but small residuals can actually improve the precision of the regression estimates.

4. Relationship Between Hat Matrix and Leverage Points

The hat matrix provides a direct way to quantify the leverage of each observation:

• The diagonal elements h_ii measure the potential for an observation to influence the regression.
• Larger h_ii values indicate higher leverage.
• The hat matrix allows for easy identification of leverage points without having to calculate distances in multi-dimensional space.

In essence, the hat matrix bridges the gap between the algebraic formulation of linear regression and its geometric interpretation, with leverage points being a key concept in understanding the influence of individual observations on the regression model.