Matrix Proof: Orthogonality of Residuals and Centered Predicted Values

1. Setup and Definitions

Let's define our matrices and vectors:

y: n × 1 vector of observed values
ŷ: n × 1 vector of predicted values
e: n × 1 vector of residuals (e = y - ŷ)
1: n × 1 vector of ones
I: n × n identity matrix
H: n × n hat matrix (H = X(X'X)⁻¹X')
ȳ: scalar, mean of y

2. Theorem to Prove

We want to prove:

\[ \sum_{i=1}^n (y_i - \hat{y}_i)(\hat{y}_i - \bar{y}) = 0 \]

In matrix notation, this is equivalent to:

\[ \mathbf{e}'(\hat{\mathbf{y}} - \bar{y}\mathbf{1}) = 0 \]

3. Proof

Step 1: Express e and ŷ in terms of y and H

\[ \begin{aligned} \mathbf{e} &= \mathbf{y} - \hat{\mathbf{y}} = (\mathbf{I} - \mathbf{H})\mathbf{y} \\ \hat{\mathbf{y}} &= \mathbf{H}\mathbf{y} \end{aligned} \]

Step 2: Express ȳ in terms of y

\[ \bar{y} = \frac{1}{n}\mathbf{1}'\mathbf{y} \]

Step 3: Substitute into the expression we want to prove

\[ \begin{aligned} \mathbf{e}'(\hat{\mathbf{y}} - \bar{y}\mathbf{1}) &= [(\mathbf{I} - \mathbf{H})\mathbf{y}]'[\mathbf{H}\mathbf{y} - \frac{1}{n}\mathbf{1}'\mathbf{y}\mathbf{1}] \\ &= \mathbf{y}'(\mathbf{I} - \mathbf{H})[\mathbf{H} - \frac{1}{n}\mathbf{1}\mathbf{1}']\mathbf{y} \end{aligned} \]

Step 4: Use properties of H and 1

Key properties:

H is idempotent: H² = H
H is symmetric: H' = H
H1 = 1 (the regression includes an intercept)
(I - H)1 = 0 (consequence of H1 = 1)

Step 5: Simplify the expression

\[ \begin{aligned} \mathbf{y}'(\mathbf{I} - \mathbf{H})[\mathbf{H} - \frac{1}{n}\mathbf{1}\mathbf{1}']\mathbf{y} &= \mathbf{y}'[\mathbf{H} - \mathbf{H}^2 - \frac{1}{n}(\mathbf{I} - \mathbf{H})\mathbf{1}\mathbf{1}']\mathbf{y} \\ &= \mathbf{y}'[\mathbf{H} - \mathbf{H} - \frac{1}{n}(\mathbf{0})\mathbf{1}']\mathbf{y} \\ &= \mathbf{y}'\mathbf{0}\mathbf{y} = 0 \end{aligned} \]

4. Interpretation and Implications

Geometric Interpretation: The residuals are orthogonal to the centered predicted values.
Regression Through the Mean: This property ensures that the regression line passes through the point (x̄, ȳ).
Partitioning of Sums of Squares: This orthogonality allows for the clean decomposition of total sum of squares into explained and residual sum of squares.
Efficiency of OLS: It's a key property in proving that OLS estimators are BLUE (Best Linear Unbiased Estimators).

5. Conclusion

We have proven that Σ(yᵢ - ŷᵢ)(ŷᵢ - ȳ) = 0 using matrix notation. This proof demonstrates the orthogonality of residuals to centered predicted values, which is a fundamental property in regression analysis. It underlies many important results in linear regression theory and provides insights into the nature of least squares estimation.

1. Vector Space Setup

Let's visualize regression in n-dimensional space:

y: The observed values vector (point in n-dimensional space)
C(X): The column space of X (subspace where all possible ŷ lie)
1: The vector of ones (represents the mean direction)
ŷ: The predicted values (projection of y onto C(X))
e: The residual vector (y - ŷ)

2. Key Geometric Insights

ŷ is the orthogonal projection of y onto C(X)
e is perpendicular to all vectors in C(X)
1 (the mean direction) is in C(X) if there's an intercept
(ŷ - ȳ1) is ŷ centered around its mean

3. Geometric Proof

To show that Σ(yᵢ - ŷᵢ)(ŷᵢ - ȳ) = 0, or equivalently, e'(ŷ - ȳ1) = 0:

Step 1: Decompose ŷ

We can write ŷ as the sum of its mean component and the centered component:

\[ \hat{\mathbf{y}} = \bar{y}\mathbf{1} + (\hat{\mathbf{y}} - \bar{y}\mathbf{1}) \]

Step 2: Orthogonality of e to components of ŷ

e ⊥ ȳ1 because 1 is in C(X)
e ⊥ (ŷ - ȳ1) because this vector is in C(X)

Step 3: Conclusion

Since e is orthogonal to both components of ŷ, it must be orthogonal to ŷ - ȳ1.

4. Intuitive Understanding

The residual vector e is perpendicular to the entire C(X) plane.
Both ȳ1 and (ŷ - ȳ1) lie entirely within C(X).
Therefore, e must be perpendicular to (ŷ - ȳ1).
This perpendicularity is equivalent to the statement Σ(yᵢ - ŷᵢ)(ŷᵢ - ȳ) = 0.

5. Implications

Variance Decomposition: This orthogonality allows us to decompose the total variance into explained and unexplained components.
Least Squares Property: It ensures that ŷ is indeed the closest point in C(X) to y.
Model Interpretation: The residuals contain no linear information about the predictors or their linear combinations.
Efficiency of Estimators: This property is crucial in proving that OLS estimators are BLUE (Best Linear Unbiased Estimators).

6. Conclusion

This geometric approach provides an intuitive understanding of why Σ(yᵢ - ŷᵢ)(ŷᵢ - ȳ) = 0. It's a direct consequence of how linear regression finds the best fit by projecting the observed values onto the subspace spanned by the predictors. The orthogonality of residuals to centered predicted values is a fundamental property that underpins many key results in regression analysis.