OLS Estimator Variance: Homoscedasticity vs. Heteroscedasticity

Part 1: Deriving the Variance of OLS Estimator under Homoscedasticity

Consider the linear regression model:

\[ \mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon} \]

where \(\mathbf{y}\) is an \(n \times 1\) vector of observations, \(\mathbf{X}\) is an \(n \times k\) matrix of regressors, \(\boldsymbol{\beta}\) is a \(k \times 1\) vector of coefficients, and \(\boldsymbol{\varepsilon}\) is an \(n \times 1\) vector of error terms.

The OLS estimator is given by:

\[ \hat{\boldsymbol{\beta}} = (\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\mathbf{y} \]

Substituting \(\mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon}\):

\[ \hat{\boldsymbol{\beta}} = (\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'(\mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon}) = \boldsymbol{\beta} + (\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\boldsymbol{\varepsilon} \]

To find the variance, we focus on the random part: \((\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\boldsymbol{\varepsilon}\)

Under homoscedasticity, we assume:

\[ \mathbb{E}[\boldsymbol{\varepsilon}] = \mathbf{0} \quad \text{and} \quad \mathbb{E}[\boldsymbol{\varepsilon}\boldsymbol{\varepsilon}'] = \sigma^2\mathbf{I} \]

Now, let's derive the variance:

\[ \begin{aligned} \text{Var}(\hat{\boldsymbol{\beta}}) &= \mathbb{E}[(\hat{\boldsymbol{\beta}} - \boldsymbol{\beta})(\hat{\boldsymbol{\beta}} - \boldsymbol{\beta})'] \\ &= \mathbb{E}[((\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\boldsymbol{\varepsilon})((\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\boldsymbol{\varepsilon})'] \\ &= (\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\mathbb{E}[\boldsymbol{\varepsilon}\boldsymbol{\varepsilon}']\mathbf{X}(\mathbf{X}'\mathbf{X})^{-1} \\ &= (\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'(\sigma^2\mathbf{I})\mathbf{X}(\mathbf{X}'\mathbf{X})^{-1} \\ &= \sigma^2(\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\mathbf{X}(\mathbf{X}'\mathbf{X})^{-1} \\ &= \sigma^2(\mathbf{X}'\mathbf{X})^{-1} \end{aligned} \]

Thus, under homoscedasticity, the variance of the OLS estimator is:

\[ \text{Var}(\hat{\boldsymbol{\beta}}) = \sigma^2(\mathbf{X}'\mathbf{X})^{-1} \]

Part 2: Variance of OLS Estimator under Heteroscedasticity

Now, let's consider the case of heteroscedasticity, where the error variance is not constant across observations.

Under heteroscedasticity, we assume:

\[ \mathbb{E}[\boldsymbol{\varepsilon}] = \mathbf{0} \quad \text{and} \quad \mathbb{E}[\boldsymbol{\varepsilon}\boldsymbol{\varepsilon}'] = \boldsymbol{\Omega} \]

where \(\boldsymbol{\Omega}\) is a diagonal matrix with \(\text{diag}(\boldsymbol{\Omega}) = (\sigma_1^2, \sigma_2^2, \ldots, \sigma_n^2)\).

Following the same derivation as before, but replacing \(\sigma^2\mathbf{I}\) with \(\boldsymbol{\Omega}\):

\[ \begin{aligned} \text{Var}(\hat{\boldsymbol{\beta}}) &= (\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\mathbb{E}[\boldsymbol{\varepsilon}\boldsymbol{\varepsilon}']\mathbf{X}(\mathbf{X}'\mathbf{X})^{-1} \\ &= (\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\boldsymbol{\Omega}\mathbf{X}(\mathbf{X}'\mathbf{X})^{-1} \end{aligned} \]

This is the general form of the variance of the OLS estimator under heteroscedasticity.

Comparison and Implications

  1. Efficiency: Under heteroscedasticity, OLS is no longer the Best Linear Unbiased Estimator (BLUE). It remains unbiased but is not the most efficient estimator.
  2. Standard Errors: The standard homoscedastic formula for standard errors (\(\sqrt{\hat{\sigma}^2(\mathbf{X}'\mathbf{X})^{-1}_{ii}}\)) is no longer correct under heteroscedasticity, leading to incorrect inference.
  3. Hypothesis Testing: T-tests and F-tests based on the homoscedastic variance will be invalid under heteroscedasticity.
  4. Estimation Challenges: The heteroscedastic variance \(\boldsymbol{\Omega}\) is typically unknown and needs to be estimated, which can be challenging.
  5. Solutions: Methods like Weighted Least Squares (WLS) or using heteroscedasticity-robust standard errors (e.g., White's standard errors) can address these issues.