Proof of MAP and OLS Equivalence with Normal Errors

Theorem:

Given a linear regression model with normally distributed errors and assuming a flat (improper) prior on the parameters, the Maximum A Posteriori (MAP) estimate is equivalent to the Ordinary Least Squares (OLS) estimate.

Proof:

Consider the linear regression model:

\[ \mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon} \]

where \(\mathbf{y}\) is an \(n \times 1\) vector of observations, \(\mathbf{X}\) is an \(n \times p\) matrix of predictors, \(\boldsymbol{\beta}\) is a \(p \times 1\) vector of coefficients, and \(\boldsymbol{\varepsilon}\) is an \(n \times 1\) vector of errors.

Assume \(\boldsymbol{\varepsilon} \sim \mathcal{N}(\mathbf{0}, \sigma^2\mathbf{I})\).

1. The likelihood function is:

\[ P(\mathbf{y}|\boldsymbol{\beta}, \mathbf{X}) = (2\pi\sigma^2)^{-n/2} \exp\left(-\frac{1}{2\sigma^2}(\mathbf{y} - \mathbf{X}\boldsymbol{\beta})^T(\mathbf{y} - \mathbf{X}\boldsymbol{\beta})\right) \]

2. Assuming a flat prior \(P(\boldsymbol{\beta}) = \text{constant}\), the posterior is proportional to the likelihood:

\[ P(\boldsymbol{\beta}|\mathbf{y}, \mathbf{X}) \propto P(\mathbf{y}|\boldsymbol{\beta}, \mathbf{X})P(\boldsymbol{\beta}) \propto P(\mathbf{y}|\boldsymbol{\beta}, \mathbf{X}) \]

3. The MAP estimate maximizes the posterior, which is equivalent to maximizing the likelihood. Taking the log:

\[ \log P(\mathbf{y}|\boldsymbol{\beta}, \mathbf{X}) = -\frac{n}{2}\log(2\pi\sigma^2) - \frac{1}{2\sigma^2}(\mathbf{y} - \mathbf{X}\boldsymbol{\beta})^T(\mathbf{y} - \mathbf{X}\boldsymbol{\beta}) \]

4. Maximizing this is equivalent to minimizing:

\[ (\mathbf{y} - \mathbf{X}\boldsymbol{\beta})^T(\mathbf{y} - \mathbf{X}\boldsymbol{\beta}) \]

5. This is exactly the OLS objective function. To find the minimum, we differentiate with respect to \(\boldsymbol{\beta}\) and set to zero:

\[ \frac{\partial}{\partial\boldsymbol{\beta}}(\mathbf{y} - \mathbf{X}\boldsymbol{\beta})^T(\mathbf{y} - \mathbf{X}\boldsymbol{\beta}) = -2\mathbf{X}^T(\mathbf{y} - \mathbf{X}\boldsymbol{\beta}) = \mathbf{0} \]

6. Solving this equation:

\[ \mathbf{X}^T\mathbf{X}\boldsymbol{\beta} = \mathbf{X}^T\mathbf{y} \]

7. The solution is the OLS estimator:

\[ \hat{\boldsymbol{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y} \]

Conclusion: The MAP estimate with a flat prior on \(\boldsymbol{\beta}\) and normally distributed errors is identical to the OLS estimate. This demonstrates that under these conditions, the Bayesian approach (MAP) and the frequentist approach (OLS) yield the same point estimate for the regression coefficients.