Eigen Analysis and Inverse of M = I + XX'

1. Eigenvalues

For M = I + XX', where X is an n×n matrix:

1. Let λ₁, λ₂, ..., λᵣ be the non-zero eigenvalues of XX' (r ≤ n).
2. The eigenvalues of M are:
   • 1 + λᵢ for i = 1 to r
   • 1 with multiplicity n - r

Proof sketch:

• If v is an eigenvector of XX' with eigenvalue λ: XX'v = λv
• Then, Mv = (I + XX')v = v + XX'v = v + λv = (1 + λ)v
• For vectors in the nullspace of X': X'v = 0, so Mv = v

2. Eigenvectors

1. For eigenvalues 1 + λᵢ: The eigenvectors are the same as the eigenvectors of XX' for λᵢ.
2. For eigenvalue 1: The eigenvectors are any vectors in the nullspace of X'.

3. Inverse of M

The inverse of M can be derived using the Sherman-Morrison-Woodbury formula:

Proof:

1. Let M⁻¹ = I - X(I + X'X)⁻¹X'
2. Multiply M and M⁻¹:
   \[ \begin{aligned} MM^{-1} &= (I + XX')(I - X(I + X'X)^{-1}X') \\ &= I - X(I + X'X)^{-1}X' + XX' - XX'X(I + X'X)^{-1}X' \\ &= I - X(I + X'X)^{-1}X' + X(I - (I + X'X)(I + X'X)^{-1})X' \\ &= I - X(I + X'X)^{-1}X' + X((I + X'X)(I + X'X)^{-1} - I)X' \\ &= I - X(I + X'X)^{-1}X' + X(I - I)X' = I \end{aligned} \]

4. Properties and Implications

• M is always positive definite as all its eigenvalues are greater than 1.
• The inverse formula is computationally efficient, especially when n is large and the rank of X is small.
• This structure appears in various statistical and machine learning contexts, such as in ridge regression and Gaussian process regression.