Regression Analysis: The Geometry of Projection

Regression analysis is the primary statistical tool for modeling the relationship between a dependent variable $y$and one or more independent variables$X$. At its mathematical heart, linear regression is an exercise in **orthogonal projection** within a high-dimensional vector space.

1. Ordinary Least Squares (OLS)

The goal of OLS is to find the vector of coefficients$\boldsymbol{\beta}$that minimizes the sum of squared residuals.

1.1 The Matrix Formulation

Given$n$observations and$k$predictors, we define the model in matrix form:$$\mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\epsilon}$$-$\mathbf{y} \in \mathbb{R}^n$: The vector of observations.

-$\mathbf{X} \in \mathbb{R}^{n \times k}$: The design matrix of predictors.

-$\boldsymbol{\beta} \in \mathbb{R}^k$: The vector of unknown coefficients.

-$\boldsymbol{\epsilon} \in \mathbb{R}^n$: The vector of errors.

The OLS solution that minimizes$\parallel \mathbf{y} - \mathbf{X}\boldsymbol{\beta} \parallel^2$is found by solving the **Normal Equations**:$$\mathbf{X}^T \mathbf{X} \hat{\boldsymbol{\beta}} = \mathbf{X}^T \mathbf{y}$$

$$\hat{\boldsymbol{\beta}} = (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{y}$$## 2. Geometric Intuition: Orthogonal Projection

The most powerful way to understand regression is through the geometry of linear algebra.

2.1 The Column Space of X

The matrix$\mathbf{X}$defines a subspace in$\mathbb{R}^n$called the **Column Space** (or Range) of$\mathbf{X}$, denoted$\text{Col}(\mathbf{X})$. Any prediction$\hat{\mathbf{y}} = \mathbf{X}\boldsymbol{\beta}$must lie within this subspace.

Because the observed vector$\mathbf{y}$likely contains noise, it does not lie exactly within$\text{Col}(\mathbf{X})$.

2.2 The Projection (Hat) Matrix

The "best fit"$\hat{\mathbf{y}}$is the point in$\text{Col}(\mathbf{X})$that is **closest** to$\mathbf{y}$in terms of Euclidean distance. Geometrically, this is the **orthogonal projection** of$\mathbf{y}$onto the column space.

The transformation that performs this projection is the **Hat Matrix**$\mathbf{H}$:$$\hat{\mathbf{y}} = \mathbf{H}\mathbf{y}$$

$$\mathbf{H} = \mathbf{X}(\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T$$- **Properties of H:** It is symmetric ($\mathbf{H}^T = \mathbf{H}$) and idempotent ($\mathbf{H}^2 = \mathbf{H}$).

- **The Residual Vector:**$\mathbf{e} = \mathbf{y} - \hat{\mathbf{y}} = (\mathbf{I} - \mathbf{H})\mathbf{y}$. Geometrically, the residuals are the vector component of$\mathbf{y}$that is orthogonal to the column space of$\mathbf{X}$.

3. Quantitative Foundations: The Gauss-Markov Theorem

Why is OLS so widely used? The Gauss-Markov theorem provides the justification.

3.1 BLUE (Best Linear Unbiased Estimator)

Under the assumptions of linearity, full rank, exogeneity ($\mathbb{E}[\epsilon|X] = 0$), and homoscedasticity ($\text{Var}(\epsilon) = \sigma^2 I$), the OLS estimator$\hat{\boldsymbol{\beta}}$is the **Best Linear Unbiased Estimator**.

"Best" here means that it has the **minimum variance** among all linear unbiased estimators.

Table 1: ANOVA for Linear Regression

| :--- | :--- | :--- | :--- |

| **Model** |$k - 1$|$\sum (\hat{y}_i - \bar{y})^2$|$\text{SSM} / (k-1)$|

| **Error** |$n - k$|$\sum (y_i - \hat{y}_i)^2$|$\text{SSE} / (n-k)$|

| **Total** |$n - 1$|$\sum (y_i - \bar{y})^2$| |

The Ratio$F = \text{MSM} / \text{MSE}$allows us to test if the model as a whole is statistically significant.

4. Real-World Applications

4.1 Finance: The Capital Asset Pricing Model (CAPM)

In finance, regression is used to calculate the risk of an asset. The "Beta" ($\beta$) of a stock is the slope of a linear regression where the market return is the independent variable and the stock return is the dependent variable. A$\beta > 1$indicates the stock is more volatile than the market (leveraged projection).

4.2 Machine Learning and Regularization

When the number of predictors$k$is large, the matrix$\mathbf{X}^T \mathbf{X}$may be ill-conditioned (nearly singular). This corresponds to the geometric problem of a very "thin" or "flat" column space.

**Ridge Regression** solves this by adding$\lambda \mathbf{I}$ to the diagonal, which geometrically "inflates" the subspace and prevents the coefficients from exploding, a direct application of Bayesian reasoning (Gaussian prior).