Ordinary Least Squares (OLS)

Interactive exploration of linear regression and the minimization of squared errors.

Click/Tap to add points

Manual Controls

Error Metrics

Equation
$\hat{y} =$ 1.00$x$ + 0.00
Sum of Squared Residuals (SSR)
0.00
Goal: Minimize this number

The Mathematics of OLS

The goal of Ordinary Least Squares is to find the line that fits the data "best." In statistics, "best" is defined as the line that minimizes the sum of the squared vertical distances (residuals) between the observed data points and the fitted line.

1. The Model

We assume a linear relationship between our independent variable $x$ and dependent variable $y$: $$ y_i = \beta_1 x_i + \beta_0 + \epsilon_i $$ Where $\beta_1$ is the slope, $\beta_0$ is the intercept, and $\epsilon_i$ is the error (residual) for the $i$-th point.

2. The Cost Function (SSR)

We want to minimize the Sum of Squared Residuals ($S$). The residual for a single point is the difference between the actual $y_i$ and the predicted value $\hat{y}_i = \beta_1 x_i + \beta_0$. $$ S(\beta_0, \beta_1) = \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 = \sum_{i=1}^{n} (y_i - (\beta_1 x_i + \beta_0))^2 $$

3. Minimization (Derivation)

To find the minimum, we take the partial derivatives of $S$ with respect to $\beta_0$ and $\beta_1$, and set them to zero.

Derivative w.r.t Intercept ($\beta_0$) $$ \frac{\partial S}{\partial \beta_0} = -2 \sum (y_i - \beta_1 x_i - \beta_0) = 0 $$ Simplifying gives the first normal equation: $$ \sum y_i = \beta_1 \sum x_i + n\beta_0 $$
Derivative w.r.t Slope ($\beta_1$) $$ \frac{\partial S}{\partial \beta_1} = -2 \sum x_i (y_i - \beta_1 x_i - \beta_0) = 0 $$ Simplifying gives the second normal equation: $$ \sum x_i y_i = \beta_1 \sum x_i^2 + \beta_0 \sum x_i $$

4. The Solution

Solving this system of equations yields the closed-form formulas used by the "Snap to OLS" button above:

$$ \hat{\beta_1} = \frac{n \sum x_i y_i - (\sum x_i)(\sum y_i)}{n \sum x_i^2 - (\sum x_i)^2} $$
$$ \hat{\beta_0} = \frac{\sum y_i - \hat{\beta_1} \sum x_i}{n} = \bar{y} - \hat{\beta_1} \bar{x} $$