A scatter plot does not come with a line attached. Ordinary least squares chooses the line that minimizes the sum of squared residuals. Each residual is a vertical distance: the observed outcome minus the prediction for that same value of .
From distances to a fitting criterion
In , predictions are and residuals are . Provided varies, the solution is:
Squaring prevents positive and negative errors from cancelling, while giving large deviations more influence. The drawn squares visualize this penalty; some may be clipped by the plot boundary.
Click inside the plot or enter coordinates below. Salary is in arbitrary units; the example is not a real earnings estimate.
- SSR manual / current line
- —
- SSR minimum (OLS)
- —
- Excess above the minimum
- —
Each witness adds one point to the current sample and activates OLS. Reload the sample between the two experiments for a fair comparison. The sample concentrates x between 8 and 12 to make leverage at the edge visible.
- New point leverage hᵢ
- —
- Residual before OLS refit
- —
- Residual after OLS refit
- —
- Slope change Δβ̂₁
- —
Dashed teal line: OLS before the last witness point. Solid line: current fit. The outlined point is the witness; diagnostics compare OLS fits before and after its addition.
- n
- 0
- β̂₀
- —
- β̂₁
- —
- R²
- —
- Residual standard error
- —
Residuals, squares and the table follow the current line. Coefficients with hats, R² and residual standard error refer to the OLS minimum. That minimum changes when the sample changes.
Observations and current-line residuals
| xᵢ | yᵢ | b₀ + b₁xᵢ | eᵢ | eᵢ² |
|---|
Try a line before minimizing
The sliders choose an arbitrary intercept and slope . The line, residuals, squares and table follow those values while the counter recomputes:
Try lowering it by hand, then press Fit with OLS (Minimum). The button moves to the analytical coefficients without rounding them to the displayed values. Excess above the minimum becomes zero. This is not a local minimum found by trial and error: no other straight line can achieve a lower SSR for that same sample. Writing , the reason is:
The minimum is unique when . After fitting, adding observations keeps OLS active; moving either slider switches back to manual mode. The minimum always belongs to the current sample: comparing SSR across different sample sizes is not the same optimization problem.
A large residual is not the same as high influence
Reload the sample before each witness experiment for a fair comparison. Sample education values are concentrated between 8 and 12 years to make the contrast visible:
- Vertical outlier: adds an observation at with a distant salary. It greatly increases error but leaves the OLS slope unchanged; it can still shift the intercept.
- High-leverage point: adds an observation at the plot’s x endpoint furthest from the sample center. It targets a deviation of 15 units from the previous fit, less than the vertical outlier’s deviation in the sample preset. Refitting can substantially change the slope and leave a small final residual because the point has pulled the line towards itself. If the target falls outside the salary range, it is clipped to 0–100 and the panel reports its actual residual.
The panel explicitly distinguishes residuals before and after refitting. A point exactly on the old line would not move it, however extreme its x value: high leverage means potential influence, not guaranteed influence. Leverage is calculated in the augmented sample:
The dashed teal line preserves the previous OLS fit and the outlined point identifies the added observation. Diagnostics compare OLS before and after the addition, even if you subsequently explore another line manually.
What does a good fit mean?
With an intercept, define and . SSR here means the residual sum of squares, not the explained sum:
The panel reports the residual standard error, in outcome units, rather than the standard error of the estimated slope. Two points leave no residual degrees of freedom; constant outcomes make undefined; constant values leave the slope unidentified. Those cases display a dash.
Geometry for fitting, assumptions for inference
Computing the OLS line is an algebraic and geometric operation: normality, homoskedasticity and error independence are not required. A unique solution does require variation in , or a full-rank design matrix in multiple regression.
Probability assumptions justify estimator properties and inferential procedures. In the classical Gauss–Markov formulation, exogeneity and homoskedastic, uncorrelated errors, together with full rank, make OLS the best linear unbiased estimator. Normality is not required, and uncorrelated errors need not be independent. Normality is added for exact classical finite-sample and tests and confidence intervals. Asymptotic inference and robust covariance estimators allow other conditions without changing the line that minimizes SSR.
Methodological conclusion. Add an extreme observation and watch the fitted line move. High describes an in-sample fit: it establishes neither causation nor out-of-sample accuracy and does not solve omitted variable bias. The education and salary example is entirely synthetic.
Reference: NIST, least squares and residual standard deviation.
Further reading: Penn State, influential points and leverage and Marco Taboga, Gauss–Markov theorem.