Linear Models

A linear model predicts through a score $\eta ={\beta }_0+x^{\top }\beta$. What changes across regression, logistic regression, and linear margin classifiers is the link from $\eta$ to the output and the loss used for fitting.

Defining math

For squared-error regression,

$${\hat{y}}_i={\beta }_0+x_i^{\top }\beta ,\underset{{\beta }_0,\beta }{\min }\sum _i(y_i-{\hat{y}}_i{)}^2,$$

where $x_i$ is the feature vector for example $i$, ${\beta }_0$ is the intercept, $\beta$ is the vector of coefficients (one per feature), $y_i$ is the observed target, and ${\hat{y}}_i$ is the prediction. For binary logistic regression, the same linear score $\eta ={\beta }_0+x^{\top }\beta$ is passed through the sigmoid $\sigma$ to produce a probability:

$$P(Y=1\mid x)=\sigma (\eta )=\frac{1}{1+e^{-\eta }},$$

fitted with cross-entropy loss instead of squared error. With ridge regularization, many linear objectives add a penalty

$$\lambda \Vert \beta {\Vert }_2^2,$$

where $\Vert \beta {\Vert }_2^2={\sum }_j{\beta }_j^2$ is the squared coefficient norm and $\lambda \ge 0$ is the penalty strength that trades fit against shrinkage. Lasso regularization instead uses an ${\ell }_1$ penalty and can set coefficients exactly to zero, which turns coefficient shrinkage into feature pruning; see regularization for the ridge-lasso comparison. The geometry is simple: a one-unit feature change moves the score by the corresponding coefficient, holding other features fixed. That interpretability is only meaningful when preprocessing, interactions, and scaling are explicit.

Intuition

Linear models are strong baselines because they ask whether the representation already contains the answer. If a linear model works well, the features encode the signal in an almost additive way. If it fails with systematic residuals or segment errors, the next move is often feature engineering, not immediately a larger model.

Worked example

Fit a one-feature line $\hat{y}={\beta }_0+{\beta }_{1}x$ to five points $(1,1),(2,3),(3,2),(4,5),(5,4)$ by least squares. With $\bar{x}=3$ and $\bar{y}=3$, the slope and intercept are

$${\beta }_1=\frac{{\sum }_i(x_i-\bar{x})(y_i-\bar{y})}{{\sum }_i(x_i-\bar{x}{)}^2}=\frac{8}{10}=0.8,{\beta }_0=\bar{y}-{\beta }_1\bar{x}=3-0.8(3)=\mathrm{0.6.}$$

So $\hat{y}=0.6+0.8x$. The residuals $y_i-{\hat{y}}_i$ are $-0.4,0.8,-1.0,1.2,-0.6$, and their squares sum to $3.6$. No other line achieves a smaller squared total — that is exactly the quantity least squares minimizes:

The slope $0.8$ is the model’s whole story: a one-unit increase in $x$ raises the prediction by $0.8$, regardless of where on the line you stand. That constant, additive effect is what makes linear models interpretable — and also what fails when the true effect depends on $x$ or on other features, the case for feature engineering or a nonlinear method.

Caveats

Coefficient magnitude is not comparable across differently scaled features. Correlated predictors make individual coefficient stories fragile even when predictions are stable, and lasso may prune one correlated feature while keeping another. Linear additivity also hides interactions: if risk rises only when two conditions co-occur, the model needs interaction features or a nonlinear method such as decision trees.

References

• scikit-learn User Guide: Linear Models
• An Introduction to Statistical Learning

Nav

Section — Classical Machine Learning

← Classification Logistic Regression →