Supervised Learning

Supervised learning estimates a function $f:\mathcal{X}\to \mathcal{Y}$ from labeled examples $(x_i,y_i)$. The target may be continuous, as in regression, or discrete, as in classification; the shared contract is that future examples are judged against labels drawn from the same deployment problem.

What a supervised model learns

A supervised model is not learning labels in the abstract. It is learning a reusable rule that maps information available at prediction time to a decision-relevant output. The strongest mental check is temporal: would every feature in $x$ be known before $y$ happens? If not, data leakage can make empirical risk look low while deployment risk is high.

The choice of loss decides what the model estimates:

Loss $L$	What it estimates	Used by
squared error	the conditional mean $\mathbb{E}[Y\mid X]$	regression
cross-entropy	the conditional probability $P(Y\mid X)$	logistic regression
hinge	a maximum-margin boundary	support vector machines

The learning objective

The statistical target is usually the risk

$$R(f)=\mathbb{E}[L(Y,f(X))],$$

where $L$ is the task loss and the expectation is over the true distribution of $(X,Y)$. Because that distribution is unknown, training minimizes empirical risk on the sample, often with a complexity penalty $\Omega$:

$$\hat{f}=\arg \underset{f\in \mathcal{F}}{\min }\frac{1}{n}\sum _{i=1}^{n}L(y_i,f(x_i))+\lambda \Omega (f),$$

where $\mathcal{F}$ is the model class searched, $\lambda \ge 0$ the penalty strength, and $n$ the number of training examples. The validation split belongs to the objective in practice because model selection chooses $\mathcal{F}$, $\lambda$, preprocessing, and thresholds.

Worked example

This snippet trains a linear regression model with a train-test split and reports test $R^2$ plus the learned feature coefficients.

from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
import numpy as np
 
X, y = make_regression(n_samples=80, n_features=3, noise=10, random_state=7)
Xtr, Xte, ytr, yte = train_test_split(X, y, random_state=7)
model = LinearRegression().fit(Xtr, ytr)
print("test_r2", round(model.score(Xte, yte), 3))
print("coef", np.round(model.coef_, 2))

Observed output:

test_r2 0.956
coef [46.94 31.07 51.12]

The fitted rule explains most held-out variance on this synthetic linear problem. The coefficients are the learned contribution of each feature under a squared-error objective.

Caveats

IID validation is a modelling assumption, not a default truth. Time, user, household, patient, or document-family dependence requires split rules that match deployment. Also, the fitted $\hat{f}$ is only as meaningful as the label: noisy labels increase irreducible error and can make more flexible models appear useful until they memorize annotation artifacts.

References

• scikit-learn User Guide: supervised learning
• An Introduction to Statistical Learning

Nav

Section — Classical Machine Learning

Regression →

Learning path — Foundations

← Classical Machine Learning Evaluation Metrics →