Classical Machine Learning

Classical machine learning is the part of statistical learning where the model class, loss, validation protocol, and diagnostic quantities are usually explicit. This section is organized around the questions a practitioner actually has to answer: what is being predicted or discovered, what objective is optimized, how the model fails, and how the result should be measured.

Knowledge map

Supervised prediction (linear models and tree ensembles) and unsupervised structure are the two trunks; complexity control and evaluation cut across both.

flowchart TD
  SL[Supervised Learning] --> Linear[Linear and Logistic Models]
  SL --> Trees[Trees and Ensembles]
  Linear --> Complexity[Regularization and Model Selection]
  Trees --> Complexity
  USL[Unsupervised Learning] --> DR[Dimensionality Reduction and Clustering]
  Complexity --> Eval[Evaluation, Calibration, Class Imbalance]
  DR --> Eval

Reading path

Start with supervised prediction, then complexity control, tree ensembles, unsupervised structure, and evaluation.

1. Supervised Learning: the empirical-risk framing for learning a map from features to targets.
2. Regression: continuous-target prediction and its losses.
3. Classification: discrete-label prediction through argmax or thresholds.
4. Linear Models: weighted sums of features, including least squares.
5. Logistic Regression: a linear log-odds model for class probabilities.
6. Support Vector Machines: maximum-margin classification with hinge loss and kernels.
7. Regularization: penalties that shrink unstable fits toward simpler functions.
8. Bias-Variance Trade-Off: underfitting versus sensitivity to the training sample.
9. Model Selection: choosing families and hyperparameters without spending the test set.
10. Data Leakage: contamination that makes validation estimates unrealistically good.
11. Feature Engineering: changing the representation so simple models can express the structure.
12. Decision Trees: recursive partitions with impurity or variance-reduction splits.
13. Random Forests: bagged, feature-randomized trees that reduce variance.
14. Gradient Boosting: additive trees fit stage by stage to negative gradients.
15. Interpretability: connecting model behavior back to features and examples.
16. Unsupervised Learning: learning structure without target labels.
17. Dimensionality Reduction: lower-dimensional representations for compression and denoising.
18. PCA: the linear projection that maximizes retained variance.
19. Clustering: partitioning by distance, density, or probabilistic structure.
20. Anomaly Detection: flagging observations unlikely under the fitted normality.
21. Evaluation Metrics: task-specific functions that turn predictions into comparable numbers.
22. Calibration: whether predicted probabilities match observed frequencies.
23. Class Imbalance: rare-class settings where accuracy and default thresholds mislead.

Connections

• Probability and Statistics supplies the estimation and testing these methods rely on.
• Deep Learning generalizes these ideas to learned representations and larger models.

Nav

Learning path — Foundations

← Probability and Statistics Supervised Learning →