Feature Engineering

Feature engineering changes the representation $x\mapsto \varphi (x)$ so a model can express the relevant relationship. It is especially important for linear models, where the model may be simple but the features can encode nonlinearities, interactions, lags, bins, or domain aggregates.

Why representation matters

A simple model can only use what the representation makes visible. If the target is quadratic in a raw input, a straight-line model fails; adding $x^2$ gives the same estimator the right coordinate system. Common transformations each expose a different kind of structure:

Transformation	Example	Structure it exposes
Polynomial	$x\to [x,x^2]$	curvature
Interaction	$x_1,x_2\to x_1{x}_2$	joint effects
Binning	age → age bracket	non-monotone effects
Lag / rolling	$x_t\to x_{t-1}$, moving average	temporal structure
Target-aware encoding	category → mean of $y$	high-cardinality categories (leakage-prone)

The feature map

A model trained on engineered features predicts $\hat{y}=f(\varphi (x))$, where $\varphi$ is a fixed feature map that transforms the raw input $x$ and $f$ is the learned model. Polynomial features, for example, expand a single variable to degree $d$: $\varphi (x)=[x,x^2,\dots ,x^d]$. Feature maps alter both approximation power and risk. More features can reduce bias, but they increase variance and often require regularization. Every learned transformation must be fit inside the training split to avoid data leakage.

Worked example

The code uses a target that is quadratic in one raw feature. It compares a plain linear model with the same model after adding a degree-2 polynomial feature.

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
 
rng = np.random.default_rng(16)
X = rng.uniform(-2, 2, size=(160, 1))
y = 3 * X[:, 0] ** 2 + rng.normal(0, .5, size=160)
Xtr, Xte, ytr, yte = train_test_split(X, y, random_state=16)
for name, pipe in [("linear", LinearRegression()),
                   ("poly2", make_pipeline(PolynomialFeatures(2, include_bias=False), LinearRegression()))]:
    pipe.fit(Xtr, ytr)
    print(name, "test_r2", round(pipe.score(Xte, yte), 3))

Observed output:

linear test_r2 -0.106
poly2 test_r2 0.991

The raw linear model misses the U-shape completely. The quadratic feature makes the structure available without changing the final estimator class.

Caveats

Target encodings, rolling aggregates, and normalization are high-risk leakage points. High-cardinality categorical expansion can create sparse features that overfit rare categories. Feature engineering should be evaluated through the same model selection protocol as model hyperparameters.

References

• scikit-learn User Guide: Preprocessing data
• scikit-learn User Guide: Pipelines

Nav

Section — Classical Machine Learning

← Data Leakage Decision Trees →