Feature Extraction

Feature extraction converts an image representation into measurements a downstream model can compare or classify. Classical features are hand-designed descriptors; learned features are intermediate activations or embeddings trained by CNN architectures, vision transformers, or self-supervised visual learning.

Features as a mapping

A feature extractor is a map

$$\varphi :{\mathbb{R}}^{C\times H\times W}\to {\mathbb{R}}^d.$$

The input has $C$ channels, height $H$, and width $W$; the output is a $d$-dimensional feature vector. The map $\varphi$ can be a fixed descriptor, a CNN trunk, a vision-transformer encoder, or a task-specific embedding model.

For classical gradient features, $\varphi$ may summarize local derivative magnitudes and orientations. For learned retrieval, $\varphi (x)$ is usually normalized and compared by cosine similarity:

$$\mathrm{sim}(x,q)=\frac{\varphi (x{)}^{\top }\varphi (q)}{\Vert \varphi (x){\Vert }_2\Vert \varphi (q){\Vert }_2}.$$

Here $x$ is a candidate image and $q$ is the query image. Cosine similarity compares the angle between feature vectors, so two images can be close even if their raw pixel values differ.

Worked example

This small example uses gradient magnitude as a hand-designed feature. It shows the principle, not a modern production extractor: pixels are converted into edge-strength measurements and then summarized into histogram counts.

import numpy as np
 
img = np.array([[0,0,1,1],[0,0,1,1],[2,2,3,3],[2,2,3,3]], float)
gy, gx = np.gradient(img)
mag = np.sqrt(gx**2 + gy**2)
hist, _ = np.histogram(mag, bins=[0, .25, .75, 1.5])
print("gradient_magnitude")
print(np.round(mag, 2))
print("hist_bins", [0, .25, .75, 1.5], "counts", hist.tolist())
print("mean_feature", round(float(mag.mean()), 3))

Observed output:

gradient_magnitude
[[0.   0.5  0.5  0.  ]
 [1.   1.12 1.12 1.  ]
 [1.   1.12 1.12 1.  ]
 [0.   0.5  0.5  0.  ]]
hist_bins [0, 0.25, 0.75, 1.5] counts [4, 4, 8]
mean_feature 0.655

This tiny descriptor captures edge strength but loses exact spatial layout. That tradeoff is acceptable for constrained inspection, but poor for object detection where localization matters.

Caveats

Features can encode shortcuts: lighting, scanner type, crop border, watermark, or background. Always inspect nearest neighbors in content-based image retrieval and evaluate features on domain slices rather than assuming representation quality transfers.

References

• Computer Vision: Algorithms and Applications, 2nd ed.
• Torchvision feature extraction documentation

Nav

Section — Computer Vision

← Classical Image Processing Data Augmentation →