Singular Value Decomposition

Singular value decomposition writes any real matrix as orthogonal input directions, nonnegative gains, and orthogonal output directions. Unlike eigenvalues and eigenvectors, it applies to rectangular matrices and does not require the matrix to preserve a single space.

Defining math

For $A\in {\mathbb{R}}^{m\times n}$, the compact SVD is

$$A=U_r{\Sigma }_r{V}_r^{\top }=\sum _{i=1}^r{\sigma }_i{u}_i{v}_i^{\top },$$

where $r=\mathrm{rank}(A)$, $U_r^{\top }{U}_r=I$, $V_r^{\top }{V}_r=I$, and ${\sigma }_1\ge \cdots \ge {\sigma }_r>0$. The right singular vectors $v_i$ are orthogonal input directions; $A{v}_i={\sigma }_i{u}_i$ sends them to orthogonal output directions. The nonzero ${\sigma }_i^2$ are eigenvalues of $A^{\top }A$, which is why SVD connects directly to matrix decompositions and PCA.

The truncated SVD

$$A_k=\sum _{i=1}^k{\sigma }_i{u}_i{v}_i^{\top }$$

is the best low-rank approximation in Frobenius norm:

$$\Vert A-A_k{\Vert }_F=\sqrt{\sum _{i=k+1}^r{\sigma }_i^2}.$$

This Eckart-Young result is the reason SVD is a clean mathematical baseline for compression, denoising, latent semantic analysis, PCA, classical SVD recommenders, and truncated SVD. It is also the baseline for understanding why sparse recommender pages distinguish ordinary SVD on sparse utility matrices from learned matrix factorization.

Geometrically, $V^{\top }$ chooses orthogonal input coordinates, $\Sigma$ stretches them by singular values, and $U$ rotates the stretched axes into the output space. Truncating the SVD keeps the longest axes first, which is why the discarded singular values determine the low-rank approximation error.

Executed demo

This snippet decomposes a matrix with SVD, verifies exact reconstruction, and compares the rank-1 error with the discarded singular-value tail.

import numpy as np
 
A = np.array([[3., 1., 1.], [-1., 3., 1.], [0., 2., 4.], [2., 0., 2.]])
U, s, Vt = np.linalg.svd(A, full_matrices=False)
Ahat1 = (U[:, :1] * s[:1]) @ Vt[:1]
print("singular_values", np.round(s, 4))
print("reconstruction_error", round(np.linalg.norm(A - (U*s)@Vt), 12))
print("rank1_fro_error", round(np.linalg.norm(A - Ahat1, "fro"), 4))
print("tail_singular_fro", round(np.sqrt(np.sum(s[1:]**2)), 4))

Observed output:

singular_values [5.6569 3.7417 2.    ]
reconstruction_error 0.0
rank1_fro_error 4.2426
tail_singular_fro 4.2426

The exact reconstruction error is numerically zero, and the rank-1 truncation error equals the Frobenius norm of the discarded singular values. Numerically tiny singular values should be interpreted relative to scale, as in rank, not by exact equality to zero.

Connections

Page	How it uses SVD
PCA	Applies SVD to centered data; right singular vectors become principal axes and squared singular values determine explained variance.
Low-rank approximation	Uses the Eckart-Young theorem to quantify the best rank-$k$ reconstruction error.
Classical SVD	Applies SVD to a complete dense matrix in recommender examples.
Truncated SVD	Computes only leading singular components for compact representations.
SVD versus matrix factorization	Contrasts decomposing a complete matrix with learning factors from sparse observed entries.

References

• NumPy documentation: numpy.linalg.svd
• MIT OpenCourseWare: 18.06 Linear Algebra

Nav

Section — Mathematical Foundations

← Matrix Decompositions Low-Rank Approximation →