Gradients

The gradient of a scalar function is the vector of partial derivatives. It gives the local direction of steepest increase under the Euclidean norm, so its negative is the basic descent direction used by gradient descent and neural-network backpropagation.

Defining math

For $f:{\mathbb{R}}^d\to \mathbb{R}$,

$$\nabla f(x)=\left[\begin{array}{c}\frac{\partial f}{\partial x_1}\\ ⋮\\ \frac{\partial f}{\partial x_d}\end{array}\right].$$

The first-order approximation is

$$f(x+\Delta )\approx f(x)+\nabla f(x{)}^{\top }\Delta .$$

For squared error on one linear prediction, $L(w)=(x^{\top }w-y{)}^2$, the gradient is

$${\nabla }_{w}L=2(x^{\top }w-y)x.$$

This formula is the small local object that becomes batched matrix expressions in matrix multiplication and stochastic estimates in stochastic gradient descent.

Worked example

Take weights $w=(1.5,-0.5)$, input $x=(2,-1)$, and target $y=4$. The prediction is $x^{\top }w=2(1.5)+(-1)(-0.5)=3.5$, so the residual is $x^{\top }w-y=-0.5$ and the loss is $(-0.5{)}^2=0.25$. Substituting into ${\nabla }_{w}L=2(x^{\top }w-y)x$,

$${\nabla }_{w}L=2(-0.5)\left[\begin{array}{c}2\\ -1\end{array}\right]=\left[\begin{array}{c}-2\\ 1\end{array}\right].$$

A centered finite-difference check, $(L(w+ϵe_j)-L(w-ϵe_j))/2ϵ$ for each coordinate $j$, reproduces the same vector — the standard sanity test when implementing custom derivatives.

Caveats

Gradients are local. A small gradient can mean a minimum, a saddle point, saturation, or bad scaling. For curvature and second-order checks, use Jacobians and Hessians rather than gradient magnitude alone.

References

• MIT OpenCourseWare: 18.02SC Multivariable Calculus
• Boyd and Vandenberghe, Convex Optimization

Nav

Section — Mathematical Foundations

← Calculus Jacobians and Hessians →