Jacobians and Hessians

Jacobians organize first derivatives of vector-valued functions. Hessians organize second derivatives of scalar functions. They are the matrix form of local change, so they connect calculus to optimization, curvature, and chain-rule computation.

Defining math

For $f:{\mathbb{R}}^n\to {\mathbb{R}}^m$, the Jacobian is

$$J_f(x{)}_{ij}=\frac{\partial f_i}{\partial x_j}.$$

For $g:{\mathbb{R}}^n\to \mathbb{R}$, the Hessian is

$${\nabla }^{2}g(x{)}_{ij}=\frac{{\partial }^{2}g}{\partial x_i\partial x_j}.$$

The Jacobian composes through matrix multiplication:

$$J_{f\circ h}(x)=J_f(h(x))J_h(x).$$

The Hessian describes local curvature. In unconstrained twice-differentiable convex optimization, ${\nabla }^{2}g(x)⪰0$ everywhere is a curvature certificate.

Worked example

For

$$f(x_1,x_2)=\left[\begin{array}{c}{x}_1^2+x_2\\ x_1{x}_2\end{array}\right],$$

the Jacobian is

$$J_f(x)=\left[\begin{array}{cc}2x_1& 1\\ x_2& x_1\end{array}\right].$$

At $(1,2)$ this becomes

$$J_f(1,2)=\left[\begin{array}{cc}2& 1\\ 2& 1\end{array}\right].$$

For $g(x_1,x_2)=x_1^3+x_2^2$, the Hessian is

$${\nabla }^{2}g(x)=\left[\begin{array}{cc}6x_1& 0\\ 0& 2\end{array}\right],$$

so ${\nabla }^{2}g(1,2)=\left[\begin{array}{cc}6& 0\\ 0& 2\end{array}\right]$. Its eigenvalues are 6 and 2 because the matrix is diagonal. Both are positive, so the surface has locally positive curvature in both coordinate directions at that point.

Caveats

Full Jacobians and Hessians can be too large to materialize. Modern autodiff often computes Jacobian-vector or vector-Jacobian products instead, which is the practical form used by backpropagation.

References

• MIT OpenCourseWare: 18.02SC Multivariable Calculus
• Boyd and Vandenberghe, Convex Optimization

Nav

Section — Mathematical Foundations

← Gradients Optimization →