Calculus

Calculus turns functions into local rates of change and accumulated quantities. In machine learning, the most common use is differential: approximate how a loss changes when inputs, weights, or logits move a small amount.

Defining math

For a scalar function $f$, the derivative at $x$ is

$$f^{\prime }(x)=\underset{h\to 0}{\lim }\frac{f(x+h)-f(x)}{h}.$$

The first-order Taylor approximation is

$$f(x+\Delta x)\approx f(x)+f^{\prime }(x)\Delta x.$$

Multivariable calculus replaces $f^{\prime }(x)$ with gradients, Jacobians and Hessians, and directional derivatives. The same chain rule is what lets backpropagation compose local derivatives through a computational graph.

Executed demo

This snippet compares a central-difference derivative of $f(x)=\sin (x)e^{-0.2x}$ with the analytic derivative at $x=0.7$, then reports the absolute numerical error.

import numpy as np
 
h = 1e-5
f = lambda x: np.sin(x) * np.exp(-0.2*x)
x = 0.7
fd = (f(x+h) - f(x-h)) / (2*h)
true = np.exp(-0.2*x) * (np.cos(x) - 0.2*np.sin(x))
print("central_diff", round(fd, 8))
print("analytic", round(true, 8))
print("abs_error", f"{abs(fd-true):.2e}")

Observed output:

central_diff 0.55291066
analytic 0.55291066
abs_error 7.01e-12

The code estimates the derivative of $f(x)=\sin (x)e^{-0.2x}$ at $x=0.7$ using the symmetric slope between $x-h$ and $x+h$. That finite-difference estimate, 0.55291066, matches the analytic derivative to an absolute error of about $7.01\times 10^{-12}$, so the local tangent slope is being measured accurately.

The dashed tangent in the plot is the geometric version of the number printed by the code. That local slope is exactly the kind of signal gradient descent uses when minimizing a loss.

Caveats

Numerical derivatives depend on step size: too large gives truncation error, too small magnifies floating-point cancellation. Non-smooth points can have subgradients or one-sided derivatives rather than a single ordinary derivative.

References

• MIT OpenCourseWare: 18.01SC Single Variable Calculus
• MIT OpenCourseWare: 18.02SC Multivariable Calculus

Nav

Section — Mathematical Foundations

← Low-Rank Approximation Gradients →