PyTorch

PyTorch is a tensor and automatic-differentiation framework for building neural networks in ordinary Python. Its defining training workflow is: run tensor operations, record a dynamic computation graph, call backward(), then update parameters with an optimizer. Compared with TensorFlow and Keras, PyTorch exposes the lower-level imperative loop more directly.

Autograd and the computation graph

For parameters $\theta$ and scalar loss $L$, PyTorch records operations that produce

$$L=f_{\theta }(x)$$

and reverse-mode autograd computes

$${\nabla }_{\theta }L=\left(\frac{\partial L}{\partial {\theta }_1},\dots ,\frac{\partial L}{\partial {\theta }_p}\right).$$

The common loop is

$$\text{zero\_grad}\to \text{forward}\to \text{loss}\to \text{backward}\to \text{step}.$$

That makes backpropagation explicit while still delegating derivative bookkeeping to autograd and loss-specific kernels to modules such as loss functions.

Worked example

This snippet builds a one-parameter PyTorch computation, runs backpropagation, applies an SGD step, and prints the autograd function type.

import torch
 
torch.manual_seed(14)
w = torch.tensor([1.5], requires_grad=True)
x = torch.tensor([2.0])
y = (w * x).pow(2) if w.item() > 1 else w * x
y.backward()
with torch.no_grad():
    w -= 0.1 * w.grad
print("grad", round(w.grad.item(), 3))
print("updated_w", round(w.item(), 3))
print("grad_fn", type(y.grad_fn).__name__)

Observed output:

grad 12.0
updated_w 0.3
grad_fn PowBackward0

The branch is ordinary Python, but the operations actually executed produce a differentiable graph. The update is wrapped in no_grad() so the optimizer step itself is not recorded.

Caveats

Dynamic graphs are easy to debug but easy to mutate accidentally. In-place tensor operations can invalidate saved backward values. model.eval() changes module behavior for dropout and batch norm, while torch.no_grad() changes gradient recording; confusing the two creates subtle evaluation bugs.

References

• PyTorch documentation: Autograd mechanics
• PyTorch tutorial: Automatic Differentiation with torch.autograd

Nav

Section — Deep Learning

← Generative Adversarial Networks TensorFlow and Keras →