Matrix Multiplication

Matrix multiplication is the operation that lets one linear map follow another. In data science it also computes many dot products at once: scores, projections, attention logits, and neural-network layer outputs are all variations on $XW$.

Defining math

If $A\in {\mathbb{R}}^{m\times n}$ and $B\in {\mathbb{R}}^{n\times p}$, then

$$(AB{)}_{ij}=\sum _{k=1}^n{A}_{ik}{B}_{kj}.$$

The inner dimensions must match because each output entry is a dot product between one row of $A$ and one column of $B$. Multiplication is associative, $(AB)C=A(BC)$, but usually not commutative: $AB\ne BA$. As a composition rule, applying $B$ then $A$ gives $A(Bx)=(AB)x$.

This is why gradients in linear models contain terms like $X^{\top }(p-y)$ and why backpropagation is full of matrix products and transposes. Jacobians and Hessians generalize the same composition idea to derivatives.

Worked example

Take

$$X=\left[\begin{array}{ccc}1& 2& 0\\ 0& 1& 1\end{array}\right],W=\left[\begin{array}{cc}2& -1\\ 0& 1\\ 3& 1\end{array}\right].$$

Each entry of $XW$ is one row of $X$ dotted with one column of $W$. Writing every entry as its row-by-column sum,

$$XW=\left[\begin{array}{cc}1\cdot 2+2\cdot 0+0\cdot 3& 1(-1)+2\cdot 1+0\cdot 1\\ 0\cdot 2+1\cdot 0+1\cdot 3& 0(-1)+1\cdot 1+1\cdot 1\end{array}\right]=\left[\begin{array}{cc}2& 1\\ 3& 2\end{array}\right].$$

The first row, second column is $1(-1)+2(1)+0(1)=1$. Reading $X$ as two examples and $W$ as two coefficient vectors, $XW$ produces two scores for each example.

Caveats

The order of multiplication encodes meaning. $XW$ and $WX$ may both be valid in some dimensions but represent different maps. Large products can also be dominated by conditioning and scale; those issues connect to numerical stability, not to the algebraic definition alone.

References

• MIT OpenCourseWare: 18.06 Linear Algebra

Nav

Section — Mathematical Foundations

← Vectors and Matrices Determinants →