Matrices and matrix multiplication

A matrix is a table of numbers — rows and columns. Multiply it by a vector and you get a new vector: every row of the matrix is dotted with the vector.

[2 0]   [3]   [2·3 + 0·1]   [6]
[1 1] · [1] = [1·3 + 1·1] = [4]

That is exactly what a layer in a neural network does: the weights are the matrix, the input is the vector, the output is the new vector. A network = matrix multiplication, bend, matrix multiplication, bend …

A is m×n (m rows, n columns), B is n×p. Then AB is an m×p matrix with (AB)ᵢⱼ = Σₖ Aᵢₖ Bₖⱼ — row i of A times column j of B.

The dimension rule: the inner numbers must match (n = n), the outer ones give the result (m×p). A 3×4 matrix times a 4×2 gives 3×2. A 3×4 times a 3×4 does not work.

Matrix multiplication is not commutative: AB ≠ BA in general. The identity matrix I (ones on the diagonal) is neutral: AI = IA = A. The transpose Aᵀ swaps rows for columns.

A batch of 32 examples of 784 pixels is a 32×784 matrix; a layer of 128 neurons has a 784×128 weight matrix; the product is 32×128 — 128 numbers per example.