Linear maps

A matrix is not just a grid of numbers — it is a function that moves the space. Multiply every point by the matrix and the space is rotated, scaled, reflected or sheared. Straight lines stay straight and the origin stands still.

The determinant is the area scale: det = 2 means that all areas double, det = 0 that the space collapses to a line (the information cannot be recovered — the matrix has no inverse), a negative det means that the orientation is reversed.

Eigenvectors are the directions the map does not turn — only scales: $A v = \lambda v$ where $\lambda$ is the eigenvalue, the scale factor in that direction.

For $A = \begin{pmatrix}2 & 0\\ 0 & 3\end{pmatrix}$, $e_1$ and $e_2$ are eigenvectors with eigenvalues 2 and 3: the map stretches the x-axis twofold and the y-axis threefold.

Why it matters in deep learning:

• Every layer in a network is a linear map (plus a non-linearity). Depth = many maps in sequence.
• If every layer's matrix has eigenvalues > 1 the signal grows exponentially with the depth (exploding activations/gradients); if they are < 1 it shrinks (vanishing). Initialisation strategies are precisely about keeping them near 1.
• SVD writes every matrix as rotation → scaling → rotation, and the singular values are the scale factors. That is the basis of low-rank approximation and LoRA.
• A determinant of zero = rank deficiency = information loss — exactly what LoRA exploits when ΔW is deliberately given a low rank.

import numpy as np

A = np.array([[2.0, 1.0], [0.0, 3.0]])
corners = np.array([[0,0], [1,0], [1,1], [0,1]], dtype=float)
print((A @ corners.T).T)          # the unit square's corners after the map
print(np.linalg.det(A))           # 6.0  → areas become 6× larger

values, vectors = np.linalg.eig(A)
print(values.round(3))            # [2. 3.]
print(vectors.round(3))           # the columns are the directions that are only scaled

B = np.array([[1.0, 2.0], [2.0, 4.0]])
print(np.linalg.det(B), np.linalg.matrix_rank(B))   # 0.0 1  ← collapses to a line