Loss functions

A loss measures how wrong the model is — a number the training minimises.

• Regression (predict a number): MSE, the mean squared error. Large errors are punished hard (the square).
• Classification (predict a probability): cross-entropy, −log(the probability the model gave the correct class). If the model gives 0.9 to the right class: loss 0.1. If it gives 0.01: loss 4.6. Being confident and wrong costs enormously — exactly as it should.

Why not MSE on classes? With a sigmoid or softmax the gradient becomes nearly zero when the model is confident and wrong — it does not learn from its worst mistakes. Cross-entropy gives a strong gradient exactly there.

import numpy as np

def mse(y, yhat):
    return np.mean((y - yhat) ** 2)

def cross_entropy(p_correct):       # p_correct: the model's probability for the right class, per example
    return -np.mean(np.log(np.clip(p_correct, 1e-12, 1)))

print(mse(np.array([3.0, 5.0]), np.array([2.5, 6.0])))      # 0.625
print(cross_entropy(np.array([0.9, 0.9, 0.9])))            # 0.105
print(cross_entropy(np.array([0.9, 0.9, 0.01])))           # 1.61 — one confident mistake dominates

In PyTorch: nn.MSELoss(), nn.CrossEntropyLoss() (takes logits and does the softmax + log internally — more numerically stable).

Suppose the model gives $p_\theta(y\mid x)$. Maximum likelihood picks the $\theta$ that maximises $\prod_i p_\theta(y_i\mid x_i)$, that is, minimises $-\sum_i \log p_\theta(y_i\mid x_i)$ — which is cross-entropy.

For regression under the assumption $y = f_\theta(x) + \varepsilon$, $\varepsilon\sim\mathcal N(0,\sigma^2)$: $-\log p = \frac{(y - f_\theta(x))^2}{2\sigma^2} + \text{const}$ — minimising it gives MSE. So both are the same principle with different noise assumptions.

The gradient for sigmoid + cross-entropy: $\partial L/\partial z = \hat p - y$ — linear in the error. For sigmoid + MSE: $(\hat p - y)\,\hat p(1-\hat p)$ — the factor $\hat p(1-\hat p)\to 0$ as $\hat p\to 0$ or $1$: the gradient dies exactly when the model is confident and wrong.