Regularisation: dropout, weight decay, early stopping

Regularisation = anything that makes the model less inclined to memorise.

Method	What it does	When
Weight decay (L2)	penalises large weights	nearly always, 0.01–0.1
Dropout	zeroes a random share p of the activations during training	fully connected layers, p = 0.1–0.5
Early stopping	stop when the validation loss turns	always — it is free
Data augmentation	more variants of the same example	images, audio; often the largest effect
Label smoothing	the target becomes 0.9 instead of 1.0	classification, reduces overconfidence

The rule: more data beats all regularisation. If you cannot get more data, augment. If you cannot augment, regularise.

import torch, torch.nn as nn

model = nn.Sequential(
    nn.Linear(784, 256), nn.ReLU(), nn.Dropout(0.3),
    nn.Linear(256, 128), nn.ReLU(), nn.Dropout(0.3),
    nn.Linear(128, 10),
)
opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)
criterion = nn.CrossEntropyLoss(label_smoothing=0.05)

best, patience, waited = float("inf"), 5, 0
for epoch in range(100):
    model.train()                      # dropout ON
    for xb, yb in train_loader:
        loss = criterion(model(xb), yb); loss.backward(); opt.step(); opt.zero_grad()
    model.eval()                       # dropout OFF — otherwise you are measuring noise
    with torch.no_grad():
        val = sum(criterion(model(xb), yb).item() for xb, yb in val_loader) / len(val_loader)
    if val < best - 1e-4:
        best, waited = val, 0; torch.save(model.state_dict(), "best.pt")
    else:
        waited += 1
        if waited >= patience:
            print(f"early stop at epoch {epoch}, best val {best:.4f}"); break
model.load_state_dict(torch.load("best.pt"))

The most common mistake: forgetting model.eval() at evaluation. Dropout is then active and the validation loss looks randomly worse — and early stopping fires on the wrong grounds.