Image classification end to end

The project: an image classifier on data you have collected or fetched yourself, with an honest evaluation.

Deliverables:

Part	Requirement
Data	≥ 4 classes, ≥ 150 images per class, a stratified train/val/test split
Baseline	the majority class plus a pretrained model with a frozen body
Model	transfer learning with augmentation
Evaluation	per class, a confusion matrix, bootstrap intervals
Error analysis	30 misclassified images reviewed and categorised
Action	one improvement derived from the error analysis, with before/after

What separates a pass from a strong result is the error analysis. Looking at 30 errors takes twenty minutes and nearly always reveals something structural: a class with ambiguous labels, a background shortcut, or images that should not be there.

import numpy as np, torch
from sklearn.metrics import classification_report, confusion_matrix

@torch.no_grad()
def evaluate(model, loader, class_names, device="cuda"):
    model.eval()
    y, p, conf, paths_all = [], [], [], []
    for xb, yb, paths in loader:
        out = model(xb.to(device)).softmax(-1).cpu()
        p += out.argmax(1).tolist(); conf += out.max(1).values.tolist()
        y += yb.tolist(); paths_all += list(paths)
    y, p, conf = np.array(y), np.array(p), np.array(conf)

    print(classification_report(y, p, target_names=class_names, digits=3))
    print(confusion_matrix(y, p))

    # A bootstrap interval for the accuracy
    rng = np.random.default_rng(0)
    acc = [np.mean(y[i] == p[i]) for i in (rng.integers(0, len(y), len(y)) for _ in range(2000))]
    print("accuracy", round(float(np.mean(y == p)), 3), np.percentile(acc, [2.5, 97.5]).round(3))

    # The 30 worst errors: wrong AND confident — that is where the interesting problems are
    errors = np.where(y != p)[0]
    worst = errors[np.argsort(-conf[errors])][:30]
    return [{"file": paths_all[i], "true": class_names[y[i]], "guessed": class_names[p[i]],
             "confidence": round(float(conf[i]), 3)} for i in worst]

Categorise the errors into four piles once you have reviewed them: a wrong label in the data, a genuinely hard image, a shortcut the model has taken, missing variation in the training data. Each pile has its own action — and only the last is solved with more training.