Visualising what a CNN sees

What has the network learnt? Three windows in:

1. The first layer's filters can be drawn as small images: they become edge detectors, colour transitions, small textures — surprisingly similar to the biological visual system.
2. Activation maps: feed in an image, look at the output of a layer. Bright patches = «this filter responded here».
3. Saliency / Grad-CAM: the gradient of the class score with respect to the input (or the last convolutional layer) shows which pixels influenced the decision. If «husky» is decided by the snow in the background you see it here — and you have found a shortcut in the data.

A reservation: beautiful images prove nothing. A visualisation is a hypothesis about what the network is doing — test it with interventions (mask out the area, see whether the decision changes).

import torch, matplotlib.pyplot as plt

# 1) the first layer's filters
w = m.f[0].weight.detach()                    # (32, 3, 3, 3)
fig, ax = plt.subplots(4, 8, figsize=(8, 4))
for i, a in enumerate(ax.flat):
    f = w[i]; f = (f - f.min()) / (f.max() - f.min())
    a.imshow(f.permute(1, 2, 0)); a.axis("off")
plt.savefig("filters.png")

# 2) activations via a hook
act = {}
m.f[3].register_forward_hook(lambda mod, i, o: act.__setitem__("conv2", o.detach()))
m(x[None]); print(act["conv2"].shape)         # (1, 64, 16, 16)

# 3) saliency: |dScore/dPixel|
x.requires_grad_(True)
m(x[None])[0, cls].backward()
sal = x.grad.abs().max(0).values              # (H, W)
plt.imsave("saliency.png", sal.numpy(), cmap="hot")

# an intervention: mask the most salient area and see whether the probability falls