Autoencoders

An autoencoder is trained to recreate its own input — which sounds pointless until you see the bottleneck:

input (784) → encoder → latent (32) → decoder → output (784)
                            ↑
                      the bottleneck

Since all the information has to pass through 32 numbers, the network is forced to compress: keep what is essential, throw the rest away. What remains in the bottleneck is a learnt representation.

The loss is simply the distance between the input and the output — no labelling is needed. It is self-supervised learning in its simplest form.

The connection to PCA: an autoencoder with linear layers and squared loss finds the same subspace as PCA. It is with the non-linearities that it becomes interesting — then it can follow curved manifolds that PCA cannot.

Four uses, in descending order of how often they are actually the right choice:

Use	How	Comment
Denoising	train on (noisy, clean) pairs	works very well
Anomaly detection	a high reconstruction error = an outlier	the most common production use
Dimensionality reduction	use the bottleneck as features	often beaten by simpler methods
Generation	sample in the latent space	requires a VAE — see below

Why an ordinary autoencoder cannot generate. The latent space has no guarantees: between two trained points there can be holes where the decoder produces nonsense. Sample a random point and you usually get rubbish.

A VAE solves that by encoding each input into a distribution instead of a point, and punishing the deviation from a normal distribution (the KL term). That makes the space coherent and samplable — at the price of blurrier reconstructions.

Variants:

Variant	The idea
Denoising	add noise to the input, require a clean output
Sparse	punish the number of active latent units
Contractive	punish the sensitivity to small input changes
VAE	a probabilistic latent, a KL penalty
Masked (MAE)	hide 75 % of the image, recreate it

The last is the modern variant: masked autoencoders are one of the most successful pretraining methods for image models.

Sparse autoencoders in interpretability research are a contemporary application worth knowing about: a wider but sparse autoencoder is trained on a language model's activations, in order to break polysemantic neurons up into monosemantic directions. There the bottleneck is not compression but sparsity.

Anomaly detection in practice: train only on normal examples, measure the reconstruction error, set the threshold at, for instance, the 99th percentile of the training error. Outliers are reconstructed badly because the network never learnt them. The pitfall is that a sufficiently large autoencoder learns to reconstruct everything, outliers included — the bottleneck has to be genuinely narrow.

import torch, torch.nn as nn

class Autoencoder(nn.Module):
    def __init__(self, in_dim=784, latent=32):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(in_dim, 256), nn.ReLU(),
            nn.Linear(256, 64), nn.ReLU(),
            nn.Linear(64, latent),
        )
        self.decoder = nn.Sequential(
            nn.Linear(latent, 64), nn.ReLU(),
            nn.Linear(64, 256), nn.ReLU(),
            nn.Linear(256, in_dim), nn.Sigmoid(),
        )

    def forward(self, x):
        z = self.encoder(x)
        return self.decoder(z), z

# Denoising: noise in, clean out
def train_denoising(model, dataloader, opt, noise=0.3, epochs=10):
    for _ in range(epochs):
        for x, _ in dataloader:
            x = x.flatten(1)
            xb = (x + noise * torch.randn_like(x)).clamp(0, 1)
            out, _ = model(xb)
            loss = nn.functional.mse_loss(out, x)       # compare against the CLEAN image
            opt.zero_grad(); loss.backward(); opt.step()

# Anomaly detection: train only on normal examples
def anomaly_threshold(model, normal_loader, percentile=99):
    errors = []
    model.eval()
    with torch.no_grad():
        for x, _ in normal_loader:
            x = x.flatten(1)
            out, _ = model(x)
            errors += ((out - x) ** 2).mean(dim=1).tolist()
    return float(torch.tensor(errors).quantile(percentile / 100))

def is_anomaly(model, x, threshold):
    with torch.no_grad():
        out, _ = model(x.flatten(1))
        return ((out - x.flatten(1)) ** 2).mean(dim=1) > threshold

# Check the bottleneck: too large a latent → the network learns to copy everything
for latent in (2, 8, 32, 128, 784):
    m = Autoencoder(latent=latent)
    compression = latent / 784
    print(f"latent {latent:>3}: {compression:.1%} of the input"
          + ("   ← no bottleneck at all" if latent >= 784 else ""))

# A linear autoencoder ≈ PCA — check it yourself
import numpy as np
X = np.random.default_rng(0).normal(size=(500, 20)) @ np.random.default_rng(1).normal(size=(20, 20))
Xc = X - X.mean(0)
U, s, Vt = np.linalg.svd(Xc, full_matrices=False)
print("PCA keeps", np.round((s[:5]**2 / (s**2).sum()).sum(), 3), "of the variance with 5 components")

The row with latent=784 is the point of the bottleneck: if the latent dimension is as large as the input, the network can learn the identity map and has learnt nothing at all. For anomaly detection that is particularly treacherous, since a sufficiently large autoencoder reconstructs the outliers perfectly too.