Evaluating generative models

How do you measure «good images»? There is no answer key per image — only distributions to compare.

FID alone is never enough. A model that has memorised the training images gets an excellent FID. A model that has lost half the distribution can also get a decent FID if what it does produce is sharp.

FID assumes that the feature vectors (pool3 in Inception-v3) are normally distributed and compares the mean and the covariance: $\text{FID} = \|\mu_r - \mu_g\|^2 + \text{Tr}\big(\Sigma_r + \Sigma_g - 2(\Sigma_r\Sigma_g)^{1/2}\big)$

Pitfalls that make published FID numbers hard to compare:

• The sample size: FID is biased downwards with more samples — 10 k and 50 k images give different numbers for the same model.
• Preprocessing: resizing and the interpolation method have a measurable effect.
• The feature space: Inception is trained on ImageNet and fits badly for, say, medical images or faces.

Precision and recall for generative models (Kynkäänniemi et al. 2019) separate the two errors: low precision = ugly or implausible images, low recall = whole modes are missing. That is exactly the split FID lacks.

A memorisation test that should always be run: for every generated image, find the nearest neighbour in the training data (in feature space) and look manually at the hundred closest. A model that reproduces training data is both a quality problem and a copyright problem.

import numpy as np
from scipy import linalg

def fid(feat_real: np.ndarray, feat_gen: np.ndarray) -> float:
    mu_r, mu_g = feat_real.mean(0), feat_gen.mean(0)
    s_r, s_g = np.cov(feat_real, rowvar=False), np.cov(feat_gen, rowvar=False)
    covmean, _ = linalg.sqrtm(s_r @ s_g, disp=False)
    if np.iscomplexobj(covmean):
        covmean = covmean.real
    return float(((mu_r - mu_g) ** 2).sum() + np.trace(s_r + s_g - 2 * covmean))

def nearest_neighbour_check(feat_gen, feat_train, k=1):
    """A memorisation test: how close do the generated images lie to the training data?"""
    d = ((feat_gen[:, None, :] - feat_train[None, :, :]) ** 2).sum(-1)
    nearest = np.sort(d, axis=1)[:, :k].mean(1)
    return {"median_distance": float(np.median(nearest)),
            "suspect": int((nearest < np.percentile(nearest, 1)).sum())}

# Always report: FID (with n), precision, recall, the CLIP score for text→image,
# human pairwise on a sample, and the memorisation check.