Model extraction and membership attacks

A trained model leaks information about its training data. Four attacks, in increasing severity:

Membership sounds harmless — but it is not. If the model was trained on patients with a particular diagnosis, «was this person included?» reveals the diagnosis.

The underlying mechanism is the same for all four: the model is more confident about data it saw during training. That difference can be measured from the outside, and it is enough.

Carlini et al. have shown that large language models can be made to reproduce training data verbatim — including personal data — and that the risk grows with model size and with how many times a text appeared in the corpus.

A membership attack in its simplest form: measure the loss for an example. If it is below a threshold, the example was probably in the training data.

More accurate variants calibrate against a reference model trained without the example, and look at the loss ratio rather than the absolute value. That removes the effect of some examples simply being easier than others.

Differential privacy gives a mathematical guarantee. A mechanism $\mathcal{M}$ is $(\varepsilon,\delta)$-differentially private if, for all datasets $D, D'$ differing in one record and all outcome sets $S$:

$P[\mathcal{M}(D) \in S] \le e^{\varepsilon}\,P[\mathcal{M}(D') \in S] + \delta$

In words: the result must not depend appreciably on whether your data was included. If that is true, no attacker — however clever, with however much side information — can determine whether you were in the data.

DP-SGD implements it in the training:

1. Compute the gradient per example.
2. Clip each individual gradient to norm $C$ — so that a single example has bounded influence.
3. Add Gaussian noise proportional to $C$.
4. Take the step.

The cost is real:

The last row is the ethically most important and the least well known: the noise hits hardest at patterns that exist in only a few examples, and those are often the minority groups.

What $\varepsilon$ means in practice:

Protections in order of what they cost:

The first four should be done in every case. The last one when the data is sensitive enough to justify the cost.

import numpy as np, torch

# A simple membership attack: a loss threshold
def membership_attack(model, members, non_members, loss_fn):
    lm = np.array([float(loss_fn(model, x, y)) for x, y in members])
    ln = np.array([float(loss_fn(model, x, y)) for x, y in non_members])
    from sklearn.metrics import roc_auc_score
    y = np.r_[np.ones(len(lm)), np.zeros(len(ln))]
    score = -np.r_[lm, ln]                     # a lower loss = probably a member
    return {"auc": round(float(roc_auc_score(y, score)), 4),
            "mean_loss_member": round(float(lm.mean()), 4),
            "mean_loss_non_member": round(float(ln.mean()), 4)}
# AUC 0.5 = no leak. AUC 0.75 = a clear leak.

# The calibrated variant: compare against a reference model without the example
def calibrated_attack(model, reference, examples, loss_fn):
    return [float(loss_fn(reference, x, y)) - float(loss_fn(model, x, y))
            for x, y in examples]
# A large positive value → the model is unusually confident → probably a member

# DP-SGD: clip per example, add noise
def dp_sgd_step(model, batch, opt, C=1.0, noise_multiplier=1.1, loss_fn=None):
    summed = [torch.zeros_like(p) for p in model.parameters()]
    for x, y in batch:                              # the gradient PER EXAMPLE
        opt.zero_grad()
        loss_fn(model, x, y).backward()
        grads = [p.grad.detach().clone() for p in model.parameters()]
        norm = torch.sqrt(sum((g ** 2).sum() for g in grads))
        scale = min(1.0, C / float(norm + 1e-12))   # clip individually
        for s, g in zip(summed, grads):
            s += g * scale
    n = len(batch)
    opt.zero_grad()
    for p, s in zip(model.parameters(), summed):
        noise = torch.normal(0.0, noise_multiplier * C, size=s.shape, device=s.device)
        p.grad = (s + noise) / n                    # add the noise, take the mean
    opt.step()

# A memorisation test: can the model reproduce its training data?
def memorisation_test(model, tokenizer, prefixes, references, max_tokens=50):
    exact = 0
    for prefix, reference in zip(prefixes, references):
        ids = tokenizer(prefix, return_tensors="pt").input_ids
        with torch.no_grad():
            out = model.generate(ids, max_new_tokens=max_tokens, do_sample=False)
        generated = tokenizer.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
        exact += int(generated.strip().startswith(reference.strip()[:40]))
    return {"exactly_reproduced": exact, "of": len(prefixes),
            "share": round(exact / max(len(prefixes), 1), 4)}

# Defend the API against model stealing
class Protected:
    def __init__(self, model, max_calls_per_day=1000, return_top=1):
        self.model, self.cap, self.top = model, max_calls_per_day, return_top
        self.count = {}

    def predict(self, user, x):
        n = self.count.get(user, 0)
        if n >= self.cap:
            return {"error": "the daily quota is used up"}
        self.count[user] = n + 1
        p = self.model.predict_proba([x])[0]
        idx = int(np.argmax(p))
        # Do NOT return the whole probability vector — it makes membership attacks easier
        return {"class": idx} if self.top == 1 else {"top": np.argsort(-p)[:self.top].tolist()}