Bias and fairness in models

A model with 95 % accuracy overall can have 98 % for one group and 74 % for another. The average hides it.

Where the skew comes from:

The first step is always the same: break every measure down by group and look. What you do not measure you cannot see.

Three common fairness measures:

• Demographic parity: $P(\hat Y = 1 \mid G=a) = P(\hat Y=1\mid G=b)$ — the same share of positive decisions in every group.
• Equal opportunity: $P(\hat Y=1\mid Y=1, G=a) = P(\hat Y=1\mid Y=1, G=b)$ — the same recall among those who actually satisfy the criterion.
• Calibration: among those given the score $s$, the share with $Y=1$ should be $s$ in every group.

The impossibility result (Kleinberg et al. 2016; Chouldechova 2017): if the base rates differ between the groups, calibration and equal error rates cannot be satisfied at the same time, except in trivial cases. There is therefore no measure that is «the fair one» — the choice is normative and has to be justified in its context.

So the practical minimum routine is: measure per group, state which measure you prioritise and why, and let a human being decide in borderline cases.

import numpy as np

def per_group(y, pred, group):
    for g in np.unique(group):
        m = group == g
        tp = ((pred[m] == 1) & (y[m] == 1)).sum(); fp = ((pred[m] == 1) & (y[m] == 0)).sum()
        fn = ((pred[m] == 0) & (y[m] == 1)).sum()
        print(f"{g}: n={m.sum():4d}  share positive={pred[m].mean():.2f}  "
              f"recall={tp / max(tp + fn, 1):.2f}  precision={tp / max(tp + fp, 1):.2f}")

per_group(y_test, model.predict(X_test), group_test)
# A: n= 800  share positive=0.42  recall=0.91  precision=0.88
# B: n= 120  share positive=0.19  recall=0.63  precision=0.71   ← a large difference