Samples and uncertainty

You measure the accuracy on 100 test examples: 87 %. Had you chosen 100 other examples you would have got 84 % or 90 %. The figure has uncertainty because the test is a sample of all the possible examples.

The standard error (SE) measures that uncertainty: for a proportion p, SE = √(p(1−p)/n). With n = 100 and p = 0.87: SE ≈ 0.034 — that is, ±3.4 percentage points. With n = 10 000: ±0.3.

The uncertainty shrinks with √n: four times more data gives only half the error. Two models at 87 % and 89 % on 100 examples are not distinguishable.

import numpy as np
rng = np.random.default_rng(0)
true_acc = 0.87

for n in (30, 100, 1000, 10_000):
    estimates = rng.binomial(n, true_acc, size=5000) / n
    print(n, round(estimates.std(), 4), round(np.sqrt(true_acc * (1 - true_acc) / n), 4))
# 30     0.0614  0.0614
# 100    0.0336  0.0336
# 1000   0.0106  0.0106
# 10000  0.0034  0.0034

The simulation and the formula agree. The bootstrap gives the same thing without a formula: draw n examples with replacement from the test set 1 000 times, compute the metric each time, look at the spread — it works for any metric at all (F1, BLEU …).