Skip to content
AI-grafen
EUniversityStatistics and probability· about 60 min· evolving, reviewed regularly· verified 2026-10-22· EN

Confidence intervals

Be able to compute and interpret confidence intervals for means and proportions.

See the full experiment: from hypothesis to decision

Intuition

A 95 % confidence interval is an interval computed so that, if you repeated the whole sampling procedure many times, 95 % of the intervals would cover the true value. It is not «a 95 % probability that the truth lies here» — the truth is fixed, the interval is what is random.

For a mean: x̄ ± 1.96·s/√n (large n). For a proportion: p̂ ± 1.96·√(p̂(1−p̂)/n). At a small n or proportions near 0/1: the Wilson interval.

In ML, report test metrics with intervals. If two models are tested on the same cases, calculate an interval for the paired difference; overlap between separate intervals does not settle the question. Bootstrap can estimate uncertainty for metrics without a simple interval formula.

Formal

Example: two versions of an AI tutor. In a randomized experiment, 80 of 200 students finish a lesson with the old version and 96 of 200 with the new one. The rates are 40% and 48%; the difference is +8 percentage points.

For two independent groups, a simple approximation is Δ^±1.96p^0(1−p^0)/n0+p^1(1−p^1)/n1\hat\Delta \pm 1.96\sqrt{\hat p_0(1-\hat p_0)/n_0 + \hat p_1(1-\hat p_1)/n_1}. The interval here is roughly −2 to +18 percentage points. It contains zero: the data do not yet give clear support for switching. It also allows for an improvement that would be useful. Gather more observations using a plan set before the next analysis.

Intervals for each group's rate do not answer the question about the difference. Calculate an interval for the difference. If the same students or test cases get both versions, observations are paired and the analysis must preserve those pairs, for example with a paired bootstrap.

Code

import numpy as np
from scipy import stats

# Mean of eight measurements: a t interval when spread is estimated.
errors = np.array([3.1, 2.4, 4.0, 3.3, 2.9, 3.8, 3.5, 2.7])
mean = errors.mean()
se = errors.std(ddof=1) / np.sqrt(len(errors))
lo, hi = stats.t.interval(0.95, len(errors) - 1, loc=mean, scale=se)
print(round(mean, 2), round(lo, 2), round(hi, 2))  # 3.21 2.76 3.67

# One proportion: Wilson performs better near 0/1.
def wilson(k, n, z=1.96):
    p = k / n
    d = 1 + z**2 / n
    center = (p + z**2 / (2 * n)) / d
    half = z * np.sqrt(p * (1 - p) / n + z**2 / (4 * n**2)) / d
    return center - half, center + half

print(tuple(float(round(x, 2)) for x in wilson(87, 100)))  # (0.79, 0.92)

# Difference between two independent proportions in the school's trial.
old, new, n = 80 / 200, 96 / 200, 200
diff = new - old
diff_se = np.sqrt(old * (1 - old) / n + new * (1 - new) / n)
print(tuple(float(round(x * 100, 1)) for x in
            (diff - 1.96 * diff_se, diff + 1.96 * diff_se)))  # (-1.7, 17.7)

# Bootstrap a statistic without a simple interval formula: the median.
rng = np.random.default_rng(0)
medians = [np.median(rng.choice(errors, len(errors), replace=True))
           for _ in range(2000)]
print(np.percentile(medians, [2.5, 97.5]))

Mastery means

  • Computes a confidence interval for a mean and for a proportion
  • Interprets the interval correctly (frequentist)
  • Uses the bootstrap for arbitrary metrics

Prerequisites

Sign in to do the exercises and build your mastery up.

Sources

All the sources and licences