Integrals in probability: the expectation

For a discrete distribution the expectation is a sum: E[X] = Σ x·P(x). For a continuous distribution the sum becomes an integral:

$\mathbb E[X] = \int_{-\infty}^{\infty} x\,p(x)\,dx$

The same idea: every value is weighted by its probability density.

The variance measures the spread: $\text{Var}(X) = \mathbb E[(X-\mu)^2] = \mathbb E[X^2] - \mu^2$

The second form is nearly always easier to compute with.

Uniform on [a, b]: $p(x) = 1/(b-a)$, $\mathbb E[X] = \frac{a+b}{2}$, $\text{Var}(X) = \frac{(b-a)^2}{12}$.

Exponential with parameter λ: $p(x) = \lambda e^{-\lambda x}$ for $x\ge 0$, $\mathbb E[X] = 1/\lambda$, $\text{Var}(X) = 1/\lambda^2$.

Normal: $\mathbb E[X] = \mu$, $\text{Var}(X) = \sigma^2$.

Linearity — the most used property: $\mathbb E[aX + bY] = a\mathbb E[X] + b\mathbb E[Y]$ holds always, even when X and Y are dependent. The variance, on the other hand, is not linear: $\text{Var}(aX) = a^2\text{Var}(X), \qquad \text{Var}(X+Y) = \text{Var}(X)+\text{Var}(Y)+2\,\text{Cov}(X,Y)$ The last term disappears only if X and Y are uncorrelated.

The connection to ML: every loss function is an expectation over the data distribution, $\mathcal L(\theta) = \mathbb E_{(x,y)\sim D}[\ell(f_\theta(x), y)]$. We cannot compute the integral — we estimate it with a sample, which is exactly what a minibatch is. That $\mathbb E$ of the minibatch gradient is the true gradient (unbiased) is the whole basis for SGD working.

import numpy as np
from scipy import integrate

# Exponential with lambda = 2: E[X] = 0.5, Var = 0.25
p = lambda x: 2 * np.exp(-2 * x)
E  = integrate.quad(lambda x: x * p(x), 0, np.inf)[0]
E2 = integrate.quad(lambda x: x**2 * p(x), 0, np.inf)[0]
print(round(E, 4), round(E2 - E**2, 4))      # 0.5 0.25

# Monte Carlo — the same answer without solving the integral
rng = np.random.default_rng(0)
x = rng.exponential(scale=0.5, size=200_000)
print(round(x.mean(), 4), round(x.var(), 4))  # 0.4997 0.2494

# The minibatch gradient is an unbiased estimate of the true one
def true_gradient(data, theta):
    return np.mean([grad(d, theta) for d in data], axis=0)
def minibatch_gradient(data, theta, bs, rng):
    return np.mean([grad(d, theta) for d in rng.choice(data, bs, replace=False)], axis=0)
# E[minibatch_gradient] == true_gradient  → that is why SGD converges