Scaling laws

Scaling laws say that the loss falls predictably as a power law in three quantities: the number of parameters $N$, the number of training tokens $D$, and the computation $C$.

$L(N) \approx L_\infty + \left(\frac{N_c}{N}\right)^{\alpha}$

On a log-log scale it becomes a straight line, and that is the remarkable part: you can measure on small models and extrapolate to large ones, over several orders of magnitude.

What it is used for: before spending millions on a training run you run a series of small models, fit the curve, and predict what the large one will give. That makes scaling an engineering discipline instead of a guess.

Two things to keep apart:

The Chinchilla result (Hoffmann et al. 2022) is the most practically usable. With a fixed compute budget $C \approx 6ND$ FLOPs the loss is minimised when

$D \approx 20 \cdot N$

that is, about 20 training tokens per parameter.

That was an upending conclusion, because the largest models of the time were heavily undertrained:

Chinchilla beat Gopher even though it was four times smaller — it had been trained on the right amount of data for its size.

But Chinchilla optimises the training cost, not the total cost. If the model is going to be run billions of times the inference dominates, and then it pays to train a smaller model for longer than Chinchilla says. That is why modern open models are often trained on 100–1 000 tokens per parameter — far beyond the «optimal» — because a smaller model is cheaper to run for ever.

How it is done in practice:

1. Train 5–8 small models of different sizes, each to its Chinchilla-optimal data volume.
2. Fit $L = L_\infty + aN^{-\alpha}$ in log-log.
3. Extrapolate to the target size.
4. Check against a point in the middle that was not used in the fit.

Step 4 is what separates a usable prediction from a hope.

What scaling laws do not predict:

The third row is the most practically significant: the scaling laws say nothing about better data being able to move the whole curve downwards, which it does. Cleaning and filtering the corpus is often cheaper than multiplying the computation tenfold.

import numpy as np

def chinchilla_optimal(budget_flops):
    """C ≈ 6ND and D ≈ 20N ⟹ C ≈ 120N²."""
    N = (budget_flops / 120) ** 0.5
    return {"parameters_bn": round(N / 1e9, 2),
            "tokens_bn": round(20 * N / 1e9, 1),
            "check_flops": f"{6 * N * 20 * N:.2e}"}

for budget in (1e21, 1e22, 1e23, 1e24):
    print(f"{budget:.0e} FLOPs → {chinchilla_optimal(budget)}")
# 1e+21 FLOPs → {'parameters_bn': 2.89, 'tokens_bn': 57.7, ...}
# 1e+23 FLOPs → {'parameters_bn': 28.87, 'tokens_bn': 577.4, ...}

# Fit a power law to measured data and extrapolate
def fit_power_law(N, L, L_inf=1.6):
    """L = L_inf + a·N^(-alpha). A linear regression in log-log on (L - L_inf)."""
    x = np.log(np.asarray(N, float))
    y = np.log(np.asarray(L, float) - L_inf)
    neg_alpha, log_a = np.polyfit(x, y, 1)
    return {"alpha": round(float(-neg_alpha), 4), "a": round(float(np.exp(log_a)), 2),
            "predict": lambda n: L_inf + np.exp(log_a) * n ** neg_alpha}

# Measurement points from small models
N = np.array([1e7, 3e7, 1e8, 3e8, 1e9])
L = 1.6 + 25.0 * N ** -0.30          # a synthetic «measurement»
f = fit_power_law(N, L)
print(f"fitted alpha {f['alpha']}, a {f['a']}")

for target in (1e10, 1e11, 1e12):
    print(f"  N={target:.0e}: predicted loss {float(f['predict'](target)):.4f}")

# Hold a point out for the check — this is the step most often skipped
N_tr, L_tr = np.delete(N, 2), np.delete(L, 2)
f2 = fit_power_law(N_tr, L_tr)
print(f"check point N=1e8: true {L[2]:.4f}, predicted {float(f2['predict'](N[2])):.4f}")

# Chinchilla optimises the TRAINING — work the total cost out including the inference
def total_cost(N, D, inference_tokens, sek_per_1e18_flops=50.0):
    training = 6 * N * D
    inference = 2 * N * inference_tokens
    return {"training_flops": f"{training:.2e}", "inference_flops": f"{inference:.2e}",
            "total_sek": round((training + inference) / 1e18 * sek_per_1e18_flops, 1),
            "inference_share": round(inference / (training + inference), 3)}

print(total_cost(7e9, 140e9, 1e15))       # a small model, a lot of inference
print(total_cost(70e9, 1400e9, 1e15))     # a large model, the same inference volume
#  ↑ at a high inference volume the smaller model pays off even though it was trained longer