Sampling: temperature, top-k, top-p, beam

The model gives a probability per possible next token. The decoding strategy decides how one is chosen.

Strategy	How	Gives
Greedy	always the highest probability	deterministic, repetitive
Temperature	scale the logits by 1/T before the softmax	T<1 more cautious, T>1 more varied
Top-k	draw only among the k most likely	cuts the tail, a fixed k
Top-p (nucleus)	draw among the smallest set summing to p	adaptive — narrow when the model is sure
Beam search	keep b partial sequences, keep the best	best for translation, dull for free text

The default choices: T ≈ 0.7 and top-p ≈ 0.9 for creative text; T = 0 (greedy) for classification, extraction and code that has to be reproducible.

import numpy as np

def softmax(z):
    z = z - z.max(); e = np.exp(z); return e / e.sum()

def sample(logits, T=1.0, top_k=None, top_p=None, rng=np.random.default_rng(0)):
    if T <= 0:
        return int(np.argmax(logits))                 # greedy
    p = softmax(logits / T)
    if top_k:
        cut = np.argsort(p)[:-top_k]
        p[cut] = 0
    if top_p:
        order = np.argsort(p)[::-1]
        cum = np.cumsum(p[order])
        keep = order[: int(np.searchsorted(cum, top_p)) + 1]
        mask = np.zeros_like(p, dtype=bool); mask[keep] = True
        p[~mask] = 0
    p = p / p.sum()
    return int(rng.choice(len(p), p=p))

logits = np.array([4.0, 3.5, 1.0, 0.5, -2.0])
print(softmax(logits).round(3))                 # [0.55 0.334 0.027 0.017 0.001]
print(sample(logits, T=0))                      # 0  (greedy)
print([sample(logits, T=1.0, top_p=0.9) for _ in range(5)])

Beam search's weakness: it maximises the sequence's total probability, and the most likely text is often generic and repetitive («I do not know. I do not know.»). People do not write maximally probable sentences — which is why sampling is used for free text and beam only where there is a «right» answer.