BERT and masked language modelling

GPT reads from the left: every token sees only what came before. Good for generating. But to understand a sentence — classify it, find entities, compare — you want to see the whole sentence in both directions.

BERT is an encoder: no causal mask. The training: mask 15 % of the tokens ([MASK]) and guess them — masked language modelling. The model learns contextual representations but cannot generate text.

How it is used: put a small classification head on top of the [CLS] token's vector (or mean-pool them all) and fine-tune the whole model for a few epochs. With small datasets (hundreds of examples) a fine-tuned encoder often beats a prompted LLM on both accuracy and cost.

from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline

name = "KB/bert-base-swedish-cased"          # a Swedish BERT (KB-lab)
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name, num_labels=3)

batch = tok(["Servicen var utmärkt", "Aldrig mer"], padding=True, truncation=True, return_tensors="pt")
print(batch["input_ids"].shape)              # (2, T)
logits = model(**batch).logits               # (2, 3) — an untrained head until you fine-tune it

# an MLM demo
fill = pipeline("fill-mask", model=name)
print(fill("Stockholm är Sveriges [MASK].")[0]["token_str"])   # huvudstad ("capital")

Fine-tuning: Trainer or an ordinary PyTorch loop, lr ≈ 2e-5, 3 epochs, batch 16–32. Freeze nothing — the whole model is updated.

The MLM objective: for the masked positions $\mathcal M$, minimise $-\sum_{t\in\mathcal M}\log p_\theta(x_t\mid \tilde x)$ where $\tilde x$ is the sequence with the mask. BERT uses 80/10/10 (mask/random token/unchanged) to reduce the discrepancy between training and fine-tuning, since [MASK] never appears at inference. Encoder attention is full ($T\times T$ without a mask), which gives bidirectional context but makes the model non-autoregressive. Sentence embeddings (SBERT) are trained contrastively so that the cosine similarity between pooled vectors reflects semantic similarity — the basis of semantic retrieval.