Language models and probabilities

Consider the sentence «I am going to send an email to …». Most people think of a colleague, a client, or a supplier. Few think of a robot or an asteroid.

You guess based on what you have experienced before. A language model does exactly that — except it has processed vast amounts of text and calculates: “colleague” 40%, “client” 20%, “boss” 10% … Then it selects a word, adds it, and calculates the probability of the next word.

An entire text is just a series of these probability calculations.

Try it yourself: Write the beginning of a sentence, e.g. «The meeting started at …». What is the most likely answer? Perhaps “9:00” or “10:00”. That is the word’s “probability” in that context.

«I paid the invoice with …» → cash, card, check. «The weather in Stockholm is …» → rainy, sunny, windy.

Try a sentence where it is hard to guess: «My favourite colour is …» — here the answers are scattered, and a language model also becomes uncertain.