Words as points: similar words close together
Be able to place words on a grid based on similarity and see that 'cat' ends up near 'dog'.
Prerequisites
Intuition
Imagine a map where words are points. Words with similar meanings are located close to each other: cat is near dog, far from tractor. King and queen are close to each other — and the arrow from king to queen resembles the arrow from man to woman.
A language model has such a map, but with hundreds of dimensions instead of two. Each word is a list of numbers — a vector. The model learned the map by reading: words that appear in similar contexts end up close together.
Interactive
Make your own map: draw two axes. x = «how alive» (0 = object, 10 = animal/human). y = «how big» (0 = small, 10 = enormous).
Place: ant, elephant, stone, mountain, cat, car.
- ant (10, 1) · elephant (10, 9) · stone (0, 1) · mountain (0, 10) · cat (10, 3) · car (1, 5).
Which two are closest? Ant and cat, or stone and ant? Measure with a ruler. Would a third axis («can move») change which ones are closest? That is why real word vectors have many dimensions.
Mastery means
- Places words on a grid based on similarity and justifies the choice
- Explains that the distance between points indicates semantic similarity
Sign in to do the exercises and build your mastery up.
Sources
- Wikipedia — Word embedding (CC BY-SA 4.0) — CC BY-SA 4.0
- CS Unplugged (CC BY-SA 4.0) — CC BY-SA 4.0