Attention: how words influence each other

"The bank lay by the river." — Which bank is meant? The word river decides: it is a riverbank, not an office. To understand "the bank", you must look at the river, not at "lay" or "by".

A language model does this with attention: for every word, it calculates how much it should "look at" every other word. "The bank" → high attention on "the river", low attention on "by".

The word it in "The customer called support until it got tired" — who got tired? Attention decides, and it can be wrong — just like for us.

Draw arrows: Write a sentence on a piece of paper. Choose a word. Draw arrows to the words needed to understand it; thicker arrow = more important.

"The boss gave the employee the report because she wanted to read it."

• "she" → arrow to employee (likely — but "boss" is possible).
• "it" → arrow to report.

Do the same with "... because she was done with it". Now "she" likely points to the boss. Same word, different context, different arrow. This is exactly what attention calculates.