A visual tour · modern language models

How does an LLM work?

From text → attention → the next token.

01 / 08Use ← → or swipe
The whole idea

Predict what comes next

Theskyisblue
blue
72%
clear
18%
falling
4%
02 / 08Generate one token · repeat
Pre-training

Practice on enormous amounts of text

adjust
weights
stories & prose
code </>
articles & facts
conversations
03 / 08Predict → compare → adjust
Tokens become numbers

Text enters as vectors

“bank”token
→
[ .2, −.7, .9, … ]embedding + position
04 / 08Meaning starts as a pattern of numbers
Self-attention

Each token looks for useful context

Idepositedmoneyatthebanktoday
Target word
bankWhat does it mean here?
→
Useful context
deposited + moneyAttention gives these clues more weight
→
Contextual meaning
financial bankNot a river bank
05 / 08“money” + “deposited” → financial bank
The “transform” in transformer

Context transforms the representation

bank
general token
+ context→↓
bank
financial institution, in this sentence
06 / 08Same word · context-aware numbers
Many transformer blocks

Refine context, layer after layer

token vectors
+ position
attention
what matters?
mix & refine
neural network
repeat × N
richer context
next-token odds
softmax
07 / 08Attention + computation + residual connections
GPT, in one picture

Three ideas working together

G
Generativeproduce the next token
P
Pre-trainedlearn patterns beforehand
T
Transformermake tokens context-aware
08 / 08Not a database: learned statistical patterns