Architectures · intermediate · concept 79 of 311 · Updated Nov 4, 2026
Transformer Architecture
THE architecture that changed everything (2017). Uses self-attention to process sequences in parallel, replacing RNNs. Foundation of GPT, BERT, and virtually every modern AI system.
Interactive · 3D
The transformer stack, explorable in 3D →
Attention beams, expert routing, the KV cache: click every stage.
Top AI contributors on this
- Dario AmodeiAnthropic
- Sam AltmanOpenAI
- Jeff DeanDiscovery Loop
- Noam ShazeerOpenAI
- Alec RadfordIndependent / Thinking Machines Lab (adviser)
- Oriol VinyalsDiscovery Loop
Key terms
Self-attentionMulti-head attentionPositional encodingEncoder-decoder
Learn these first
Videos
▶ Transformers, the tech behind LLMs | Deep Learning Chapter 5 ↗
3Blue1Brown · YouTube
▶ Attention in transformers, step-by-step | Deep Learning Chapter 6 ↗
3Blue1Brown · YouTube
Guides and articles
CS25: Transformers United ↗
Stanford
CS50 AI: Language ↗
Harvard
Neural Networks: Zero to Hero ↗
Andrej Karpathy
The Illustrated Transformer ↗
Jay Alammar
Attention Is All You Need ↗
Vaswani and others
How do Transformers work? · Hugging Face ↗
Hugging Face
This unlocks