Architectures · intermediate · concept 80 of 311 · Updated Nov 4, 2026
Attention Mechanism
A technique allowing models to focus on relevant parts of input when producing output. Self-attention (in Transformers) lets each position attend to all other positions in a sequence.
Interactive · 3D
Watch attention poll the sentence →
The journey step where every token asks every earlier token what matters.
Top AI contributors on this
- Yoshua BengioLawZero / Université de Montréal
- Ilya SutskeverSafe Superintelligence Inc. (SSI)
- Noam ShazeerOpenAI
- Alec RadfordIndependent / Thinking Machines Lab (adviser)
- Jürgen SchmidhuberKAUST / IDSIA
- Oriol VinyalsDiscovery Loop
Key terms
QueryKeyValueAttention weightsSoftmax
Learn these first
Videos
▶ Attention in transformers, step-by-step | Deep Learning Chapter 6 ↗
3Blue1Brown · YouTube
▶ Transformers, the tech behind LLMs | Deep Learning Chapter 5 ↗
3Blue1Brown · YouTube
Guides and articles
CS50 AI: Language ↗
Harvard
The Illustrated Transformer ↗
Jay Alammar
Dive into Deep Learning ↗
Aston Zhang and others
Sequence to sequence and attention ↗
Lena Voita
Courses, papers, and more
This unlocks