Architectures · intermediate · concept 81 of 311 · Updated Nov 4, 2026
Self-Attention
A special case of attention where a sequence attends to itself, each token computes relationships with every other token. This is the core mechanism powering Transformers.
Top AI contributors on this
- Noam ShazeerOpenAI
- Christopher ManningStanford University / AIX Ventures
- Aidan GomezCohere
- Ashish VaswaniNvidia (via Essential AI)
- Albert GuCarnegie Mellon University / Cartesia
Key terms
QKV matricesAttention scoreMulti-headCausal masking
Learn these first
Videos
▶ Attention in transformers, step-by-step | Deep Learning Chapter 6 ↗
3Blue1Brown · YouTube
▶ Attention for Neural Networks, Clearly Explained!!! ↗
StatQuest with Josh Starmer · YouTube
Guides and articles
CS25: Transformers United ↗
Stanford
The Illustrated Transformer ↗
Jay Alammar
Attention Is All You Need ↗
Vaswani and others
Courses, papers, and more