Architectures · advanced · concept 106 of 311 · Updated Nov 4, 2026
GQA, MQA, Multi-head Latent and Sliding Window Attention
Changes to multi-head attention that shrink the KV cache, the memory that grows with every token. Multi-query attention shares one key and value head across all query heads. Grouped-query attention shares them within groups, as in Llama 3. Multi-head latent attention compresses keys and values into a small latent vector; DeepSeek-V2 reports a 93.3% smaller KV cache. Sliding window attention, as in Mistral 7B, limits each token to a fixed window.
Top AI contributors on this
- Dario AmodeiAnthropic
- Sam AltmanOpenAI
- Jeff DeanDiscovery Loop
- Noam ShazeerOpenAI
- Alec RadfordIndependent / Thinking Machines Lab (adviser)
- Oriol VinyalsDiscovery Loop
Key terms
Where you meet it in the real world
Llama 2 and 3, Mistral 7B, DeepSeek-V2 and V3, Gemma, long-document models, memory-bound serving on consumer GPUs
Videos
Welch Labs · YouTube
Julien Simon · YouTube
Guides and articles
Courses, papers, and more