FlashAttention
IO-aware attention kernel underpinning long context
FlashAttention (Tri Dao et al.) is the IO-aware exact-attention algorithm that removes the memory bottleneck of the attention mechanism, enabling the long-context windows shipped across virtually every frontier model. FlashAttention-3 targets Hopper/Blackwell with FP8 and asynchrony. It is a rare academic artifact that became universal infrastructure.
Author
Tri Dao (Together/Princeton)
Impact
Universal in transformers
How it fits the stack
FlashAttention with what it depends on (above) and what it feeds (below). The figure renders as a crawlable diagram and upgrades to an interactive 3D graph as it scrolls into view.
FlashAttention in the AI stack. FlashAttention with its immediate upstream dependencies (top) and downstream dependents (bottom) in the AI value chain. Hover a node in 3D, or read the full relationships below.
Graph data (text) — 3 entities, 2 relationships
- FlashAttention —depends on→ Transformer / MoE Architecture
- Llama —uses→ FlashAttention
Depends on ↑ · 1