Architectures · advanced · concept 105 of 311 · Updated Nov 4, 2026

⚡FlashAttention and IO-Aware Attention

An exact attention algorithm that is fast because it moves less data. Standard attention writes an N by N score matrix to GPU memory, which dominates cost at long sequence lengths. FlashAttention tiles the computation so each block stays in fast on-chip SRAM and recomputes scores in the backward pass. Same answer, no approximation. The first paper reported a 3x speedup on GPT-2 at sequence length 1K; later versions target Hopper GPUs and FP8.