Architectures · advanced · concept 72 of 176

🏗️Positional Encoding & RoPE

Attention treats its input as an unordered set, so word order has to be injected explicitly. Early transformers added sine-wave position signals; modern LLMs rotate query and key vectors by position (RoPE), which extends more gracefully to long contexts. Context-window tricks usually live here.