Encyclopedia · 311 concepts
Learn this concept as a feed

Architectures · advanced · concept 102 of 311 · Updated Nov 4, 2026

Positional Encoding & RoPE

Attention treats its input as an unordered set, so word order has to be injected explicitly. Early transformers added sine-wave position signals; modern LLMs rotate query and key vectors by position (RoPE), which extends more gracefully to long contexts. Context-window tricks usually live here.

L7 · LLM and generative AI engineerDevelopersTech leadersQA and testing

Top AI contributors on this

Key terms

Position embeddingsSinusoidal encodingRoPERelative positionContext extension

Learn these first

Where you meet it in the real world

Why a model knows 'dog bites man' from 'man bites dog', and how million-token contexts are stretched

Videos

Guides and articles

Courses, papers, and more