Transformer / MoE Architecture
The 'Attention Is All You Need' substrate of all frontier models
The Transformer (Vaswani et al., 2017) and its Mixture-of-Experts variants are the architectural substrate on which every frontier lab builds; DeepSeek-V3, Llama 4, GPT, Gemini, and Qwen all rely on sparse MoE, RoPE positional encoding, and RMSNorm derivatives. Architectural innovations (MLA, grouped-query attention, sparse experts) are the core research currency of the field. It is the shared genome of the entire model layer.
Origin
Google, 2017
Dominant variant
Sparse MoE
How it fits the stack
Transformer / MoE Architecture with what it depends on (above) and what it feeds (below). The figure renders as a crawlable diagram and upgrades to an interactive 3D graph as it scrolls into view.
Transformer / MoE Architecture in the AI stack. Transformer / MoE Architecture with its immediate upstream dependencies (top) and downstream dependents (bottom) in the AI value chain. Hover a node in 3D, or read the full relationships below.
Graph data (text) — 5 entities, 4 relationships
- Claude (Opus / Sonnet) —uses→ Transformer / MoE Architecture
- DeepSeek-V3 / R1 —uses→ Transformer / MoE Architecture
- FlashAttention —depends on→ Transformer / MoE Architecture
- GPT-5 / o-series —uses→ Transformer / MoE Architecture