Architectures · advanced · concept 100 of 311 · Updated Nov 4, 2026

🔀Mixture-of-Experts (MoE)

A Transformer variant where each layer holds many parallel expert subnetworks and a router activates only a few per token. This decouples total parameter count from per-token compute, so models can grow far larger without proportional cost. Many frontier and open-weight LLMs use MoE.