Architectures · advanced · concept 70 of 176

🔀Mixture-of-Experts (MoE)

A Transformer variant where each layer holds many parallel expert subnetworks and a router activates only a few per token. This decouples total parameter count from per-token compute, so models can grow far larger without proportional cost. Many frontier and open-weight LLMs use MoE.