DeepSpeed / Megatron
ZeRO + tensor/pipeline parallelism for trillion-param training
Microsoft DeepSpeed (ZeRO, ZeRO-Offload) and Nvidia Megatron-LM provide the distributed-training primitives — tensor, pipeline, sequence, and expert parallelism — used to train frontier dense and MoE models. Megatron-Core is the reference for large-scale GPU training; the two are frequently combined. They are the invisible plumbing behind most >100B-parameter runs.
Key idea
ZeRO sharding + 4D parallelism
Owners
Microsoft / Nvidia
How it fits the stack
DeepSpeed / Megatron with what it depends on (above) and what it feeds (below). The figure renders as a crawlable diagram and upgrades to an interactive 3D graph as it scrolls into view.
DeepSpeed / Megatron in the AI stack. DeepSpeed / Megatron with its immediate upstream dependencies (top) and downstream dependents (bottom) in the AI value chain. Hover a node in 3D, or read the full relationships below.
Graph data (text) — 4 entities, 3 relationships
- DeepSpeed / Megatron —depends on→ Nvidia Data-Center GPU (Blackwell/Rubin)
- Meta AI (FAIR) —uses→ DeepSpeed / Megatron
- OpenAI —uses→ DeepSpeed / Megatron
Depends on ↑ · 1