NCCL / RCCL collectives
Collective-communication library that makes multi-GPU training work
NCCL is NVIDIA's collective-communications library (all-reduce, all-gather) that every large training run depends on to synchronize gradients across thousands of GPUs; AMD's RCCL clones its API. It is an underappreciated part of the CUDA moat because distributed-training performance and fault tolerance hinge on it, and it is tightly co-designed with NVLink and InfiniBand.
Ops
all-reduce, all-gather
AMD equivalent
RCCL
How it fits the stack
NCCL / RCCL collectives with what it depends on (above) and what it feeds (below). The figure renders as a crawlable diagram and upgrades to an interactive 3D graph as it scrolls into view.
NCCL / RCCL collectives in the AI stack. NCCL / RCCL collectives with its immediate upstream dependencies (top) and downstream dependents (bottom) in the AI value chain. Hover a node in 3D, or read the full relationships below.
Graph data (text) — 4 entities, 3 relationships
- NCCL / RCCL collectives —depends on→ InfiniBand (NVIDIA Quantum)
- NCCL / RCCL collectives —depends on→ NVLink / NVSwitch
- Nvidia Data-Center GPU (Blackwell/Rubin) —depends on→ NCCL / RCCL collectives
Depends on ↑ · 2