The mechanism
A model can be partitioned across GPUs because one device may not provide enough memory or enough compute. NVLink supplies a fast path for those devices to exchange tensor pieces and partial results. Physical links connect GPUs to peers or switches; software still chooses a parallel strategy and schedules the required communication. A connected group is not automatically one ordinary GPU with uniform access costs. Collective operations, message size, synchronization, and topology all affect useful performance. When a tensor is partitioned across devices, its local computations and any required reduction create separate arithmetic and communication costs. NVLink is distinct from the internal die-to-die interface and from the network that joins racks.
- Software partitions a workload and identifies the tensor values or partial results that must cross GPU boundaries.
- NVLink-connected endpoints transfer those buffers through the supported peer or switched fabric.
- Collective operations combine or distribute the results, then dependent computation proceeds when the required data are ready.
Science and assumptions
Communication has volume and startup costs
Ring time is approximately two times ranks minus one startup costs, plus two times ranks minus one divided by ranks times payload divided by bandwidth.
Symbols and units
- T_ring
- Modeled ring all-reduce time
s
- p
- Participating ranks
positive integer count
- α
- Startup latency per ring step
s/step
- M
- Input payload per rank
bytes
- β
- Effective one-direction payload bandwidth per rank
bytes/s
p is the number of ranks, M is each rank’s input size in bytes, α is startup time per ring step, and β is effective one-direction payload bandwidth per rank. The first term exposes why many small messages can be latency limited. The second describes the distributed data movement of a ring all-reduce, which reduces partial values and shares the combined result.
Assumptions A balanced ring using reduce-scatter plus all-gather with equal links and no contention, reduction-compute bottleneck, or overlap. NCCL can choose other algorithms, including switch-assisted schemes; β must not be substituted with a summed bidirectional marketing rating.
Engineering example
A ring all-reduce sends approximately 2(p−1)M/p bytes per rank; startup latency can dominate small messages.
Distinctions and tradeoffs
A 1.8 TB/s bidirectional rating sums traffic capacity in both directions. It does not promise one transfer moving at 1.8 TB/s in a single direction.
More communication capacity helps larger parallel models, but every synchronization remains work that must be scheduled. Partitioning a model too finely can make transfers dominate the saved compute time.
Tensor and expert parallelism depend on exchanging activations or partial sums between accelerators. Scale-up communication lets nearby GPUs cooperate more efficiently on one model.
Specifications and interfaces
- Blackwell NVLink 5
- 1.8 TB/s bidirectional per GPU
Aggregate across 18 links; both directions summed.
- Link count
- 18 per GPU
100 GB/s bidirectional per link in NVIDIA’s specification.
- NVL72 aggregation
- About 130 TB/s
Rack aggregate interface bandwidth; not the bandwidth of one transfer.
Interfaces
- Local tensors Input
- GPU memory and collective-operation buffers
- Remote transfers Output
- Tensor partitions, reductions, and peer memory traffic
- Remote results Input
- Peer tensor data and synchronization
Check your understanding
Which work does NVLink most directly support?
- Refreshing a single DRAM capacitor.
- Encoding a text token into its vocabulary ID.
- Exchanging tensors among GPUs in a scale-up domain.
Answer and reasoning
Exchanging tensors among GPUs in a scale-up domain.
NVLink connects cooperating accelerators. The mathematical partition and communication schedule are still chosen by software.
Historical context
2016 · Pascal P100 brings NVLink to NVIDIA accelerators
P100’s launch combined NVLink with a new accelerator and memory architecture, giving cooperating GPUs a communication path beyond the host PCIe interface.
Evidence and scope
The ring is an ideal topology model; implementations may use trees or hierarchies.
Vary ranks, payload, effective bandwidth and startup in an ideal ring all-reduce; this is not a topology-specific or every-collective traffic model.