The mechanism
Distributed computation often needs a group operation rather than an isolated message. An all-reduce combines contributions from every participating rank and makes the result available to each; all-gather and reduce-scatter have different input and output layouts. NCCL provides GPU-oriented implementations of these collective operations over supported communication paths. The atlas depicts ranks as participants and tensor segments as the objects being moved and combined, then overlays the underlying rack or network fabric. A ring is one useful teaching algorithm, not the only implementation. Communication correctness requires compatible participation and operation ordering. Its performance depends on message size, topology, available bandwidth, and how well communication overlaps with independent computation.
- Define the participating ranks and the collective operation.
- Move and, when required, reduce tensor segments over supported paths.
- Return the specified result layout to each participant.
Science and assumptions
All-reduce traffic normalization
Conventional all-reduce bus bandwidth is payload divided by elapsed time, multiplied by two times ranks minus one divided by ranks.
Symbols and units
- B_bus
- Conventionally normalized all-reduce bus bandwidth
bytes/s
- S
- Input payload per rank
bytes
- t
- Measured all-reduce duration
s
- P
- Participating rank count
positive integer count
For the conventional bandwidth accounting, all-reduce includes reduction and distribution traffic. NCCL tests normalize algorithm bandwidth using this factor so it can be compared with a communication bottleneck.
Assumptions P is rank count and S is bytes per rank; conventional point-to-point all-reduce model, not a universal traffic law for hardware-assisted reductions or every algorithm.
Engineering example
Synthetic four-rank all-reduce: each rank contributes 1 GB and completion takes 20 ms. Algorithm bandwidth is 50 GB/s. Applying 2(P − 1)/P = 1.5 gives normalized bus bandwidth of 75 GB/s for the stated accounting convention.
Distinctions and tradeoffs
All-reduce is a communication-and-reduction operation; it does not mean every worker retrains the entire model from scratch.
Collectives simplify distributed coordination. Synchronization and communication can become the dominant waiting cost.
Training workers exchange gradients and sharded state. Collective operations connect those pieces into a consistent computation.
Specifications and interfaces
- All-reduce result
- Combined value at every rank
Reduction operation such as sum is specified.
- Related collectives
- All-gather and reduce-scatter
Different distribution semantics.
- Participants
- Ranks in a communicator
Logical membership, not necessarily one entire rack.
Interfaces
- Per-rank tensors Input
- Distributed contributions
- Collective result Output
- Combined or redistributed tensors
Check your understanding
What does a sum all-reduce return?
- Only the first rank’s original tensor
- The sum of contributions to every participating rank
- A new physical GPU
Answer and reasoning
The sum of contributions to every participating rank
Each participant receives the reduced result, unlike a reduction that delivers only to one root.
Evidence and scope
The factor is the conventional all-reduce model in the linked NCCL bandwidth note. It is not a universal per-link traffic law for every algorithm or hardware-assisted reduction; ranks, message bytes and completion boundary must match.
Vary ranks, payload, effective bandwidth and startup in an ideal ring all-reduce; this is not a topology-specific or every-collective traffic model.