The mechanism
A collective describes communication among a group of participants. All-reduce combines values and returns the result to all ranks; all-gather assembles distributed pieces; reduce-scatter combines values while leaving different output pieces on different ranks. Libraries such as NCCL choose implementations suited to message size and topology. The logical collective is not identical to one physical link transfer. Congestion, competing storage traffic and poorly placed ranks can reduce useful bandwidth, while many small exchanges expose latency. Correct diagnosis follows both the operation’s semantics and the actual traffic schedule, including any communication that overlaps computation.
- Define the collective and the tensor bytes supplied by each rank.
- Choose rank placement and an implementation suited to the topology.
- Map the collective phases onto actual links and identify shared cuts.
- Measure effective bandwidth, phase latency and exposed synchronization under competing traffic.
- Change placement, scheduling or algorithm only after identifying the binding network behavior.
Science and assumptions
Ideal ring all-reduce
Ideal ring duration is two times ranks minus one startup costs, plus two times ranks minus one divided by ranks times per-rank payload divided by bandwidth.
Symbols and units
- T
- Idealized ring all-reduce duration
s
- n
- Participating ranks
positive integer count
- α
- Startup latency per ring step
s/step
- S
- Input payload per rank
bytes
- B
- Effective one-direction payload bandwidth per rank
bytes/s
n is ranks, α seconds per step, S bytes/rank and B effective bytes/s.
Assumptions A simplified ring model with equal links; topology, pipelining, overlap and congestion can change actual timing.
Engineering example
Synthetic eight-rank ring with 1 GiB/rank and 50 GB/s effective bandwidth has about 37.6 ms of bandwidth time plus 14 latency terms. Using raw line rate as effective bandwidth would be optimistic.
Distinctions and tradeoffs
A 400 Gbit/s port is not a measured 400 GB/s collective, and bidirectional rates cannot be silently used as one-way payload bandwidth.
Algorithms with fewer phases can help small messages, while larger transfers reward bandwidth-efficient routing; the best choice changes with topology.
Synchronized training and tensor-parallel inference repeatedly wait for collective results, turning network behavior into model-step or token latency.
Specifications and interfaces
- Message size
- Bytes per participating rank
Do not confuse one rank’s input with total wire traffic.
- Effective bandwidth
- Useful bytes/s under the chosen algorithm
Line rate omits protocol and contention losses.
Interfaces
- Rank-local tensors Input
- Values requiring collection, distribution or reduction
- Collective output Output
- Correctly placed combined or gathered tensors
Check your understanding
What is 400 Gbit/s in raw decimal bytes/s?
- 50 GB/s before overhead
- 400 GB/s
- 3,200 GB/s
Answer and reasoning
50 GB/s before overhead
Eight bits form one byte; useful payload rate is lower when coding, protocol and contention losses apply.
Evidence and scope
This is a teaching algorithm, not a claim that NCCL always chooses a ring. Rank placement and software version belong with any measured result.
Vary ranks, payload, effective bandwidth and startup in an ideal ring all-reduce; this is not a topology-specific or every-collective traffic model.