{"id":"nccl","level":"software","name":"Collective communication library","shortName":"NCCL collectives","role":"Coordinates group communication operations across participating GPUs.","description":"Distributed computation often needs a group operation rather than an isolated message. An all-reduce combines contributions from every participating rank and makes the result available to each; all-gather and reduce-scatter have different input and output layouts. NCCL provides GPU-oriented implementations of these collective operations over supported communication paths. The atlas depicts ranks as participants and tensor segments as the objects being moved and combined, then overlays the underlying rack or network fabric. A ring is one useful teaching algorithm, not the only implementation. Communication correctness requires compatible participation and operation ordering. Its performance depends on message size, topology, available bandwidth, and how well communication overlaps with independent computation.","specs":[{"label":"All-reduce result","value":"Combined value at every rank","context":"Reduction operation such as sum is specified."},{"label":"Related collectives","value":"All-gather and reduce-scatter","context":"Different distribution semantics."},{"label":"Participants","value":"Ranks in a communicator","context":"Logical membership, not necessarily one entire rack."}],"interfaces":[{"name":"Per-rank tensors","direction":"in","signal":"Distributed contributions"},{"name":"Collective result","direction":"out","signal":"Combined or redistributed tensors"}],"tradeoff":"Collectives simplify distributed coordination. Synchronization and communication can become the dominant waiting cost.","whyAI":"Training workers exchange gradients and sharded state. Collective operations connect those pieces into a consistent computation.","related":["nvlink-switch","leaf-switch","spine-switch","parallelism"],"sources":[{"title":"NVIDIA NCCL — Collective Operations","url":"https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html"},{"title":"NVIDIA NCCL Tests — Bandwidth Accounting","url":"https://github.com/NVIDIA/nccl-tests/blob/master/doc/PERFORMANCE.md"}],"scale":"Logical multi-GPU communication overlay","kind":"collective","mechanism":["Define the participating ranks and the collective operation.","Move and, when required, reduce tensor segments over supported paths.","Return the specified result layout to each participant."],"misconception":"All-reduce is a communication-and-reduction operation; it does not mean every worker retrains the entire model from scratch.","check":{"question":"What does a sum all-reduce return?","options":["Only the first rank’s original tensor","The sum of contributions to every participating rank","A new physical GPU"],"answer":1,"explanation":"Each participant receives the reduced result, unlike a reduction that delivers only to one root."},"branch":"Distributed software","subbranch":"Collective algorithms","science":[{"title":"All-reduce traffic normalization","equation":"B_bus = (S / t) × 2(P − 1)/P","explanation":"For the conventional bandwidth accounting, all-reduce includes reduction and distribution traffic. NCCL tests normalize algorithm bandwidth using this factor so it can be compared with a communication bottleneck.","assumptions":"P is rank count and S is bytes per rank; conventional point-to-point all-reduce model, not a universal traffic law for hardware-assisted reductions or every algorithm."}],"history":[],"prerequisites":["nvlink-port","network-adapter"],"learningObjective":"Normalize conventional all-reduce throughput while distinguishing algorithm bandwidth from traffic on links.","engineeringExample":"Synthetic four-rank all-reduce: each rank contributes 1 GB and completion takes 20 ms. Algorithm bandwidth is 50 GB/s. Applying 2(P − 1)/P = 1.5 gives normalized bus bandwidth of 75 GB/s for the stated accounting convention.","evidenceNotes":"The factor is the conventional all-reduce model in the linked NCCL bandwidth note. It is not a universal per-link traffic law for every algorithm or hardware-assisted reduction; ranks, message bytes and completion boundary must match.","labId":"engineering","scope":"Vary ranks, payload, effective bandwidth and startup in an ideal ring all-reduce; this is not a topology-specific or every-collective traffic model.","deepLabId":"network","investigationId":"execution","labScope":"Vary ranks, payload, effective bandwidth and startup in an ideal ring all-reduce; this is not a topology-specific or every-collective traffic model."}