# Collective communication library

Canonical: https://semiaifoundry.com/learn/atlas/nccl/
Revision: 2026-10-07
Rights: https://semiaifoundry.com/ai-atlas/licenses.html

Distributed software · Atlas v1.3.0

# Collective communication library

Coordinates group communication operations across participating GPUs.

[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#nccl)Logical multi-GPU communication overlay

Learning objectiveNormalize conventional all-reduce throughput while distinguishing algorithm bandwidth from traffic on links.

## The mechanism

Distributed computation often needs a group operation rather than an isolated message. An all-reduce combines contributions from every participating rank and makes the result available to each; all-gather and reduce-scatter have different input and output layouts. NCCL provides GPU-oriented implementations of these collective operations over supported communication paths. The atlas depicts ranks as participants and tensor segments as the objects being moved and combined, then overlays the underlying rack or network fabric. A ring is one useful teaching algorithm, not the only implementation. Communication correctness requires compatible participation and operation ordering. Its performance depends on message size, topology, available bandwidth, and how well communication overlaps with independent computation.

1.  Define the participating ranks and the collective operation.
2.  Move and, when required, reduce tensor segments over supported paths.
3.  Return the specified result layout to each participant.

## Science and assumptions

### All-reduce traffic normalization

$$
B_{\mathrm{bus}}=\frac{S}{t}\frac{2(P-1)}{P}
$$

Conventional all-reduce bus bandwidth is payload divided by elapsed time, multiplied by two times ranks minus one divided by ranks.

Symbols and units

B\_bus

Conventionally normalized all-reduce bus bandwidth

bytes/s

S

Input payload per rank

bytes

t

Measured all-reduce duration

s

P

Participating rank count

positive integer count

For the conventional bandwidth accounting, all-reduce includes reduction and distribution traffic. NCCL tests normalize algorithm bandwidth using this factor so it can be compared with a communication bottleneck.

Assumptions P is rank count and S is bytes per rank; conventional point-to-point all-reduce model, not a universal traffic law for hardware-assisted reductions or every algorithm.

## Engineering example

Synthetic four-rank all-reduce: each rank contributes 1 GB and completion takes 20 ms. Algorithm bandwidth is 50 GB/s. Applying 2(P − 1)/P = 1.5 gives normalized bus bandwidth of 75 GB/s for the stated accounting convention.

## Distinctions and tradeoffs

All-reduce is a communication-and-reduction operation; it does not mean every worker retrains the entire model from scratch.

Collectives simplify distributed coordination. Synchronization and communication can become the dominant waiting cost.

Training workers exchange gradients and sharded state. Collective operations connect those pieces into a consistent computation.

## Specifications and interfaces

All-reduce result

Combined value at every rank

Reduction operation such as sum is specified.

Related collectives

All-gather and reduce-scatter

Different distribution semantics.

Participants

Ranks in a communicator

Logical membership, not necessarily one entire rack.

### Interfaces

Per-rank tensors Input

Distributed contributions

Collective result Output

Combined or redistributed tensors

## Check your understanding

### What does a sum all-reduce return?

1.  Only the first rank’s original tensor
2.  The sum of contributions to every participating rank
3.  A new physical GPU

Answer and reasoning

The sum of contributions to every participating rank

Each participant receives the reduced result, unlike a reduction that delivers only to one root.

## Evidence and scope

The factor is the conventional all-reduce model in the linked NCCL bandwidth note. It is not a universal per-link traffic law for every algorithm or hardware-assisted reduction; ranks, message bytes and completion boundary must match.

Vary ranks, payload, effective bandwidth and startup in an ideal ring all-reduce; this is not a topology-specific or every-collective traffic model.

## Sources

-   [NVIDIA NCCL — Collective Operations](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html)
-   [NVIDIA NCCL Tests — Bandwidth Accounting](https://github.com/NVIDIA/nccl-tests/blob/master/doc/PERFORMANCE.md)

[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/nccl-6adf3f210d984e2b.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)

## Continue exploring

### Foundations

-   [NVLink GPU interface](https://semiaifoundry.com/learn/atlas/nvlink-port/)
-   [Network interface adapter](https://semiaifoundry.com/learn/atlas/network-adapter/)

### Related lessons

-   [NVLink switch tray and scale-up fabric](https://semiaifoundry.com/learn/atlas/nvlink-switch/)
-   [Leaf switch and endpoint network](https://semiaifoundry.com/learn/atlas/leaf-switch/)
-   [Spine switching and fabric topology](https://semiaifoundry.com/learn/atlas/spine-switch/)
-   [Data, tensor, pipeline, and state parallelism](https://semiaifoundry.com/learn/atlas/parallelism/)