# Parallelism as a memory and communication budget

Canonical: https://semiaifoundry.com/learn/atlas/parallelism-communication-budget/
Revision: 2026-10-07
Rights: https://semiaifoundry.com/ai-atlas/licenses.html

Model–machine co-design · Atlas v1.3.0

# Parallelism as a memory and communication budget

Chooses tensor, pipeline, data and state partitions under topology and dependency constraints.

[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#parallelism-communication-budget)Accelerator group to serving pool; routes schematic

Learning objectiveDerive communication consequences from a chosen partition and topology.

## The mechanism

Distributed execution divides work and state in several ways. Data parallelism processes different examples and synchronizes updates. Tensor parallelism splits operators and exchanges partial results. Pipeline parallelism assigns layer groups to stages, introducing activation transfers and possible bubbles. State sharding distributes parameters, gradients or optimizer state that would otherwise be replicated. These choices often combine. More devices increase available memory and arithmetic, but also add collectives, synchronization and topology constraints. A good plan places frequent exchanges on suitable links and measures how much communication is exposed on the critical path instead of assuming that all network time overlaps with computation.

1.  Choose which dimensions of work and state need partitioning to meet local memory and time constraints.
2.  Derive the transfers and collectives implied by each partition rather than starting from a chip count.
3.  Map frequent communication onto topology regions with sufficient latency and bandwidth characteristics.
4.  Schedule overlap only where producer and consumer dependencies allow it without premature buffer reuse.
5.  Measure completed step or token latency and adjust partition size when smaller local compute no longer amortizes communication.

## Science and assumptions

### Communication approximation

$$
T_{\mathrm{comm}}\approx n\alpha+\frac{Q}{B_{\mathrm{eff}}}
$$

Communication time is approximately the phase count times startup latency plus transferred bytes divided by effective bandwidth.

Symbols and units

T\_comm

Modeled communication time

s

n

Latency-bearing communication phase count

phases

α

Startup time per phase

s/phase

Q

Payload transferred by the chosen algorithm at its boundary

bytes

B\_eff

Effective useful payload bandwidth

bytes/s

n is latency-bearing phases, α seconds/phase, Q transferred bytes and Beff useful bytes/s for the chosen algorithm and topology.

Assumptions A coarse model; contention, overlap, reduction compute and protocol behavior can change both terms.

## Engineering example

Synthetic collective: four latency-bearing phases at 5 μs plus 100 MB at 50 GB/s gives about 2.02 ms. Cutting local compute to 1 ms exposes this cost unless a valid overlap schedule exists.

## Distinctions and tradeoffs

Adding devices increases aggregate resources but can lower efficiency; aggregate link rates do not determine the latency of a particular collective.

Sharding can make a model fit but increase traffic; larger parallel groups can reduce local work enough that communication dominates.

Training and inference have different collective frequencies and batch shapes, so a partition that trains efficiently may serve poorly.

## Specifications and interfaces

Partitioned object

Examples, tensor dimensions, layers or persistent state

Different partitions create different communication patterns.

Collective cost

Latency plus payload transfer

Small messages can be latency-limited; large messages can be bandwidth-limited.

### Interfaces

Model and device topology Input

Tensor sizes, layer graph, state layout and link capacities

Distributed step Output

Scheduled compute, collectives and synchronization

## Check your understanding

### Which partition commonly splits a single matrix operation across devices?

1.  Tensor parallelism
2.  Only dataset shuffling
3.  Recovery checkpoint compression

Answer and reasoning

Tensor parallelism

Tensor parallelism partitions operator dimensions and requires communication to assemble or combine the resulting values.

## Evidence and scope

No particular collective algorithm is assumed universally. Published TPU topology and ZeRO methods provide examples, not interchangeable performance constants.

Vary ranks, payload, effective bandwidth and startup in an ideal ring all-reduce; this is not a topology-specific or every-collective traffic model.

## Sources

-   [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://arxiv.org/abs/1910.02054)
-   [TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings](https://arxiv.org/abs/2304.01433v3)

[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/parallelism-communication-budget-0dc71e564bfad9c8.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)

## Continue exploring

### Foundations

-   [Data, tensor, pipeline, and state parallelism](https://semiaifoundry.com/learn/atlas/parallelism/)
-   [Collective communication library](https://semiaifoundry.com/learn/atlas/nccl/)

### Related lessons

-   [Training state and activation rematerialization](https://semiaifoundry.com/learn/atlas/training-memory-state/)