The mechanism
Distributed execution divides work and state in several ways. Data parallelism processes different examples and synchronizes updates. Tensor parallelism splits operators and exchanges partial results. Pipeline parallelism assigns layer groups to stages, introducing activation transfers and possible bubbles. State sharding distributes parameters, gradients or optimizer state that would otherwise be replicated. These choices often combine. More devices increase available memory and arithmetic, but also add collectives, synchronization and topology constraints. A good plan places frequent exchanges on suitable links and measures how much communication is exposed on the critical path instead of assuming that all network time overlaps with computation.
- Choose which dimensions of work and state need partitioning to meet local memory and time constraints.
- Derive the transfers and collectives implied by each partition rather than starting from a chip count.
- Map frequent communication onto topology regions with sufficient latency and bandwidth characteristics.
- Schedule overlap only where producer and consumer dependencies allow it without premature buffer reuse.
- Measure completed step or token latency and adjust partition size when smaller local compute no longer amortizes communication.
Science and assumptions
Communication approximation
Communication time is approximately the phase count times startup latency plus transferred bytes divided by effective bandwidth.
Symbols and units
- T_comm
- Modeled communication time
s
- n
- Latency-bearing communication phase count
phases
- α
- Startup time per phase
s/phase
- Q
- Payload transferred by the chosen algorithm at its boundary
bytes
- B_eff
- Effective useful payload bandwidth
bytes/s
n is latency-bearing phases, α seconds/phase, Q transferred bytes and Beff useful bytes/s for the chosen algorithm and topology.
Assumptions A coarse model; contention, overlap, reduction compute and protocol behavior can change both terms.
Engineering example
Synthetic collective: four latency-bearing phases at 5 μs plus 100 MB at 50 GB/s gives about 2.02 ms. Cutting local compute to 1 ms exposes this cost unless a valid overlap schedule exists.
Distinctions and tradeoffs
Adding devices increases aggregate resources but can lower efficiency; aggregate link rates do not determine the latency of a particular collective.
Sharding can make a model fit but increase traffic; larger parallel groups can reduce local work enough that communication dominates.
Training and inference have different collective frequencies and batch shapes, so a partition that trains efficiently may serve poorly.
Specifications and interfaces
- Partitioned object
- Examples, tensor dimensions, layers or persistent state
Different partitions create different communication patterns.
- Collective cost
- Latency plus payload transfer
Small messages can be latency-limited; large messages can be bandwidth-limited.
Interfaces
- Model and device topology Input
- Tensor sizes, layer graph, state layout and link capacities
- Distributed step Output
- Scheduled compute, collectives and synchronization
Check your understanding
Which partition commonly splits a single matrix operation across devices?
- Tensor parallelism
- Only dataset shuffling
- Recovery checkpoint compression
Answer and reasoning
Tensor parallelism
Tensor parallelism partitions operator dimensions and requires communication to assemble or combine the resulting values.
Evidence and scope
No particular collective algorithm is assumed universally. Published TPU topology and ZeRO methods provide examples, not interchangeable performance constants.
Vary ranks, payload, effective bandwidth and startup in an ideal ring all-reduce; this is not a topology-specific or every-collective traffic model.