Data, tensor, pipeline, and state parallelism

The mechanism

Parallel training has several independent axes. Data parallelism processes different examples with model replicas; tensor parallelism splits work inside layers; pipeline parallelism assigns different layer groups to stages. Other schemes partition sequence context, experts, or model state. These choices can be combined, but every partition creates communication and scheduling consequences. The atlas lets a viewer apply different splits to the same model diagram and observe which tensors must cross device boundaries. More GPUs are therefore not a universal speed multiplier. The best arrangement depends on model shape, sequence length, memory capacity, network topology, batch behavior, and the overhead of coordinating the pieces into one correct training or inference computation.

Choose which workload dimensions or state components to partition.
Assign the resulting pieces to devices with suitable connectivity.
Schedule computation and required exchanges so the complete operation remains correct.

Science and assumptions

Global batch accounting

B_{\mathrm{global}}=B_{\mathrm{micro}}N_{\mathrm{accumulation}}N_{\mathrm{data\ parallel}}

Global batch equals per-replica microbatch size times accumulated microbatches times data-parallel replicas.

Symbols and units

B_global: Examples per global optimizer update
examples/update
B_micro: Examples per microbatch per data-parallel replica
examples/microbatch
N_accumulation: Accumulated microbatches per replica per update
microbatches/update
N_data parallel: Data-parallel replica count
count

Data-parallel replicas contribute different examples. Tensor- and pipeline-parallel workers cooperate on those examples and therefore do not automatically multiply the effective global batch.

Assumptions Uniform per-replica microbatch and accumulation count; variable batches and specialized schedules require explicit accounting.

Engineering example

A microbatch of 8 with 16 accumulation steps and 64 data-parallel replicas represents a nominal global batch of 8192 samples.

Distinctions and tradeoffs

Doubling GPU count does not guarantee halving runtime; communication and serial dependencies remain.

Partitioning increases usable capacity. Communication and pipeline idle time can reduce scaling efficiency.

Large models often exceed one device’s practical resources. Parallelism makes their execution possible across a coordinated system.

Specifications and interfaces

Data parallelism: Split batch dimension
Workers coordinate the resulting updates.
Tensor / pipeline: Within layers / across depth
Different communication and scheduling patterns.
State sharding: Partition stored model state
Can reduce per-device memory use.

Interfaces

Model, batch, and hardware Input: Workload structure and resource constraints
Distributed execution plan Output: Device partitions and communication requirements

Check your understanding

Which strategy splits work inside an individual layer?

Pipeline parallelism only
Data replication only
Tensor parallelism

Answer and reasoning

Tensor parallelism

Tensor parallelism divides a layer’s tensor computations; pipeline parallelism instead partitions groups of layers.

Historical context

2021 · Large-model tensor, pipeline, and data parallelism analyzed together

The Megatron-LM study quantified interactions between parallelism strategies, emphasizing that model partitioning and hardware communication must be designed together.

Narayanan et al. — Efficient Large-Scale Language Model Training

Evidence and scope

Sequence packing and uneven batches can change the effective count.

Vary ranks, payload, effective bandwidth and startup in an ideal ring all-reduce; this is not a topology-specific or every-collective traffic model.