# Data, tensor, pipeline, and state parallelism

Canonical: https://semiaifoundry.com/learn/atlas/parallelism/
Revision: 2026-10-07
Rights: https://semiaifoundry.com/ai-atlas/licenses.html

Distributed software · Atlas v1.3.0

# Data, tensor, pipeline, and state parallelism

Splits model work and state across devices to fit memory and exploit concurrent execution.

[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#parallelism)Logical model-to-device mapping

Learning objectiveAccount for accumulation and replication when sizing global batch.

## The mechanism

Parallel training has several independent axes. Data parallelism processes different examples with model replicas; tensor parallelism splits work inside layers; pipeline parallelism assigns different layer groups to stages. Other schemes partition sequence context, experts, or model state. These choices can be combined, but every partition creates communication and scheduling consequences. The atlas lets a viewer apply different splits to the same model diagram and observe which tensors must cross device boundaries. More GPUs are therefore not a universal speed multiplier. The best arrangement depends on model shape, sequence length, memory capacity, network topology, batch behavior, and the overhead of coordinating the pieces into one correct training or inference computation.

1.  Choose which workload dimensions or state components to partition.
2.  Assign the resulting pieces to devices with suitable connectivity.
3.  Schedule computation and required exchanges so the complete operation remains correct.

## Science and assumptions

### Global batch accounting

$$
B_{\mathrm{global}}=B_{\mathrm{micro}}N_{\mathrm{accumulation}}N_{\mathrm{data\ parallel}}
$$

Global batch equals per-replica microbatch size times accumulated microbatches times data-parallel replicas.

Symbols and units

B\_global

Examples per global optimizer update

examples/update

B\_micro

Examples per microbatch per data-parallel replica

examples/microbatch

N\_accumulation

Accumulated microbatches per replica per update

microbatches/update

N\_data parallel

Data-parallel replica count

count

Data-parallel replicas contribute different examples. Tensor- and pipeline-parallel workers cooperate on those examples and therefore do not automatically multiply the effective global batch.

Assumptions Uniform per-replica microbatch and accumulation count; variable batches and specialized schedules require explicit accounting.

## Engineering example

A microbatch of 8 with 16 accumulation steps and 64 data-parallel replicas represents a nominal global batch of 8192 samples.

## Distinctions and tradeoffs

Doubling GPU count does not guarantee halving runtime; communication and serial dependencies remain.

Partitioning increases usable capacity. Communication and pipeline idle time can reduce scaling efficiency.

Large models often exceed one device’s practical resources. Parallelism makes their execution possible across a coordinated system.

## Specifications and interfaces

Data parallelism

Split batch dimension

Workers coordinate the resulting updates.

Tensor / pipeline

Within layers / across depth

Different communication and scheduling patterns.

State sharding

Partition stored model state

Can reduce per-device memory use.

### Interfaces

Model, batch, and hardware Input

Workload structure and resource constraints

Distributed execution plan Output

Device partitions and communication requirements

## Check your understanding

### Which strategy splits work inside an individual layer?

1.  Pipeline parallelism only
2.  Data replication only
3.  Tensor parallelism

Answer and reasoning

Tensor parallelism

Tensor parallelism divides a layer’s tensor computations; pipeline parallelism instead partitions groups of layers.

## Historical context

### 2021 · Large-model tensor, pipeline, and data parallelism analyzed together

The Megatron-LM study quantified interactions between parallelism strategies, emphasizing that model partitioning and hardware communication must be designed together.

[Narayanan et al. — Efficient Large-Scale Language Model Training](https://arxiv.org/abs/2104.04473)

## Evidence and scope

Sequence packing and uneven batches can change the effective count.

Vary ranks, payload, effective bandwidth and startup in an ideal ring all-reduce; this is not a topology-specific or every-collective traffic model.

## Sources

-   [NVIDIA Megatron Core — Parallelism Strategies](https://docs.nvidia.com/megatron-core/developer-guide/latest/user-guide/parallelism-guide.html)
-   [Narayanan et al. — Efficient Large-Scale Language Model Training](https://arxiv.org/abs/2104.04473)

[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/parallelism-9dfa2adfc60daa86.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)

## Continue exploring

### Foundations

-   [Multiply–accumulate and fused arithmetic](https://semiaifoundry.com/learn/atlas/mac/)
-   [Collective communication library](https://semiaifoundry.com/learn/atlas/nccl/)

### Related lessons

-   [Cluster scheduler and resource allocation](https://semiaifoundry.com/learn/atlas/scheduler/)
-   [Integrated accelerator rack: GB200 NVL72 example](https://semiaifoundry.com/learn/atlas/rack/)
-   [Distributed checkpoint and recovery state](https://semiaifoundry.com/learn/atlas/checkpoint/)
-   [Loss, gradients & optimizer](https://semiaifoundry.com/learn/atlas/loss-optimizer/)