{"id":"parallelism","level":"software","name":"Data, tensor, pipeline, and state parallelism","shortName":"Parallelism","role":"Splits model work and state across devices to fit memory and exploit concurrent execution.","description":"Parallel training has several independent axes. Data parallelism processes different examples with model replicas; tensor parallelism splits work inside layers; pipeline parallelism assigns different layer groups to stages. Other schemes partition sequence context, experts, or model state. These choices can be combined, but every partition creates communication and scheduling consequences. The atlas lets a viewer apply different splits to the same model diagram and observe which tensors must cross device boundaries. More GPUs are therefore not a universal speed multiplier. The best arrangement depends on model shape, sequence length, memory capacity, network topology, batch behavior, and the overhead of coordinating the pieces into one correct training or inference computation.","specs":[{"label":"Data parallelism","value":"Split batch dimension","context":"Workers coordinate the resulting updates."},{"label":"Tensor / pipeline","value":"Within layers / across depth","context":"Different communication and scheduling patterns."},{"label":"State sharding","value":"Partition stored model state","context":"Can reduce per-device memory use."}],"interfaces":[{"name":"Model, batch, and hardware","direction":"in","signal":"Workload structure and resource constraints"},{"name":"Distributed execution plan","direction":"out","signal":"Device partitions and communication requirements"}],"tradeoff":"Partitioning increases usable capacity. Communication and pipeline idle time can reduce scaling efficiency.","whyAI":"Large models often exceed one device’s practical resources. Parallelism makes their execution possible across a coordinated system.","related":["nccl","scheduler","rack","checkpoint","loss-optimizer"],"sources":[{"title":"NVIDIA Megatron Core — Parallelism Strategies","url":"https://docs.nvidia.com/megatron-core/developer-guide/latest/user-guide/parallelism-guide.html"},{"title":"Narayanan et al. — Efficient Large-Scale Language Model Training","url":"https://arxiv.org/abs/2104.04473"}],"scale":"Logical model-to-device mapping","kind":"parallelism","mechanism":["Choose which workload dimensions or state components to partition.","Assign the resulting pieces to devices with suitable connectivity.","Schedule computation and required exchanges so the complete operation remains correct."],"misconception":"Doubling GPU count does not guarantee halving runtime; communication and serial dependencies remain.","check":{"question":"Which strategy splits work inside an individual layer?","options":["Pipeline parallelism only","Data replication only","Tensor parallelism"],"answer":2,"explanation":"Tensor parallelism divides a layer’s tensor computations; pipeline parallelism instead partitions groups of layers."},"branch":"Distributed software","subbranch":"Work and state partitioning","science":[{"title":"Global batch accounting","equation":"B_global = B_micro × N_accumulation × N_data-parallel","explanation":"Data-parallel replicas contribute different examples. Tensor- and pipeline-parallel workers cooperate on those examples and therefore do not automatically multiply the effective global batch.","assumptions":"Uniform per-replica microbatch and accumulation count; variable batches and specialized schedules require explicit accounting."}],"history":[{"year":2021,"title":"Large-model tensor, pipeline, and data parallelism analyzed together","significance":"The Megatron-LM study quantified interactions between parallelism strategies, emphasizing that model partitioning and hardware communication must be designed together.","source":{"title":"Narayanan et al. — Efficient Large-Scale Language Model Training","url":"https://arxiv.org/abs/2104.04473"}}],"learningObjective":"Account for accumulation and replication when sizing global batch.","engineeringExample":"A microbatch of 8 with 16 accumulation steps and 64 data-parallel replicas represents a nominal global batch of 8192 samples.","evidenceNotes":"Sequence packing and uneven batches can change the effective count.","visualFamily":"software-flow","prerequisites":["mac","nccl"],"labId":"engineering","scope":"Vary ranks, payload, effective bandwidth and startup in an ideal ring all-reduce; this is not a topology-specific or every-collective traffic model.","deepLabId":"network","investigationId":"execution","labScope":"Vary ranks, payload, effective bandwidth and startup in an ideal ring all-reduce; this is not a topology-specific or every-collective traffic model."}