{
  "schemaVersion": 1,
  "id": "atlas-lesson-parallelism",
  "title": "Data, tensor, pipeline, and state parallelism",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Splits model work and state across devices to fit memory and exploit concurrent execution.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/parallelism/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Distributed software",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Sequence packing and uneven batches can change the effective count.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/parallelism-9dfa2adfc60daa86.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#parallelism",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "9d2b996b216f76fc37edd1552777bb97143d565c579d6ce4b90b5c46f293d93d",
  "contentLength": 5339,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-parallelism.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-parallelism.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Data, tensor, pipeline, and state parallelism",
      "start": 37,
      "end": 364
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 364,
      "end": 1406
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1406,
      "end": 1434
    },
    {
      "id": "section-4",
      "title": "Global batch accounting",
      "start": 1434,
      "end": 2312
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2312,
      "end": 2463
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2463,
      "end": 2838
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 2838,
      "end": 3136
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3136,
      "end": 3316
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 3316,
      "end": 3345
    },
    {
      "id": "section-10",
      "title": "Which strategy splits work inside an individual layer?",
      "start": 3345,
      "end": 3644
    },
    {
      "id": "section-11",
      "title": "Historical context",
      "start": 3644,
      "end": 3667
    },
    {
      "id": "section-12",
      "title": "2021 · Large-model tensor, pipeline, and data parallelism analyzed together",
      "start": 3667,
      "end": 4019
    },
    {
      "id": "section-13",
      "title": "Evidence and scope",
      "start": 4019,
      "end": 4261
    },
    {
      "id": "section-14",
      "title": "Sources",
      "start": 4261,
      "end": 4717
    },
    {
      "id": "section-15",
      "title": "Continue exploring",
      "start": 4717,
      "end": 4740
    },
    {
      "id": "section-16",
      "title": "Foundations",
      "start": 4740,
      "end": 4933
    },
    {
      "id": "section-17",
      "title": "Related lessons",
      "start": 4933,
      "end": 5339
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#parallelism"
    },
    {
      "title": "Narayanan et al. — Efficient Large-Scale Language Model Training",
      "url": "https://arxiv.org/abs/2104.04473"
    },
    {
      "title": "NVIDIA Megatron Core — Parallelism Strategies",
      "url": "https://docs.nvidia.com/megatron-core/developer-guide/latest/user-guide/parallelism-guide.html"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/parallelism-9dfa2adfc60daa86.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Multiply–accumulate and fused arithmetic",
      "url": "https://semiaifoundry.com/learn/atlas/mac/"
    },
    {
      "title": "Collective communication library",
      "url": "https://semiaifoundry.com/learn/atlas/nccl/"
    },
    {
      "title": "Cluster scheduler and resource allocation",
      "url": "https://semiaifoundry.com/learn/atlas/scheduler/"
    },
    {
      "title": "Integrated accelerator rack: GB200 NVL72 example",
      "url": "https://semiaifoundry.com/learn/atlas/rack/"
    },
    {
      "title": "Distributed checkpoint and recovery state",
      "url": "https://semiaifoundry.com/learn/atlas/checkpoint/"
    },
    {
      "title": "Loss, gradients & optimizer",
      "url": "https://semiaifoundry.com/learn/atlas/loss-optimizer/"
    }
  ],
  "recordHash": "58fc796d1d9b7d6489811cf8ba0e070dc75e67964a053f99b93a0cc2902fbd43",
  "contentFormat": "text/markdown",
  "content": "Distributed software · Atlas v1.3.0\n\n# Data, tensor, pipeline, and state parallelism\n\nSplits model work and state across devices to fit memory and exploit concurrent execution.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#parallelism)Logical model-to-device mapping\n\nLearning objectiveAccount for accumulation and replication when sizing global batch.\n\n## The mechanism\n\nParallel training has several independent axes. Data parallelism processes different examples with model replicas; tensor parallelism splits work inside layers; pipeline parallelism assigns different layer groups to stages. Other schemes partition sequence context, experts, or model state. These choices can be combined, but every partition creates communication and scheduling consequences. The atlas lets a viewer apply different splits to the same model diagram and observe which tensors must cross device boundaries. More GPUs are therefore not a universal speed multiplier. The best arrangement depends on model shape, sequence length, memory capacity, network topology, batch behavior, and the overhead of coordinating the pieces into one correct training or inference computation.\n\n1.  Choose which workload dimensions or state components to partition.\n2.  Assign the resulting pieces to devices with suitable connectivity.\n3.  Schedule computation and required exchanges so the complete operation remains correct.\n\n## Science and assumptions\n\n### Global batch accounting\n\n$$\nB_{\\mathrm{global}}=B_{\\mathrm{micro}}N_{\\mathrm{accumulation}}N_{\\mathrm{data\\ parallel}}\n$$\n\nGlobal batch equals per-replica microbatch size times accumulated microbatches times data-parallel replicas.\n\nSymbols and units\n\nB\\_global\n\nExamples per global optimizer update\n\nexamples/update\n\nB\\_micro\n\nExamples per microbatch per data-parallel replica\n\nexamples/microbatch\n\nN\\_accumulation\n\nAccumulated microbatches per replica per update\n\nmicrobatches/update\n\nN\\_data parallel\n\nData-parallel replica count\n\ncount\n\nData-parallel replicas contribute different examples. Tensor- and pipeline-parallel workers cooperate on those examples and therefore do not automatically multiply the effective global batch.\n\nAssumptions Uniform per-replica microbatch and accumulation count; variable batches and specialized schedules require explicit accounting.\n\n## Engineering example\n\nA microbatch of 8 with 16 accumulation steps and 64 data-parallel replicas represents a nominal global batch of 8192 samples.\n\n## Distinctions and tradeoffs\n\nDoubling GPU count does not guarantee halving runtime; communication and serial dependencies remain.\n\nPartitioning increases usable capacity. Communication and pipeline idle time can reduce scaling efficiency.\n\nLarge models often exceed one device’s practical resources. Parallelism makes their execution possible across a coordinated system.\n\n## Specifications and interfaces\n\nData parallelism\n\nSplit batch dimension\n\nWorkers coordinate the resulting updates.\n\nTensor / pipeline\n\nWithin layers / across depth\n\nDifferent communication and scheduling patterns.\n\nState sharding\n\nPartition stored model state\n\nCan reduce per-device memory use.\n\n### Interfaces\n\nModel, batch, and hardware Input\n\nWorkload structure and resource constraints\n\nDistributed execution plan Output\n\nDevice partitions and communication requirements\n\n## Check your understanding\n\n### Which strategy splits work inside an individual layer?\n\n1.  Pipeline parallelism only\n2.  Data replication only\n3.  Tensor parallelism\n\nAnswer and reasoning\n\nTensor parallelism\n\nTensor parallelism divides a layer’s tensor computations; pipeline parallelism instead partitions groups of layers.\n\n## Historical context\n\n### 2021 · Large-model tensor, pipeline, and data parallelism analyzed together\n\nThe Megatron-LM study quantified interactions between parallelism strategies, emphasizing that model partitioning and hardware communication must be designed together.\n\n[Narayanan et al. — Efficient Large-Scale Language Model Training](https://arxiv.org/abs/2104.04473)\n\n## Evidence and scope\n\nSequence packing and uneven batches can change the effective count.\n\nVary ranks, payload, effective bandwidth and startup in an ideal ring all-reduce; this is not a topology-specific or every-collective traffic model.\n\n## Sources\n\n-   [NVIDIA Megatron Core — Parallelism Strategies](https://docs.nvidia.com/megatron-core/developer-guide/latest/user-guide/parallelism-guide.html)\n-   [Narayanan et al. — Efficient Large-Scale Language Model Training](https://arxiv.org/abs/2104.04473)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/parallelism-9dfa2adfc60daa86.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Multiply–accumulate and fused arithmetic](https://semiaifoundry.com/learn/atlas/mac/)\n-   [Collective communication library](https://semiaifoundry.com/learn/atlas/nccl/)\n\n### Related lessons\n\n-   [Cluster scheduler and resource allocation](https://semiaifoundry.com/learn/atlas/scheduler/)\n-   [Integrated accelerator rack: GB200 NVL72 example](https://semiaifoundry.com/learn/atlas/rack/)\n-   [Distributed checkpoint and recovery state](https://semiaifoundry.com/learn/atlas/checkpoint/)\n-   [Loss, gradients & optimizer](https://semiaifoundry.com/learn/atlas/loss-optimizer/)"
}