{"id":"parallelism-communication-budget","level":"software","name":"Parallelism as a memory and communication budget","shortName":"Parallelism as a memory and communication budget","role":"Chooses tensor, pipeline, data and state partitions under topology and dependency constraints.","description":"Distributed execution divides work and state in several ways. Data parallelism processes different examples and synchronizes updates. Tensor parallelism splits operators and exchanges partial results. Pipeline parallelism assigns layer groups to stages, introducing activation transfers and possible bubbles. State sharding distributes parameters, gradients or optimizer state that would otherwise be replicated. These choices often combine. More devices increase available memory and arithmetic, but also add collectives, synchronization and topology constraints. A good plan places frequent exchanges on suitable links and measures how much communication is exposed on the critical path instead of assuming that all network time overlaps with computation.","specs":[{"label":"Partitioned object","value":"Examples, tensor dimensions, layers or persistent state","context":"Different partitions create different communication patterns."},{"label":"Collective cost","value":"Latency plus payload transfer","context":"Small messages can be latency-limited; large messages can be bandwidth-limited."}],"interfaces":[{"name":"Model and device topology","direction":"in","signal":"Tensor sizes, layer graph, state layout and link capacities"},{"name":"Distributed step","direction":"out","signal":"Scheduled compute, collectives and synchronization"}],"tradeoff":"Sharding can make a model fit but increase traffic; larger parallel groups can reduce local work enough that communication dominates.","whyAI":"Training and inference have different collective frequencies and batch shapes, so a partition that trains efficiently may serve poorly.","related":["parallelism","nccl","training-memory-state"],"sources":[{"title":"ZeRO: Memory Optimizations Toward Training Trillion Parameter Models","url":"https://arxiv.org/abs/1910.02054"},{"title":"TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings","url":"https://arxiv.org/abs/2304.01433v3"}],"scale":"Accelerator group to serving pool; routes schematic","kind":"workload","branch":"Model–machine co-design","subbranch":"Distributed work and serving choices","mechanism":["Choose which dimensions of work and state need partitioning to meet local memory and time constraints.","Derive the transfers and collectives implied by each partition rather than starting from a chip count.","Map frequent communication onto topology regions with sufficient latency and bandwidth characteristics.","Schedule overlap only where producer and consumer dependencies allow it without premature buffer reuse.","Measure completed step or token latency and adjust partition size when smaller local compute no longer amortizes communication."],"misconception":"Adding devices increases aggregate resources but can lower efficiency; aggregate link rates do not determine the latency of a particular collective.","check":{"question":"Which partition commonly splits a single matrix operation across devices?","options":["Tensor parallelism","Only dataset shuffling","Recovery checkpoint compression"],"answer":0,"explanation":"Tensor parallelism partitions operator dimensions and requires communication to assemble or combine the resulting values."},"science":[{"title":"Communication approximation","equation":"Tcomm ≈ n α + Q / Beff","explanation":"n is latency-bearing phases, α seconds/phase, Q transferred bytes and Beff useful bytes/s for the chosen algorithm and topology.","assumptions":"A coarse model; contention, overlap, reduction compute and protocol behavior can change both terms."}],"prerequisites":["parallelism","nccl"],"learningObjective":"Derive communication consequences from a chosen partition and topology.","workstream":"W11","visualFamily":"model-workload","engineeringExample":"Synthetic collective: four latency-bearing phases at 5 μs plus 100 MB at 50 GB/s gives about 2.02 ms. Cutting local compute to 1 ms exposes this cost unless a valid overlap schedule exists.","evidenceNotes":"No particular collective algorithm is assumed universally. Published TPU topology and ZeRO methods provide examples, not interchangeable performance constants.","labId":"engineering","scope":"Vary ranks, payload, effective bandwidth and startup in an ideal ring all-reduce; this is not a topology-specific or every-collective traffic model.","deepLabId":"network","investigationId":"execution","labScope":"Vary ranks, payload, effective bandwidth and startup in an ideal ring all-reduce; this is not a topology-specific or every-collective traffic model."}