# From memory placement to network, power and cooling Canonical: https://semiaifoundry.com/learn/atlas/memory-to-hall-propagation/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html Infrastructure · Atlas v1.3.0 # From memory placement to network, power and cooling Trace how a tensor-placement decision changes link demand and useful service at the hall boundary. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#memory-to-hall-propagation)Package/link topology → rack services → facility accounting; schematic Learning objectiveTrace how a tensor-placement decision changes link demand and useful service at the hall boundary. ## The mechanism Moving state out of HBM can free capacity while creating traffic on package, CPU–GPU, PCIe, storage or network paths. Every path has its own direction, contention domain and service overhead. A larger batch may amortize weight reads while increasing KV and queueing; disaggregated serving may improve resource matching while adding KV transfers. The resulting workload changes electrical demand and heat distribution, but fixed conversion losses and idle power remain. A credible comparison closes memory, movement, compute, facility services and response-quality requirements on the same time interval. 1. Start with the working-set plan and trace every moved byte across named physical interfaces. 2. Aggregate traffic by link and direction, separating expert-weight reads, token routing, KV movement and checkpoints. 3. Find a workload-rate bound from compute, memory and network demand; use a schedule to expose dependencies and overlap. 4. Apply measured or explicitly assumed power at the chosen IT boundary and the facility overhead at the same interval. 5. Report accepted outputs meeting quality and latency targets, and identify whether the constraint moved after the placement change. ## Science and assumptions ### Resource-normalized service bound $$ R\leq\min\left(\frac{F}{f},\frac{B_{\mathrm{mem}}}{v_{\mathrm{mem}}},\frac{B_{\mathrm{net}}}{v_{\mathrm{net}}}\right) $$ R ≤ min(F/f, Bmem/vmem, Bnet/vnet) Symbols and units R Completed defined work units per second work/s F Usable compute rate operations/s f Operations per work unit operations/work Bmem Usable memory bandwidth bytes/s vmem Memory bytes per work unit bytes/work Bnet Usable network bandwidth bytes/s vnet Network bytes per work unit bytes/work Normalize each resource to the same defined unit of work before taking the minimum. Assumptions Steady aggregate upper bound, independent capacities and positive demand; omit an unused resource. Serial phases, latency and skew can lower performance. ### Energy per accepted work $$ e=\frac{P_{\mathrm{IT}}\,\mathrm{PUE}}{R_{\mathrm{accepted}}} $$ e = PIT PUE / Raccepted Symbols and units e Facility energy per accepted work unit J/work PIT Mean IT equipment power W PUE Facility/IT energy ratio for that interval dimensionless Raccepted Accepted work rate work/s Use average IT power and accepted-work rate over the same measured interval; PUE supplies the facility boundary conversion. Assumptions Stable or interval-averaged boundaries; Raccepted must be positive and enforce a defined quality/latency criterion. This is not a universal joules-per-token constant. ## Engineering example Synthetic service: compute supports 300 work/s, memory 200 work/s and network 100 work/s; the simple bound is 100 work/s. Halving memory traffic alone leaves that bound unchanged. At 12 kW IT, PUE 1.2 and 90 accepted work/s, facility energy is 160 J per accepted work unit. ## Distinctions and tradeoffs A local improvement can move the bottleneck to another interface or increase work elsewhere in the system. Capacity offload can reduce device count but increase transfer stalls or energy elsewhere; no component metric alone predicts the net result. This closes the wafer → package → memory → hall → model journey through actual work rather than installed hardware totals. ## Specifications and interfaces Same work unit Token, request or training update Define model, precision, quality and latency target. Boundary Link → rack → IT → facility Do not mix package power with facility-wide outputs. ### Interfaces Memory/runtime policy Input Resident state, misses, transfer volume and schedule Hall service result Output Accepted work, power, cooling and deployment requirements ## Check your understanding ### A placement change lowers HBM traffic but doubles remote transfers. What should be compared? 1. The complete workload at consistent quality, latency and energy boundaries 2. Only HBM bandwidth utilization 3. Only the capacity printed on the memory package Answer and reasoning The complete workload at consistent quality, latency and energy boundaries A local improvement can move the bottleneck to another interface or increase work elsewhere in the system. ## Evidence and scope All numerical values are scenario inputs. No relationship between a megawatt and tokens is assumed without a named workload. CRG provides an attributed way to organize these realization boundaries; the arithmetic is a resource-accounting model. Propagate a stated facility power budget, PUE, readiness and utilization to illustrative work; detailed conversion, battery, cooling and grid models are not included. ## Sources - [NVIDIA — DGX GB rack hardware guide](https://docs.nvidia.com/dgx/dgxgb200-user-guide/hardware.html) - [MIT OpenCourseWare — Introduction to Heat Transfer](https://ocw.mit.edu/courses/2-051-introduction-to-heat-transfer-fall-2015/) - [NVIDIA — Mastering LLM Techniques: Inference Optimization](https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/) - [Semi AI Foundry — Communication Theory of Computation Realization](https://semiaifoundry.com/updates/communication-theory-of-computation-realization/) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/memory-to-hall-propagation-1502229faf4cfc16.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Foundations - [Place the working set, not just the model](https://semiaifoundry.com/learn/atlas/memory-working-set-placement/) - [Collective traffic and congestion](https://semiaifoundry.com/learn/atlas/fabric-collective-traffic/) - [Cold plates, manifolds and separated coolant loops](https://semiaifoundry.com/learn/atlas/coolant-loop-separation/) ### Related lessons - [Qualify the memory system and retain useful work](https://semiaifoundry.com/learn/atlas/memory-system-qualification/) - [Package and system preserve the contract](https://semiaifoundry.com/learn/atlas/crg-package-system-closure/) - [Verification closes the realization loop](https://semiaifoundry.com/learn/atlas/crg-verification-loop/) - [Commissioning a usable AI cluster](https://semiaifoundry.com/learn/atlas/cluster-commissioning-gates/)