From memory placement to network, power and cooling

The mechanism

Moving state out of HBM can free capacity while creating traffic on package, CPU–GPU, PCIe, storage or network paths. Every path has its own direction, contention domain and service overhead. A larger batch may amortize weight reads while increasing KV and queueing; disaggregated serving may improve resource matching while adding KV transfers. The resulting workload changes electrical demand and heat distribution, but fixed conversion losses and idle power remain. A credible comparison closes memory, movement, compute, facility services and response-quality requirements on the same time interval.

Start with the working-set plan and trace every moved byte across named physical interfaces.
Aggregate traffic by link and direction, separating expert-weight reads, token routing, KV movement and checkpoints.
Find a workload-rate bound from compute, memory and network demand; use a schedule to expose dependencies and overlap.
Apply measured or explicitly assumed power at the chosen IT boundary and the facility overhead at the same interval.
Report accepted outputs meeting quality and latency targets, and identify whether the constraint moved after the placement change.

Science and assumptions

Resource-normalized service bound

R\leq\min\left(\frac{F}{f},\frac{B_{\mathrm{mem}}}{v_{\mathrm{mem}}},\frac{B_{\mathrm{net}}}{v_{\mathrm{net}}}\right)

R ≤ min(F/f, Bmem/vmem, Bnet/vnet)

Symbols and units

R: Completed defined work units per second
work/s
F: Usable compute rate
operations/s
f: Operations per work unit
operations/work
Bmem: Usable memory bandwidth
bytes/s
vmem: Memory bytes per work unit
bytes/work
Bnet: Usable network bandwidth
bytes/s
vnet: Network bytes per work unit
bytes/work

Normalize each resource to the same defined unit of work before taking the minimum.

Assumptions Steady aggregate upper bound, independent capacities and positive demand; omit an unused resource. Serial phases, latency and skew can lower performance.

Energy per accepted work

e=\frac{P_{\mathrm{IT}}\,\mathrm{PUE}}{R_{\mathrm{accepted}}}

e = PIT PUE / Raccepted

Symbols and units

e: Facility energy per accepted work unit
J/work
PIT: Mean IT equipment power
W
PUE: Facility/IT energy ratio for that interval
dimensionless
Raccepted: Accepted work rate
work/s

Use average IT power and accepted-work rate over the same measured interval; PUE supplies the facility boundary conversion.

Assumptions Stable or interval-averaged boundaries; Raccepted must be positive and enforce a defined quality/latency criterion. This is not a universal joules-per-token constant.

Engineering example

Synthetic service: compute supports 300 work/s, memory 200 work/s and network 100 work/s; the simple bound is 100 work/s. Halving memory traffic alone leaves that bound unchanged. At 12 kW IT, PUE 1.2 and 90 accepted work/s, facility energy is 160 J per accepted work unit.

Distinctions and tradeoffs

A local improvement can move the bottleneck to another interface or increase work elsewhere in the system.

Capacity offload can reduce device count but increase transfer stalls or energy elsewhere; no component metric alone predicts the net result.

This closes the wafer → package → memory → hall → model journey through actual work rather than installed hardware totals.

Specifications and interfaces

Same work unit: Token, request or training update
Define model, precision, quality and latency target.
Boundary: Link → rack → IT → facility
Do not mix package power with facility-wide outputs.

Interfaces

Memory/runtime policy Input: Resident state, misses, transfer volume and schedule
Hall service result Output: Accepted work, power, cooling and deployment requirements

Check your understanding

A placement change lowers HBM traffic but doubles remote transfers. What should be compared?

The complete workload at consistent quality, latency and energy boundaries
Only HBM bandwidth utilization
Only the capacity printed on the memory package

Answer and reasoning

The complete workload at consistent quality, latency and energy boundaries

A local improvement can move the bottleneck to another interface or increase work elsewhere in the system.

Evidence and scope

All numerical values are scenario inputs. No relationship between a megawatt and tokens is assumed without a named workload. CRG provides an attributed way to organize these realization boundaries; the arithmetic is a resource-accounting model.

Propagate a stated facility power budget, PUE, readiness and utilization to illustrative work; detailed conversion, battery, cooling and grid models are not included.