Integrated accelerator rack: GB200 NVL72 example

The mechanism

A rack becomes an AI system when its components are integrated around shared communication and service infrastructure. The GB200 NVL72 example combines seventy-two Blackwell GPUs and thirty-six Grace CPUs in one rack-scale NVLink domain. Compute trays, switch trays, cable cartridges, power distribution, and cooling each occupy distinct roles. The atlas exposes them as separate but connected layers: operands move through the data fabric, electricity arrives through the power system, and heat leaves through cooling. The rack is not literally one semiconductor chip, despite descriptions of it acting like one large GPU. Software must still partition work, manage memory, and coordinate communication to turn its resources into useful training or inference.

Place compute and communication trays within the rack.
Connect the trays to shared data, power, and coolant paths.
Use distributed software to map a model onto the coordinated resources.

Science and assumptions

Communication-aware throughput ceiling

R\le\min(P_{\mathrm{peak}},B_{\mathrm{mem}}I_{\mathrm{mem}},B_{\mathrm{net}}I_{\mathrm{net}})

Rack throughput is bounded by arithmetic capacity, memory bandwidth times memory intensity, and network bandwidth times network intensity; the smallest bound applies.

Symbols and units

R: Useful arithmetic throughput under the declared work convention
operations/s
P_peak: Compatible arithmetic throughput ceiling
operations/s
B_mem: Useful memory bandwidth
bytes/s
I_mem: Useful operations per byte crossing the memory boundary
operations/byte
B_net: Useful network bandwidth
bytes/s
I_net: Useful operations per byte crossing the network boundary
operations/byte

$P_{\mathrm{peak}}$ is arithmetic capacity. $I_{\mathrm{mem}}$ and $I_{\mathrm{net}}$ count useful operations per byte crossing the chosen memory and network boundaries. The limiting resource can change with the workload partition.

Assumptions Roofline-style explanatory upper bound with consistent operation formats and accounting boundaries; not a prediction of measured rack performance.

Engineering example

The smallest of $P_{\mathrm{peak}}$ , $B_{\mathrm{mem}}$ $I_{\mathrm{mem}}$ and $B_{\mathrm{net}}$ $I_{\mathrm{net}}$ bounds the workload-specific rate in the first-order roofline.

Distinctions and tradeoffs

A 72-GPU rack is not one physical GPU die, and its aggregate bandwidth is not a single pairwise connection.

Dense integration reduces communication distance. It concentrates power, cooling, and service requirements.

The rack is a useful scale for model execution. Its effective output depends on the whole system remaining supplied and coordinated.

Specifications and interfaces

GB200 NVL72 example: 72 GPUs + 36 CPUs
A documented reference system, not all AI racks.
Compute and switch trays: 18 + 9
One-rack DGX NVL72 configuration.
NVLink aggregate: ≈130 TB/s
Aggregate rack communication specification; not bandwidth between one GPU pair.

Interfaces

Facility services and jobs Input: Power, cooling, network data, and scheduled work
Computed results and heat Output: Model outputs, exchanged tensors, and thermal load

Check your understanding

What makes the rack a usable AI system?

The coordinated compute, data, power, cooling, and software layers
Its cabinet color
GPU count alone

Answer and reasoning

The coordinated compute, data, power, cooling, and software layers

All layers must work together; arithmetic units cannot run useful jobs without data, power, cooling, and coordination.

Historical context

2024 · GB200 NVL72 rack-scale architecture announced

The documented example integrates 72 GPUs through a large NVLink domain with purpose-built communication and liquid-cooling infrastructure. It illustrates system-level co-design rather than proving a model capability by itself.

NVIDIA — GB200 NVL72 Architecture, March 2024

Evidence and scope

The terms require a common workload and time boundary.

Propagate a stated facility power budget, PUE, readiness and utilization to illustrative work; detailed conversion, battery, cooling and grid models are not included.