The mechanism
A NIC converts server memory transfers into network packets and receives packets from other machines. High-performance adapters can perform remote direct memory access, allowing registered buffers to be transferred with less involvement from the remote CPU. GPU-aware paths can also reduce unnecessary staging through host memory when the platform and software support them. These capabilities do not eliminate synchronization, congestion, or network topology. On a supported path, a tensor fragment leaves a GPU buffer, passes through the adapter and reaches a remote buffer; control and synchronization govern when the transfer is usable. The NIC belongs to the scale-out path between systems; it should not be confused with NVLink’s local GPU fabric or with the ordinary management network.
- Software or a supported accelerator path identifies buffers and initiates a network operation.
- The NIC reads data, applies the appropriate protocol processing, and transmits packets through the network.
- The remote adapter places received data into supported buffers and reports completion so dependent work can continue.
Science and assumptions
Small messages and large transfers have different bottlenecks
Message time is approximately fixed startup latency plus payload size divided by sustained payload bandwidth.
Symbols and units
- T_message
- Message completion time in the path model
s
- α
- Fixed software/adapter/path startup latency
s
- M
- Payload size
bytes
- B_payload
- Sustained directional payload bandwidth
bytes/s
α represents fixed software, adapter, and path latency; M is payload bytes; Bpayload is sustained payload bandwidth in bytes per second. For small M, startup dominates. For large M, bandwidth dominates. This explains why buying a faster line-rate adapter may help bulk gradient transfers more than tiny synchronization messages, and why scheduling and aggregation matter.
Assumptions A single uncontended path and approximately constant startup cost. Congestion, multiple hops, retransmission, protocol offloads, and overlap can change both terms. Convert bits per second to bytes per second before comparing them.
Engineering example
Synthetic uncontended transfer: α = 5 μs and payload bandwidth = 25 GB/s. A 1 MB message needs 40 μs of serialization, giving about 45 μs total. A 1 kB message needs only 0.04 μs of serialization but still about 5.04 μs total, so startup dominates.
Distinctions and tradeoffs
A 400 Gb/s port is not a 400 GB/s port. Dividing by eight gives 50 GB/s before network and protocol overhead.
Faster ports help only if the host path and network can sustain them. Queueing, congestion control, and placement can dominate tail latency even below the nominal line rate.
Distributed training exchanges gradients and parameters across servers. Distributed inference and storage access also depend on efficient movement beyond one rack’s local GPU domain.
Specifications and interfaces
- ConnectX-7 example
- Up to 400 Gb/s throughput
NVIDIA adapter family rating; selected card and port modes vary.
- Byte-rate equivalent
- 50 GB/s at 400 Gb/s
Arithmetic conversion before protocol overhead; not 400 GB/s.
- Acceleration
- RDMA / RoCE and offloads
Availability depends on adapter, firmware, topology, and software.
Interfaces
- Local memory transfer Input
- Registered host or supported GPU buffers
- Network packets Output
- Tensor, storage, and application traffic
- Received packets Input
- Remote data and protocol responses
Check your understanding
What does RDMA principally reduce in a supported transfer?
- The remote CPU’s work copying and processing the payload.
- The number of bits in the model vocabulary.
- The need for any network fabric.
Answer and reasoning
The remote CPU’s work copying and processing the payload.
RDMA places data directly into registered memory with less remote CPU intervention. It still requires a functioning network, protocol, and synchronization.
Evidence and scope
The lesson’s latency-plus-payload model uses decimal bytes and one direction. These are synthetic rates; the linked network references do not make this a promise about RDMA, GPU access, congestion or any deployed adapter.
Vary ranks, payload, effective bandwidth and startup in an ideal ring all-reduce; this is not a topology-specific or every-collective traffic model.