The mechanism
PCIe is a general-purpose component interconnect built from serial lanes. A link can use several lanes in parallel, and switches can connect multiple devices into a hierarchy. Transactions carry commands and data; software uses drivers and device capabilities to arrange transfers. The generation and lane count must both be stated when comparing bandwidth. A faster endpoint may operate at a lower rate when the host, switch, or board path supports less. Transmit and receive paths contribute to different directional rates; packet overhead also separates a link rating from one-way application throughput. PCIe is one part of the AI system’s connectivity, alongside specialized coherent links, GPU scale-up fabrics, and the network between servers.
- The endpoints train a link and agree on a supported speed and lane width.
- Transactions are packetized and serialized across separate transmit and receive paths.
- The receiving device handles requests or data and sends completions as required by the protocol.
Science and assumptions
Encoding and protocol reduce raw lane bandwidth
Gen5 one-way bandwidth equals lane count times 32 billion raw bits per second per lane, times the 128-over-130 encoding fraction, divided by eight bits per byte.
Symbols and units
- B_Gen5,one way
- Gen5 directional bandwidth after 128b/130b encoding
bytes/s
- N_lanes
- Negotiated active lane count
lanes
- 32 × 10⁹ bits/s
- Raw per-lane NRZ bit rate corresponding to 32 GT/s
bits/(s·lane)
- 128/130
- Encoding payload fraction before transaction overhead
fraction
For a Gen5 link, each lane runs at 32 GT/s with 128b/130b encoding. Sixteen lanes yield about 63.0 GB/s per direction after this encoding overhead, before transaction and link-protocol costs. Full-duplex operation can carry traffic both ways, but summing the directions does not make a single transfer twice as fast.
Assumptions PCIe Gen5 at negotiated full lane width, using NRZ and 128b/130b encoding. Excludes packet headers, flow control, acknowledgments, and implementation limits. PCIe Gen6 uses a different signaling and FLIT model and must be calculated separately.
Engineering example
A Gen5 x16 one-way payload estimate multiplies lanes, raw rate and 128/130 encoding efficiency, then divides bits by eight.
Distinctions and tradeoffs
GT/s, GB/s, and bidirectional aggregate bandwidth are different measures. A raw line rate must account for lane count, direction, and protocol overhead before comparison with application throughput.
Higher speeds increase transfer capacity but tighten electrical and board-design requirements. Real throughput also depends on packet overhead, topology, transfer size, and device behavior.
NICs and NVMe drives use PCIe to deliver data into the compute system. Congestion on these paths can starve accelerators even when the GPUs themselves are fast.
Specifications and interfaces
- PCIe 5.0 signaling
- 32 GT/s per lane
Raw transfer rate; payload bandwidth is lower.
- PCIe 6.0 signaling
- 64 GT/s per lane
Uses PAM4, FLIT encoding, and forward error correction.
- PCIe 6.0 ×16
- 256 GB/s raw bidirectional
128 GB/s each direction before protocol overhead; not a GB200 board-wide guarantee.
Interfaces
- Host transactions Input
- Memory reads/writes, commands, and completions
- Device transfers Output
- DMA data and completion traffic
- Link training Input
- Negotiated width, speed, and signal equalization
Check your understanding
What happens if a fast PCIe device is connected through a slower supported host path?
- The link must always use the device’s maximum advertised generation.
- It operates at a mutually supported speed and width.
- It automatically becomes an NVLink interface.
Answer and reasoning
It operates at a mutually supported speed and width.
PCIe capability is negotiated across the actual link. The endpoint, host, switches, and board support determine what is usable.
Evidence and scope
Protocol packets, flow control and platform limits reduce delivered throughput.
Use the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism.