The mechanism
A wafer-scale processor uses a connected area much larger than a conventional reticle-sized logic die. Cerebras WSE-3 provides a public example with many compute elements and distributed on-chip SRAM, integrated into a purpose-built system. The attraction is fine-grained local communication and a large aggregate memory fabric. It also demands mechanisms for defect tolerance, routing around unavailable resources, power delivery and heat removal. Wafer-scale integration still has finite storage, routing capacity and fault-tolerance resources; a larger connected surface does not provide unlimited memory. Large models may still need external state movement or system-level streaming, so aggregate local SRAM bandwidth cannot be substituted for external memory bandwidth.
- Partition the model’s operations and state across many local compute-memory regions.
- Use redundant resources and a supported routing map so manufacturing defects do not require every region to be perfect.
- Route intermediate activations through the fabric while keeping reusable state close to its consumer.
- Schedule external transfers for state that cannot remain in distributed local SRAM.
- Close power, cooling and system I/O budgets before treating the connected wafer as deployable capacity.
Science and assumptions
Distributed storage budget
Total local capacity sums the storage regions. External transfer time is at least external payload divided by sustainable external bandwidth.
Symbols and units
- M_local
- Summed local storage capacity
bytes
- Mᵢ
- Usable capacity of local storage region i
bytes
- T_transfer
- External transfer-time lower bound
s
- Q_external
- Payload bytes crossing the external boundary
bytes
- B_external
- Sustainable external payload bandwidth
bytes/s
- i
- Local region index
integer index
Mi is local memory in bytes; Qexternal is external bytes moved and Bexternal is sustainable bytes/s.
Assumptions Summed capacity assumes usable placement; fragmentation and replication can lower usable model capacity.
Engineering example
Synthetic layout: four independent regions each move 1 TB/s locally, but a 0.2 TB/s external port still takes at least 5 ms to supply 1 GB. Summing local links cannot remove that boundary.
Distinctions and tradeoffs
A wafer-scale chip is not one uniformly accessible giant register, and aggregate on-chip bandwidth is not equivalent to HBM bandwidth at a package boundary.
Keeping communication local can reduce repeated off-chip movement, but the system must handle defects, thermal density and workloads that exceed local state capacity.
Model partitions can exploit a large connected fabric if the compiler balances compute, SRAM and routing rather than merely filling every processing element.
Specifications and interfaces
- Physical boundary
- Wafer-scale engine versus complete system
System memory, power and cooling are outside the compute-fabric outline.
- Bandwidth scope
- Aggregate local versus external
Many simultaneous local links cannot all serve one external transfer.
Interfaces
- Distributed local memory Input
- Weights, activations and routed messages
- System interfaces Output
- Model state exchange and completed tensor results
Check your understanding
Why is adding every local-link rate misleading for a single external data stream?
- The stream must cross its actual interface and topology cuts
- All local links form one external port
- Bandwidth and capacity are the same unit
Answer and reasoning
The stream must cross its actual interface and topology cuts
Aggregate rates describe parallel local transfers; a particular source-to-destination path is limited by the links and cuts it actually crosses.
Evidence and scope
WSE-3 announcement facts are vendor-reported. No proprietary defect map, yield, routing algorithm or production price is inferred.
Use the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism.