The mechanism
A systolic array moves operands rhythmically between neighboring processing elements. Instead of repeatedly fetching every pair from a distant memory, each value contributes to several multiply–accumulate operations as it crosses the array. A dataflow choice determines whether weights, inputs or partial sums remain stationary for a tile. Filling and draining the array creates overhead, especially for small or awkward matrix shapes. Google’s original TPU is a disclosed matrix-engine example; TPU v4 adds system and embedding support. The array is only part of the machine: nonlinear operations, sparse lookups, compiler scheduling and communication still require other resources.
- Tile the matrix multiplication so operands and partial sums fit the target buffers and array shape.
- Skew the entering streams so each processing element receives the matching operands at the matching cycle.
- Multiply and accumulate while forwarding reusable operands to neighboring elements.
- Drain completed partial sums, accounting for edge tiles and bubbles during startup or shape changes.
- Schedule vector, lookup and communication operations around the array so non-matrix work does not strand its capacity.
Science and assumptions
Dense matrix work
Dense matrix multiplication takes approximately two times output rows times output columns times reduction width floating-point operations.
Symbols and units
- F
- Dense matrix-product arithmetic work
FLOP
- m
- Rows in the left matrix and output
elements
- n
- Columns in the right matrix and output
elements
- k
- Reduction dimension shared by the inputs
elements
For A of shape m×k and B of shape k×n, F counts multiply and add separately. Dimensions count elements, and F counts arithmetic operations.
Assumptions Large dense matrices; exact edge counting differs slightly and sparse execution needs a separate convention.
Engineering example
Synthetic multiplication of two 4×4 matrices contains 64 products and approximately 128 operations by the two-operation MAC convention. An animation should track one operand’s reuse rather than pretend that every product requires a fresh HBM read.
Distinctions and tradeoffs
A tensor accelerator need not be one universal systolic design, and optical links in TPU v4 do not mean its arithmetic is optical.
Regular local traffic can be efficient for dense matrices but mismatched shapes and irregular indexing demand padding, remapping or another engine.
Transformer projections and feed-forward layers contain reusable matrix work, while embeddings and token-dependent control expose the limits of a matrix-only explanation.
Specifications and interfaces
- Operation convention
- One multiply plus one add = two operations
Use this convention only when explicitly comparing compatible arithmetic.
- Tile utilization
- Useful array cells and cycles
Shape padding and fill/drain cycles reduce realized throughput.
Interfaces
- Tile buffers Input
- Ordered operand streams and scheduling metadata
- Accumulator or result buffer Output
- Partial or completed matrix tiles
Check your understanding
What chiefly allows an array to reduce distant memory traffic?
- Operand reuse through local connections
- Ignoring matrix dependencies
- Using the largest possible advertised FLOPS number
Answer and reasoning
Operand reuse through local connections
Values can participate in several products after entering the array, avoiding repeated fetches from the selected outer memory level.
Evidence and scope
TPU generations are named separately. This teaching array is not a transistor-accurate reconstruction or a cross-vendor peak comparison.
Compare unblocked and tiled matrix traffic for fixed arithmetic; architecture-specific instructions, fusion and attention algorithms are context rather than simulated implementations.