# Systolic arrays and tensor dataflow

Canonical: https://semiaifoundry.com/learn/atlas/systolic-array-dataflow/
Revision: 2026-10-07
Rights: https://semiaifoundry.com/ai-atlas/licenses.html

Processing architectures · Atlas v1.3.0

# Systolic arrays and tensor dataflow

Reuses operands through a regular network of multiply–accumulate elements.

[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#systolic-array-dataflow)Processing element to accelerator package; array tiles exaggerated

Learning objectiveRelate a matrix tiling and dataflow choice to reuse, filling and array utilization.

## The mechanism

A systolic array moves operands rhythmically between neighboring processing elements. Instead of repeatedly fetching every pair from a distant memory, each value contributes to several multiply–accumulate operations as it crosses the array. A dataflow choice determines whether weights, inputs or partial sums remain stationary for a tile. Filling and draining the array creates overhead, especially for small or awkward matrix shapes. Google’s original TPU is a disclosed matrix-engine example; TPU v4 adds system and embedding support. The array is only part of the machine: nonlinear operations, sparse lookups, compiler scheduling and communication still require other resources.

1.  Tile the matrix multiplication so operands and partial sums fit the target buffers and array shape.
2.  Skew the entering streams so each processing element receives the matching operands at the matching cycle.
3.  Multiply and accumulate while forwarding reusable operands to neighboring elements.
4.  Drain completed partial sums, accounting for edge tiles and bubbles during startup or shape changes.
5.  Schedule vector, lookup and communication operations around the array so non-matrix work does not strand its capacity.

## Science and assumptions

### Dense matrix work

$$
F\approx2mnk
$$

Dense matrix multiplication takes approximately two times output rows times output columns times reduction width floating-point operations.

Symbols and units

F

Dense matrix-product arithmetic work

FLOP

m

Rows in the left matrix and output

elements

n

Columns in the right matrix and output

elements

k

Reduction dimension shared by the inputs

elements

For A of shape m×k and B of shape k×n, F counts multiply and add separately. Dimensions count elements, and F counts arithmetic operations.

Assumptions Large dense matrices; exact edge counting differs slightly and sparse execution needs a separate convention.

## Engineering example

Synthetic multiplication of two 4×4 matrices contains 64 products and approximately 128 operations by the two-operation MAC convention. An animation should track one operand’s reuse rather than pretend that every product requires a fresh HBM read.

## Distinctions and tradeoffs

A tensor accelerator need not be one universal systolic design, and optical links in TPU v4 do not mean its arithmetic is optical.

Regular local traffic can be efficient for dense matrices but mismatched shapes and irregular indexing demand padding, remapping or another engine.

Transformer projections and feed-forward layers contain reusable matrix work, while embeddings and token-dependent control expose the limits of a matrix-only explanation.

## Specifications and interfaces

Operation convention

One multiply plus one add = two operations

Use this convention only when explicitly comparing compatible arithmetic.

Tile utilization

Useful array cells and cycles

Shape padding and fill/drain cycles reduce realized throughput.

### Interfaces

Tile buffers Input

Ordered operand streams and scheduling metadata

Accumulator or result buffer Output

Partial or completed matrix tiles

## Check your understanding

### What chiefly allows an array to reduce distant memory traffic?

1.  Operand reuse through local connections
2.  Ignoring matrix dependencies
3.  Using the largest possible advertised FLOPS number

Answer and reasoning

Operand reuse through local connections

Values can participate in several products after entering the array, avoiding repeated fetches from the selected outer memory level.

## Evidence and scope

TPU generations are named separately. This teaching array is not a transistor-accurate reconstruction or a cross-vendor peak comparison.

Compare unblocked and tiled matrix traffic for fixed arithmetic; architecture-specific instructions, fusion and attention algorithms are context rather than simulated implementations.

## Sources

-   [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://arxiv.org/abs/1704.04760)
-   [TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings](https://arxiv.org/abs/2304.01433v3)

[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/systolic-array-dataflow-8d5f972ce8b5a1fc.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)

## Continue exploring

### Foundations

-   [Multiply–accumulate and fused arithmetic](https://semiaifoundry.com/learn/atlas/mac/)
-   [Shared memory SRAM](https://semiaifoundry.com/learn/atlas/shared-sram/)

### Related lessons

-   [Tensor Core matrix unit](https://semiaifoundry.com/learn/atlas/tensor-core/)
-   [Trainium and Inferentia execution paths](https://semiaifoundry.com/learn/atlas/aws-neuron-dataflow/)