{"id":"systolic-array-dataflow","level":"compute","name":"Systolic arrays and tensor dataflow","shortName":"Systolic arrays and tensor dataflow","role":"Reuses operands through a regular network of multiply–accumulate elements.","description":"A systolic array moves operands rhythmically between neighboring processing elements. Instead of repeatedly fetching every pair from a distant memory, each value contributes to several multiply–accumulate operations as it crosses the array. A dataflow choice determines whether weights, inputs or partial sums remain stationary for a tile. Filling and draining the array creates overhead, especially for small or awkward matrix shapes. Google’s original TPU is a disclosed matrix-engine example; TPU v4 adds system and embedding support. The array is only part of the machine: nonlinear operations, sparse lookups, compiler scheduling and communication still require other resources.","specs":[{"label":"Operation convention","value":"One multiply plus one add = two operations","context":"Use this convention only when explicitly comparing compatible arithmetic."},{"label":"Tile utilization","value":"Useful array cells and cycles","context":"Shape padding and fill/drain cycles reduce realized throughput."}],"interfaces":[{"name":"Tile buffers","direction":"in","signal":"Ordered operand streams and scheduling metadata"},{"name":"Accumulator or result buffer","direction":"out","signal":"Partial or completed matrix tiles"}],"tradeoff":"Regular local traffic can be efficient for dense matrices but mismatched shapes and irregular indexing demand padding, remapping or another engine.","whyAI":"Transformer projections and feed-forward layers contain reusable matrix work, while embeddings and token-dependent control expose the limits of a matrix-only explanation.","related":["mac","tensor-core","aws-neuron-dataflow"],"sources":[{"title":"In-Datacenter Performance Analysis of a Tensor Processing Unit","url":"https://arxiv.org/abs/1704.04760"},{"title":"TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings","url":"https://arxiv.org/abs/2304.01433v3"}],"scale":"Processing element to accelerator package; array tiles exaggerated","kind":"architecture","branch":"Processing architectures","subbranch":"Explicit dataflow and custom accelerators","mechanism":["Tile the matrix multiplication so operands and partial sums fit the target buffers and array shape.","Skew the entering streams so each processing element receives the matching operands at the matching cycle.","Multiply and accumulate while forwarding reusable operands to neighboring elements.","Drain completed partial sums, accounting for edge tiles and bubbles during startup or shape changes.","Schedule vector, lookup and communication operations around the array so non-matrix work does not strand its capacity."],"misconception":"A tensor accelerator need not be one universal systolic design, and optical links in TPU v4 do not mean its arithmetic is optical.","check":{"question":"What chiefly allows an array to reduce distant memory traffic?","options":["Operand reuse through local connections","Ignoring matrix dependencies","Using the largest possible advertised FLOPS number"],"answer":0,"explanation":"Values can participate in several products after entering the array, avoiding repeated fetches from the selected outer memory level."},"science":[{"title":"Dense matrix work","equation":"F ≈ 2mnk","explanation":"For A of shape m×k and B of shape k×n, F counts multiply and add separately. Dimensions count elements, and F counts arithmetic operations.","assumptions":"Large dense matrices; exact edge counting differs slightly and sparse execution needs a separate convention."}],"prerequisites":["mac","shared-sram"],"learningObjective":"Relate a matrix tiling and dataflow choice to reuse, filling and array utilization.","workstream":"W07","visualFamily":"architecture-array","engineeringExample":"Synthetic multiplication of two 4×4 matrices contains 64 products and approximately 128 operations by the two-operation MAC convention. An animation should track one operand’s reuse rather than pretend that every product requires a fresh HBM read.","evidenceNotes":"TPU generations are named separately. This teaching array is not a transistor-accurate reconstruction or a cross-vendor peak comparison.","labId":"engineering","scope":"Compare unblocked and tiled matrix traffic for fixed arithmetic; architecture-specific instructions, fusion and attention algorithms are context rather than simulated implementations.","deepLabId":"dataflow","investigationId":"execution","labScope":"Compare unblocked and tiled matrix traffic for fixed arithmetic; architecture-specific instructions, fusion and attention algorithms are context rather than simulated implementations."}