The mechanism
A multiply–accumulate operation updates a running sum with a product. Repeating this pattern computes a dot product, and many dot products form a matrix multiplication. In floating-point hardware, a fused multiply-add computes a product plus an addend with one final rounding rather than separately rounding the product first. That numerical distinction matters: mathematically equivalent evaluation orders need not produce identical finite-precision results. The atlas shows operands entering a product path and a retained sum returning through a feedback path. It represents the operation's dependencies, not a vendor's internal tensor-core layout. Precision, accumulation width, and throughput belong to a particular implementation and must be specified separately.
- Multiply the two current operand values.
- Combine their product with the accumulator according to the arithmetic format.
- Retain or forward the new sum and repeat for the next dot-product term.
Science and assumptions
Dot product and matrix entry
A dot product sums pairwise products. A matrix-product entry sums left-matrix row i times right-matrix column j over their shared index k.
Symbols and units
- y
- Dot-product output
product of operand units
- aₖ,bₖ
- Paired dot-product operands
declared operand units
- Cᵢⱼ
- Matrix-product entry at row i and column j
product of A and B element units
- Aᵢₖ
- Left-matrix element
A element units
- Bₖⱼ
- Right-matrix element
B element units
- i,j,k
- Valid row, column and reduction indices
integer indices
A matrix multiplication entry is a dot product. Repeated multiply–accumulate operations combine the contributions while the dataflow determines which values can be reused.
Assumptions Exact algebra shown; finite-precision evaluation order, rounding, overflow, and accumulation format affect computed values.
Engineering example
Worked exact integer dot product: [2, −1, 3] · [4, 5, 2] = 8 − 5 + 6 = 9. Starting at zero, the accumulator holds 8, then 3, then 9. This requires three multiplications and two nontrivial additions; a hardware convention may count three MACs as six operations.
Distinctions and tradeoffs
Counting a multiply–add as two FLOPs says nothing by itself about clock cycles, units, or achieved application throughput.
Low precision can reduce hardware and data costs. Numerical error and accumulation behavior require evaluation.
MAC patterns implement the dense linear algebra of AI. Reuse and parallel scheduling determine how efficiently the hardware stays busy.
Specifications and interfaces
- Update equation
- acc ← acc + a × b
Conceptual multiply–accumulate recurrence.
- FMA rounding
- One final rounding
Floating-point fused multiply-add semantic distinction.
- Operation count
- Often counted as 2 FLOPs
One multiply plus one add; not two clock cycles.
Interfaces
- a, b, and accumulator Input
- Operands in a defined numeric format
- Updated accumulator Output
- Partial or final dot-product result
Check your understanding
What distinguishes floating-point FMA from separately rounded multiply and add?
- FMA never produces rounding error
- FMA uses no multiplication
- FMA has one final rounding rather than an intermediate product rounding
Answer and reasoning
FMA has one final rounding rather than an intermediate product rounding
The fused operation preserves the product through the addition before rounding, which can change the numerical result.
Evidence and scope
The dot-product identity is exact; the linked floating-point discussion explains why FMA and separately rounded operations can differ. This integer example assumes enough range and does not demonstrate floating-point error.
Inspect row–column products and accumulated results in a small integer matrix; this does not simulate the individual gate network.