The mechanism
A Tensor Core implements matrix multiply and accumulate: D equals A times B plus C. It does not contain a neural network, words, or a complete attention head. Software divides larger tensors into compatible tiles, arranges their layout, and feeds these units repeatedly. Different instructions accept different input formats and accumulation rules. Low precision reduces the bits moved and processed, while scale factors help represent useful numerical ranges. A peak throughput figure assumes suitable operations, shapes, clocks, and a continuous supply of operands. A matrix tile combines row and column elements into partial products, accumulates their sums, and contributes results to a larger matrix. That arithmetic relationship describes the operation, not a literal photograph of a Tensor Core layout.
- Software divides a large matrix product into compatible tiles and selects a supported input format, layout, and scaling scheme.
- The tensor instruction multiplies corresponding tile elements and accumulates partial sums according to its numerical rules.
- Partial tiles are combined or reused, then written into the larger output tensor for the next operation.
Science and assumptions
Why matrix size and reuse matter
Dense matrix work is approximately two times m times n times k. Ideal intensity divides that work by the bytes needed to read both inputs once and write the output once.
Symbols and units
- F
- Dense matrix-product arithmetic count
FLOP
- m
- Rows of A and C
elements
- n
- Columns of B and C
elements
- k
- Shared reduction dimension
elements
- b
- Stored size per matrix element
bytes/element
- I_ideal
- Intensity when A and B are read once and C written once
FLOP/byte
A has m rows and k columns; B has k rows and n columns. Their product needs approximately mnk fused multiply-adds, conventionally counted as two floating-point operations each. If all elements occupy b bytes, the denominator describes reading A and B once and writing the output once. Increasing reuse can make compute grow faster than external data movement.
Assumptions Ideal dense C = AB traffic, with each matrix transferred once and no initial C read. Accumulation precision can differ in real kernels; a D = AB + C operation must include its extra C traffic. This is an arithmetic model, not measured Tensor Core efficiency.
Engineering example
For m×k by k×n, the multiply-accumulate count is 2mnk; operand reuse determines whether the tile reaches a compute bound.
Distinctions and tradeoffs
Tensor Core sparse peak performance is not a dense-model guarantee. The supported sparsity pattern and appropriate instructions are required to realize the sparse rating.
Lower precision can raise throughput and reduce memory traffic, but numerical error must be controlled. Sparse peak rates require the supported sparsity pattern and cannot be applied to arbitrary dense weights.
Attention projections and feed-forward layers are dominated by matrix products. Efficient tensor kernels turn this recurring mathematical structure into much more work per joule.
Specifications and interfaces
- Operation
- D = A × B + C
Matrix multiply-accumulate, or MMA.
- Generation
- Fifth-generation Tensor Cores
Datacenter Blackwell; supported instruction details depend on architecture target.
- NVFP4 pair peak
- 40 sparse / 20 dense PFLOPS
Official GB200 Superchip table: two GPUs, theoretical tensor peak.
Interfaces
- Operands Input
- A and B tiles plus scale metadata
- Accumulator Input
- C tile
- Matrix result Output
- D tile
Check your understanding
What does a Tensor Core primarily accelerate?
- Storing model files after power is removed.
- Matrix multiply-accumulate operations on compatible tiles.
- Replacing the CPU operating system.
Answer and reasoning
Matrix multiply-accumulate operations on compatible tiles.
Neural-network projections are built from large matrix products. Tensor Cores accelerate small pieces of those products, which software assembles into the complete result.
Historical context
2017 · Volta introduces NVIDIA Tensor Cores
NVIDIA announced Volta and Tesla V100 on May 10, 2017. Dedicated Tensor Cores accelerated matrix arithmetic, aligning silicon execution more directly with the multiply-and-accumulate structure of deep learning instead of relying only on general-purpose arithmetic lanes.
Evidence and scope
Tile dimensions and accumulation precision are configuration choices.
Inspect row–column products and accumulated results in a small integer matrix; this does not simulate the individual gate network.