The mechanism
Tensor Memory is a Blackwell storage space specialized for Tensor Core operations. It is neither HBM nor the Tensor Memory Accelerator. Supported instructions allocate columns, move suitable fragments, launch matrix operations, and later retrieve results. Giving tensor operations their own storage changes how software pipelines matrix multiplication and manages register pressure. The programming model has explicit access and allocation rules, so the whole array is not a freely interchangeable replacement for ordinary shared memory. Column allocation and instruction layouts determine which tensor fragments can remain in this storage across successive matrix operations. The documented shape is an architectural address view, not a photograph or map of physical memory cells.
- A supported kernel allocates columns in Tensor Memory and arranges tensor fragments according to the instruction’s required layout.
- Tensor Core operations consume operands and update accumulators using this specialized nearby storage.
- The kernel synchronizes, extracts completed results through supported instructions, and releases the allocation when it is no longer needed.
Science and assumptions
Logical shape determines storage capacity
The documented array has 128 rows by 512 columns with 32 bits per cell: 2,097,152 bits, or 262,144 bytes, or 256 KiB.
Eight bits make one byte. This is the architectural array capacity, not a per-block allocation guarantee.
Symbols and units
- C
- Capacity of the documented logical Tensor Memory array
bytes
- 128
- Logical row count
rows
- 512
- Logical column count
columns
- 32 bits
- Storage per logical cell
bits/cell
- KiB
- Binary kibibyte
1024 bytes
The documented address view has 128 rows, 512 columns, and 32 bits per cell. Dividing bits by eight gives bytes. This organization explains why column allocation reserves an entire vertical set of cells and why instruction layouts govern access. The number describes the architectural array, not a claim that every simultaneously resident block can consume an independent full-size copy.
Assumptions Logical Tensor Memory organization from the PTX ISA. Allocation, residency, and access rules still apply. KiB uses 1,024 bytes, distinguishing it from decimal KB.
Engineering example
Using the lesson’s documented 128-row, 32-bit-cell address view, a hypothetical allocation of 32 columns reserves 128 × 32 × 32 / 8 = 16,384 bytes = 16 KiB. The full 512-column view is 256 KiB; that does not grant every resident block an independent full array.
Distinctions and tradeoffs
Tensor Memory is storage; the Tensor Memory Accelerator is a copy engine. Similar names describe different parts of the data path.
Dedicated storage relieves some pressure on general registers, but introduces layout and allocation constraints. Efficient kernels must overlap movement and compute while respecting synchronization rules.
Large matrix products repeatedly update partial sums before storing final values. Keeping these accumulators next to the tensor machinery improves the opportunity for reuse.
Specifications and interfaces
- Address organization
- 128 rows × 512 columns
PTX sm_100a/sm_100f Tensor Memory view.
- Cell size
- 32 bits
Total logical array is 256 KiB by direct multiplication.
- Allocation unit
- 32 columns
Allocated column counts are powers of two under the documented rules.
Interfaces
- Tile transfer Input
- Supported tensor fragments
- Tensor execution Output
- Operands and accumulation storage
- Result extraction Output
- Tensor Memory loads into thread registers
Check your understanding
What is Tensor Memory’s main role?
- Converting AC power into DC power.
- Holding specialized Tensor Core operands and accumulators on the chip.
- Routing Ethernet packets between racks.
Answer and reasoning
Holding specialized Tensor Core operands and accumulators on the chip.
TMEM is an architectural storage space used by fifth-generation Tensor Core instructions. It is not a data center memory appliance or a copy engine.
Historical context
2025 · CUTLASS 3.8 exposes Blackwell Tensor Memory
The February 2025 CUTLASS 3.8 release added Blackwell SM100 support and exposed Tensor Memory as a distinct storage location, along with operations that move values between registers, shared memory, and Tensor Memory.
Evidence and scope
The logical dimensions are those cited from PTX ISA 8.8 for The 32-column allocation is an arithmetic example, subject to actual allocation and residency rules; it is not a physical-cell layout or a portable GPU feature.
No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.