{"id":"tensor-memory","level":"die","name":"Tensor Memory","shortName":"TMEM","role":"Provides dedicated on-chip storage for fifth-generation Tensor Core operands and accumulators.","description":"Tensor Memory is a Blackwell storage space specialized for Tensor Core operations. It is neither HBM nor the Tensor Memory Accelerator. Supported instructions allocate columns, move suitable fragments, launch matrix operations, and later retrieve results. Giving tensor operations their own storage changes how software pipelines matrix multiplication and manages register pressure. The programming model has explicit access and allocation rules, so the whole array is not a freely interchangeable replacement for ordinary shared memory. Column allocation and instruction layouts determine which tensor fragments can remain in this storage across successive matrix operations. The documented shape is an architectural address view, not a photograph or map of physical memory cells.","specs":[{"label":"Address organization","value":"128 rows × 512 columns","context":"PTX sm_100a/sm_100f Tensor Memory view."},{"label":"Cell size","value":"32 bits","context":"Total logical array is 256 KiB by direct multiplication."},{"label":"Allocation unit","value":"32 columns","context":"Allocated column counts are powers of two under the documented rules."}],"interfaces":[{"name":"Tile transfer","direction":"in","signal":"Supported tensor fragments"},{"name":"Tensor execution","direction":"out","signal":"Operands and accumulation storage"},{"name":"Result extraction","direction":"out","signal":"Tensor Memory loads into thread registers"}],"tradeoff":"Dedicated storage relieves some pressure on general registers, but introduces layout and allocation constraints. Efficient kernels must overlap movement and compute while respecting synchronization rules.","whyAI":"Large matrix products repeatedly update partial sums before storing final values. Keeping these accumulators next to the tensor machinery improves the opportunity for reuse.","related":["tensor-core","shared-sram","register-file","tensor-memory-accelerator"],"sources":[{"title":"NVIDIA PTX ISA 8.8: fifth-generation Tensor Core instructions","url":"https://docs.nvidia.com/cuda/pdf/ptx_isa_8.8.pdf"},{"title":"NVIDIA CUTLASS 3.8 release","url":"https://github.com/NVIDIA/cutlass/discussions/2125"}],"scale":"Logical 128 × 512 address array inside an SM","kind":"allocated-tensor-memory","mechanism":["A supported kernel allocates columns in Tensor Memory and arranges tensor fragments according to the instruction’s required layout.","Tensor Core operations consume operands and update accumulators using this specialized nearby storage.","The kernel synchronizes, extracts completed results through supported instructions, and releases the allocation when it is no longer needed."],"misconception":"Tensor Memory is storage; the Tensor Memory Accelerator is a copy engine. Similar names describe different parts of the data path.","check":{"question":"What is Tensor Memory’s main role?","options":["Converting AC power into DC power.","Holding specialized Tensor Core operands and accumulators on the chip.","Routing Ethernet packets between racks."],"answer":1,"explanation":"TMEM is an architectural storage space used by fifth-generation Tensor Core instructions. It is not a data center memory appliance or a copy engine."},"branch":"Memory hierarchy","subbranch":"On-chip storage","science":[{"title":"Logical shape determines storage capacity","equation":"C = 128 × 512 × 32 / 8 = 262,144 bytes = 256 KiB","explanation":"The documented sm_100a/sm_100f address view has 128 rows, 512 columns, and 32 bits per cell. Dividing bits by eight gives bytes. This organization explains why column allocation reserves an entire vertical set of cells and why instruction layouts govern access. The number describes the architectural array, not a claim that every simultaneously resident block can consume an independent full-size copy.","assumptions":"Logical Tensor Memory organization from the PTX ISA. Allocation, residency, and access rules still apply. KiB uses 1,024 bytes, distinguishing it from decimal KB."}],"history":[{"year":2025,"title":"CUTLASS 3.8 exposes Blackwell Tensor Memory","significance":"The February 2025 CUTLASS 3.8 release added Blackwell SM100 support and exposed Tensor Memory as a distinct storage location, along with operations that move values between registers, shared memory, and Tensor Memory.","source":{"title":"NVIDIA CUTLASS 3.8 release discussion","url":"https://github.com/NVIDIA/cutlass/discussions/2125"}}],"prerequisites":["tensor-core","shared-sram"],"learningObjective":"Convert a documented logical Tensor Memory allocation into bytes without confusing it with HBM or TMA.","engineeringExample":"Using the lesson’s documented 128-row, 32-bit-cell address view, a hypothetical allocation of 32 columns reserves 128 × 32 × 32 / 8 = 16,384 bytes = 16 KiB. The full 512-column view is 256 KiB; that does not grant every resident block an independent full array.","evidenceNotes":"The logical dimensions are those cited from PTX ISA 8.8 for sm_100a/sm_100f. The 32-column allocation is an arithmetic example, subject to actual allocation and residency rules; it is not a physical-cell layout or a portable GPU feature.","labId":"library","scope":"No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.","investigationId":"execution","labScope":"No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions."}