{"id":"tensor-memory-accelerator","level":"die","name":"Tensor Memory Accelerator","shortName":"TMA","role":"Moves multidimensional tensor tiles asynchronously so compute threads can spend less effort copying data.","description":"TMA is a data movement engine. Software describes a tensor and its layout, then requests transfers while other work proceeds. This reduces the need for threads to compute every address and issue every individual load during tile staging. The result is not instantaneous memory: the transfer still uses bandwidth, must complete, and needs the correct synchronization before consumers use its data. Pipelines can stage the next tile while the current tile is being multiplied. First introduced with Hopper and extended on Blackwell, this mechanism is separate from Blackwell Tensor Memory, which stores tensor operands and accumulators. The copy engine, destination buffer and completion signal have distinct roles: initiating a transfer does not make its destination ready for consumers.","specs":[{"label":"Tensor rank","value":"1D through 5D transfers","context":"Hopper TMA programming model retained and extended by later architectures."},{"label":"Main path","value":"Global memory ↔ shared memory","context":"Supported paths and instructions depend on architecture."},{"label":"Operation style","value":"Asynchronous producer/consumer","context":"Synchronization establishes when a copied tile is safe to read."}],"interfaces":[{"name":"Descriptor and request","direction":"in","signal":"Tensor layout and tile coordinates"},{"name":"Memory transfer","direction":"out","signal":"Bulk tile movement"},{"name":"Completion event","direction":"out","signal":"Barrier or synchronization update"}],"tradeoff":"Overlapping copies with arithmetic can hide movement time but requires buffers for in-flight work. More pipeline stages consume shared memory and can reduce parallel residency.","whyAI":"Transformer kernels repeatedly stage blocks of weights and activations. A copy engine frees execution resources and makes sustained matrix throughput easier to approach.","related":["shared-sram","tensor-core","tensor-memory","hbm-controller"],"sources":[{"title":"NVIDIA Hopper Tuning Guide: Tensor Memory Accelerator","url":"https://docs.nvidia.com/cuda/archive/12.1.0/pdf/Hopper_Tuning_Guide.pdf"},{"title":"NVIDIA CUTLASS 3.8 release","url":"https://github.com/NVIDIA/cutlass/discussions/2125"},{"title":"NVIDIA CUDA Programming Guide: asynchronous data copies","url":"https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/async-copies.html"},{"title":"NVIDIA: Hopper Architecture In-Depth","url":"https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/"}],"scale":"Conceptual engine and transfer paths within the GPU","kind":"asynchronous-tile-pipeline","mechanism":["Software describes a tensor’s dimensions, layout, and the tile that should be transferred.","TMA performs an asynchronous bulk copy so compute work can proceed while the tile moves into the appropriate buffer.","A completion signal tells consumers when it is safe to use the tile, enabling a pipeline of loading and computation."],"misconception":"Asynchronous movement is not instantaneous movement. Copies still consume bandwidth and need correct synchronization before their data can be used.","check":{"question":"How does a pipelined kernel benefit from TMA?","options":["It can prepare a future tile while computing on the current tile.","It eliminates the need for HBM or shared memory.","It makes every model use the same matrix dimensions."],"answer":0,"explanation":"The gain comes from overlapping movement and computation and reducing thread work spent on copying, subject to bandwidth and buffering limits."},"branch":"Compute architecture","subbranch":"Asynchronous data movement","science":[{"title":"Overlap changes a sum into a bottleneck","equation":"Tpipeline ≈ tload + tcompute + (N − 1) × max(tload, tcompute)","explanation":"N is the number of equal tiles. After filling the first stage, a two-stage pipeline can load the next tile while computing on the current one. The slower stage sets the steady-state cadence. Without overlap, the same sequence takes approximately N × (tload + tcompute), so copy/compute overlap can improve throughput without increasing the arithmetic rate.","assumptions":"Ideal independent load and compute resources, adequate double buffering, equal tile times, and negligible synchronization overhead. Shared bandwidth or resource contention can reduce or remove overlap. TMA remains a copy engine, distinct from Tensor Memory storage."}],"history":[{"year":2022,"title":"Hopper introduces Tensor Memory Accelerator","significance":"Hopper added an engine for moving tensor tiles asynchronously between global and shared memory, allowing threads and tensor arithmetic to overlap useful work with data transfer.","source":{"title":"NVIDIA: Hopper Architecture In-Depth","url":"https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/"}}],"prerequisites":["shared-sram","hbm-controller"],"learningObjective":"Calculate ideal tile-pipeline latency and explain why asynchronous copying still requires completion and synchronization.","engineeringExample":"Synthetic four-tile pipeline: each load takes 3 μs and each compute stage 5 μs. Serial execution takes 4 × (3 + 5) = 32 μs. Ideal double buffering takes 3 + 5 + 3 × max(3, 5) = 23 μs, before synchronization and contention.","evidenceNotes":"The overlap model is the lesson’s ideal two-stage pipeline, supported by the linked asynchronous-copy guidance. TMA moves data; Tensor Memory stores it. Adequate buffering and independent resources are assumptions, not guaranteed speedup.","labId":"library","scope":"No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.","investigationId":"execution","labScope":"No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions."}