# Place the working set, not just the model Canonical: https://semiaifoundry.com/learn/atlas/memory-working-set-placement/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html Software · Atlas v1.3.0 # Place the working set, not just the model Assign tensor state to tiers using lifetime, mutation, reuse distance, access size and latency requirements. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#memory-working-set-placement)Register/SRAM ↔ HBM ↔ host ↔ proposed HBF ↔ storage; logical topology Learning objectiveAssign tensor state to tiers using lifetime, mutation, reuse distance, access size and latency requirements. ## The mechanism A model is not one undifferentiated memory allocation. Parameters, activation tiles, optimizer state, KV cache, routing metadata and checkpoints have different lifetimes and traffic. Registers and SRAM serve small, frequently reused local data; HBM supports the accelerator’s active read/write state; host DDR or LPDDR offers another capacity and compute boundary. NAND-based HBF is an emerging candidate for read-heavy near-compute state, while local and remote storage retain datasets, checkpoints and less active material. These are roles, not a universal fixed ranking: effective placement depends on the actual links, controller behavior, software and workload. 1. Inventory each tensor/state object with its precision, dimensions, lifetime and mutation rate. 2. Identify the deadline for its next use, expected reuse distance and bytes consumed per fetch. 3. Reserve fast-tier space for concurrent workspaces and hot mutable state before allocating the remaining capacity to weights. 4. Define transfer, prefetch, eviction and consistency rules for each tier boundary, including persistent copies. 5. Profile misses, spills, writes and tail latency; move the object or change batching rather than judging only aggregate capacity. ## Science and assumptions ### Tier-capacity constraint $$ \sum_i x_{i,k}S_i+O_k\leq C_k $$ Σi xi,k Si + Ok ≤ Ck Symbols and units xi,k Resident fraction/copy indicator of object i in tier k dimensionless Si Object size at its stored precision bytes Ok Workspace, staging and metadata overhead bytes Ck Usable tier capacity bytes For each tier k, resident objects plus reserved buffers and overhead must fit. x indicates the fraction or copy of object i resident there. Assumptions Use simultaneous live state, not sums of mutually exclusive tensors; copied buffers consume capacity in both tiers while alive. ## Engineering example Synthetic 80 GiB usable tier: 45 GiB weights + 20 GiB KV + 12 GiB workspaces + 5 GiB other live state = 82 GiB. At least 2 GiB must be removed or reduced before allowing additional staging margin. A larger cold tier does not directly enlarge local workspace. ## Distinctions and tradeoffs Placement must account for all simultaneously live state at the relevant precision and execution phase. Keeping more weights local can displace KV or workspaces; offloading capacity creates transfer and synchronization work. Memory placement connects physical packaging choices to batch size, context, latency and productive accelerator use. ## Specifications and interfaces Placement record Lifetime • reads/writes • reuse • granularity A parameter count alone is insufficient. Tier boundary Capacity, link, service and consistency Host-accessible and device-local are different properties. ### Interfaces Tensor contract Input Shape, precision, lifetime and required access pattern Runtime memory plan Output Residency, copies, transfer schedule and recovery state ## Check your understanding ### Why can a model fail to run even when its weight file fits in HBM? 1. Live KV, activations, workspaces and metadata also need capacity 2. Weights are the only memory allocation 3. Storage capacity fixes all device-memory limits Answer and reasoning Live KV, activations, workspaces and metadata also need capacity Placement must account for all simultaneously live state at the relevant precision and execution phase. ## Evidence and scope The placement framework is an engineering synthesis of established heterogeneous-memory and inference mechanisms. HBF placement remains a conditional scenario, not a claim that the cited runtimes support HBF. Place read-only weights and mutable KV across specified tiers and inspect transfer paths; no HBF shipping-system benchmark or universal memory semantics are implied. ## Sources - [NVIDIA — Mastering LLM Techniques: Inference Optimization](https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/) - [ZeRO-Infinity: Breaking the GPU Memory Wall](https://arxiv.org/abs/2104.07857) - [Sandisk — release of first OCP HBF specification](https://investor.sandisk.com/node/8126/pdf) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/memory-working-set-placement-3f85e16bc8008563.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Foundations - [Full memory hierarchy and input path](https://semiaifoundry.com/learn/atlas/memory-hierarchy-route/) - [Model graph and tensor intermediate representation](https://semiaifoundry.com/learn/atlas/graph-tensor-ir/) ### Related lessons - [Training state across HBM, host memory and storage](https://semiaifoundry.com/learn/atlas/training-tier-placement/) - [Prefill, decode and the moving KV working set](https://semiaifoundry.com/learn/atlas/prefill-decode-tier-placement/) - [MoE experts: resident capacity, routing and misses](https://semiaifoundry.com/learn/atlas/moe-expert-tier-placement/) - [Architecture is a contract between computation and silicon](https://semiaifoundry.com/learn/atlas/crg-architecture-contract/)