{"id":"memory-working-set-placement","level":"software","name":"Place the working set, not just the model","shortName":"Place the working set, not just the model","role":"Assign tensor state to tiers using lifetime, mutation, reuse distance, access size and latency requirements.","description":"A model is not one undifferentiated memory allocation. Parameters, activation tiles, optimizer state, KV cache, routing metadata and checkpoints have different lifetimes and traffic. Registers and SRAM serve small, frequently reused local data; HBM supports the accelerator’s active read/write state; host DDR or LPDDR offers another capacity and compute boundary. NAND-based HBF is an emerging candidate for read-heavy near-compute state, while local and remote storage retain datasets, checkpoints and less active material. These are roles, not a universal fixed ranking: effective placement depends on the actual links, controller behavior, software and workload.","mechanism":["Inventory each tensor/state object with its precision, dimensions, lifetime and mutation rate.","Identify the deadline for its next use, expected reuse distance and bytes consumed per fetch.","Reserve fast-tier space for concurrent workspaces and hot mutable state before allocating the remaining capacity to weights.","Define transfer, prefetch, eviction and consistency rules for each tier boundary, including persistent copies.","Profile misses, spills, writes and tail latency; move the object or change batching rather than judging only aggregate capacity."],"science":[{"title":"Tier-capacity constraint","equation":"Σi xi,k Si + Ok ≤ Ck","explanation":"For each tier k, resident objects plus reserved buffers and overhead must fit. x indicates the fraction or copy of object i resident there.","assumptions":"Use simultaneous live state, not sums of mutually exclusive tensors; copied buffers consume capacity in both tiers while alive."}],"specs":[{"label":"Placement record","value":"Lifetime • reads/writes • reuse • granularity","context":"A parameter count alone is insufficient."},{"label":"Tier boundary","value":"Capacity, link, service and consistency","context":"Host-accessible and device-local are different properties."}],"interfaces":[{"name":"Tensor contract","direction":"in","signal":"Shape, precision, lifetime and required access pattern"},{"name":"Runtime memory plan","direction":"out","signal":"Residency, copies, transfer schedule and recovery state"}],"tradeoff":"Keeping more weights local can displace KV or workspaces; offloading capacity creates transfer and synchronization work.","whyAI":"Memory placement connects physical packaging choices to batch size, context, latency and productive accelerator use.","prerequisites":["memory-hierarchy-route","graph-tensor-ir"],"related":["training-tier-placement","prefill-decode-tier-placement","moe-expert-tier-placement","crg-architecture-contract"],"misconception":"Placement must account for all simultaneously live state at the relevant precision and execution phase.","check":{"question":"Why can a model fail to run even when its weight file fits in HBM?","options":["Live KV, activations, workspaces and metadata also need capacity","Weights are the only memory allocation","Storage capacity fixes all device-memory limits"],"answer":0,"explanation":"Placement must account for all simultaneously live state at the relevant precision and execution phase."},"engineeringExample":"Synthetic 80 GiB usable tier: 45 GiB weights + 20 GiB KV + 12 GiB workspaces + 5 GiB other live state = 82 GiB. At least 2 GiB must be removed or reduced before allowing additional staging margin. A larger cold tier does not directly enlarge local workspace.","evidenceNotes":"The placement framework is an engineering synthesis of established heterogeneous-memory and inference mechanisms. HBF placement remains a conditional scenario, not a claim that the cited runtimes support HBF.","sources":[{"title":"NVIDIA — Mastering LLM Techniques: Inference Optimization","url":"https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/"},{"title":"ZeRO-Infinity: Breaking the GPU Memory Wall","url":"https://arxiv.org/abs/2104.07857"},{"title":"Sandisk — release of first OCP HBF specification","url":"https://investor.sandisk.com/node/8126/pdf"}],"learningObjective":"Assign tensor state to tiers using lifetime, mutation, reuse distance, access size and latency requirements.","workstream":"W11","visualFamily":"memory-route","scale":"Register/SRAM ↔ HBM ↔ host ↔ proposed HBF ↔ storage; logical topology","kind":"system","labId":"inference-placement","branch":"Software","subbranch":"Working-set placement across memory tiers","scope":"Place read-only weights and mutable KV across specified tiers and inspect transfer paths; no HBF shipping-system benchmark or universal memory semantics are implied.","investigationId":"execution","labScope":"Place read-only weights and mutable KV across specified tiers and inspect transfer paths; no HBF shipping-system benchmark or universal memory semantics are implied."}