The mechanism
A model is not one undifferentiated memory allocation. Parameters, activation tiles, optimizer state, KV cache, routing metadata and checkpoints have different lifetimes and traffic. Registers and SRAM serve small, frequently reused local data; HBM supports the accelerator’s active read/write state; host DDR or LPDDR offers another capacity and compute boundary. NAND-based HBF is an emerging candidate for read-heavy near-compute state, while local and remote storage retain datasets, checkpoints and less active material. These are roles, not a universal fixed ranking: effective placement depends on the actual links, controller behavior, software and workload.
- Inventory each tensor/state object with its precision, dimensions, lifetime and mutation rate.
- Identify the deadline for its next use, expected reuse distance and bytes consumed per fetch.
- Reserve fast-tier space for concurrent workspaces and hot mutable state before allocating the remaining capacity to weights.
- Define transfer, prefetch, eviction and consistency rules for each tier boundary, including persistent copies.
- Profile misses, spills, writes and tail latency; move the object or change batching rather than judging only aggregate capacity.
Science and assumptions
Tier-capacity constraint
Σi xi,k Si + Ok ≤ Ck
Symbols and units
- xi,k
- Resident fraction/copy indicator of object i in tier k
dimensionless
- Si
- Object size at its stored precision
bytes
- Ok
- Workspace, staging and metadata overhead
bytes
- Ck
- Usable tier capacity
bytes
For each tier k, resident objects plus reserved buffers and overhead must fit. x indicates the fraction or copy of object i resident there.
Assumptions Use simultaneous live state, not sums of mutually exclusive tensors; copied buffers consume capacity in both tiers while alive.
Engineering example
Synthetic 80 GiB usable tier: 45 GiB weights + 20 GiB KV + 12 GiB workspaces + 5 GiB other live state = 82 GiB. At least 2 GiB must be removed or reduced before allowing additional staging margin. A larger cold tier does not directly enlarge local workspace.
Distinctions and tradeoffs
Placement must account for all simultaneously live state at the relevant precision and execution phase.
Keeping more weights local can displace KV or workspaces; offloading capacity creates transfer and synchronization work.
Memory placement connects physical packaging choices to batch size, context, latency and productive accelerator use.
Specifications and interfaces
- Placement record
- Lifetime • reads/writes • reuse • granularity
A parameter count alone is insufficient.
- Tier boundary
- Capacity, link, service and consistency
Host-accessible and device-local are different properties.
Interfaces
- Tensor contract Input
- Shape, precision, lifetime and required access pattern
- Runtime memory plan Output
- Residency, copies, transfer schedule and recovery state
Check your understanding
Why can a model fail to run even when its weight file fits in HBM?
- Live KV, activations, workspaces and metadata also need capacity
- Weights are the only memory allocation
- Storage capacity fixes all device-memory limits
Answer and reasoning
Live KV, activations, workspaces and metadata also need capacity
Placement must account for all simultaneously live state at the relevant precision and execution phase.
Evidence and scope
The placement framework is an engineering synthesis of established heterogeneous-memory and inference mechanisms. HBF placement remains a conditional scenario, not a claim that the cited runtimes support HBF.
Place read-only weights and mutable KV across specified tiers and inspect transfer paths; no HBF shipping-system benchmark or universal memory semantics are implied.