# Prefill, decode and the moving KV working set Canonical: https://semiaifoundry.com/learn/atlas/prefill-decode-tier-placement/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html Software · Atlas v1.3.0 # Prefill, decode and the moving KV working set Place active and reusable inference state according to the execution phase and service deadline. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#prefill-decode-tier-placement)Prefill output, live decode state and tier-crossing transfers; abstraction Learning objectivePlace active and reusable inference state according to the execution phase and service deadline. ## The mechanism Prefill processes an input sequence and creates state for subsequent generation. Decode repeatedly generates new tokens while reading prior context and appending new KV state. Weights are reused across requests, while KV belongs to a model/version, sequence position and attention configuration. A reusable prefix can be retained or offloaded, but restoring it competes with recomputing it. Moving large KV payloads between prefill and decode nodes adds network work and coordination. Larger near-compute capacity can help only if the state is available before the next dependent operation and the placement preserves correctness. 1. Identify whether the workload is prefill, decode or a mixture; record batch, context and attention architecture. 2. Budget the live KV payload and append/write behavior separately from weight reads. 3. Keep imminent mutable state accessible; choose a policy for inactive prefixes, eviction and restoration. 4. Compare restore transfer time with recomputation and service deadlines under actual contention. 5. For split serving, trace KV over its physical links and validate model/version, ownership and completion before decode uses it. ## Science and assumptions ### Conventional KV payload $$ M_{\mathrm{KV}}=2LH_{\mathrm{KV}}db\sum_s T_s $$ MKV = 2 L Hkv d b Σs Ts Symbols and units MKV Raw live KV payload bytes L Layers retaining KV count Hkv KV heads per layer count d Values per KV head count b Bytes per stored value bytes Ts Stored tokens in sequence s tokens Each stored token has key and value vectors for the specified layers and KV heads. Assumptions Full conventional KV representation; no sliding windows, compression or architecture-specific state reduction. Excludes paging overhead and padding. ### Restore or recompute $$ t_{\mathrm{restore}}\geq L_{\mathrm{path}}+\frac{V}{B_{\mathrm{path}}} $$ trestore ≥ Lpath + V/Bpath Symbols and units trestore State-restoration time s Lpath Path setup/completion delay s V Bytes restored bytes Bpath Sustained path payload bandwidth bytes/s Compare this transfer lower bound with measured recomputation time and the request deadline. Assumptions One aggregate transfer; includes no queueing or validation cost and assumes the cache is valid. ## Engineering example Synthetic conventional KV: 32 layers × 8 KV heads × 128 values × 2 bytes × 2 for K/V × 8,192 tokens is 1 GiB per sequence. Eight such sequences need 8 GiB before metadata and other model state. ## Distinctions and tradeoffs Active decoding both reads old context and appends state; placement must match those access requirements. More cached context saves recomputation but occupies memory and creates movement when placed remotely; a cache hit still has a service cost. Memory tiers determine concurrency and context support, while the restore path can affect first-token and inter-token latency. ## Specifications and interfaces Decode Repeated reads and KV appends An active KV cache is not immutable pretrained weight storage. Prefix reuse Must retain semantic validity Model, token sequence and attention configuration matter. ### Interfaces Prefill Input Prompt computation and newly produced KV Decode Output Ready prior state, appended KV and output token ## Check your understanding ### Does a large read-oriented tier automatically suit the active KV cache? 1. No: append writes, access granularity and deadlines must be evaluated 2. Yes: capacity alone establishes suitability 3. Yes: KV never changes during generation Answer and reasoning No: append writes, access granularity and deadlines must be evaluated Active decoding both reads old context and appends state; placement must match those access requirements. ## Evidence and scope The equation is an explicitly scoped conventional attention model. It is not a specification of Astra. Any HBF KV placement is a scenario requiring controller/write behavior and serving measurements, not an asserted deployed configuration. Place read-only weights and mutable KV across specified tiers and inspect transfer paths; no HBF shipping-system benchmark or universal memory semantics are implied. ## Sources - [NVIDIA — Mastering LLM Techniques: Inference Optimization](https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/) - [Sandisk — HBF fact sheet](https://documents.sandisk.com/content/dam/asset-library/en_us/assets/public/sandisk/collateral/company/Sandisk-HBF-Fact-Sheet.pdf) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/prefill-decode-tier-placement-e8be7a292be9e9c5.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Foundations - [Prefill and prompt processing](https://semiaifoundry.com/learn/atlas/prefill-resource-trace/) - [Autoregressive decode and weight traffic](https://semiaifoundry.com/learn/atlas/decode-bandwidth-trace/) - [Place the working set, not just the model](https://semiaifoundry.com/learn/atlas/memory-working-set-placement/) ### Related lessons - [HBF latency, transfer granularity and prefetch](https://semiaifoundry.com/learn/atlas/hbf-latency-prefetch/) - [Prefill/decode disaggregation and useful serving rate](https://semiaifoundry.com/learn/atlas/disaggregated-serving/) - [From memory placement to network, power and cooling](https://semiaifoundry.com/learn/atlas/memory-to-hall-propagation/)