{"id":"prefill-decode-tier-placement","level":"software","name":"Prefill, decode and the moving KV working set","shortName":"Prefill, decode and the moving KV working set","role":"Place active and reusable inference state according to the execution phase and service deadline.","description":"Prefill processes an input sequence and creates state for subsequent generation. Decode repeatedly generates new tokens while reading prior context and appending new KV state. Weights are reused across requests, while KV belongs to a model/version, sequence position and attention configuration. A reusable prefix can be retained or offloaded, but restoring it competes with recomputing it. Moving large KV payloads between prefill and decode nodes adds network work and coordination. Larger near-compute capacity can help only if the state is available before the next dependent operation and the placement preserves correctness.","mechanism":["Identify whether the workload is prefill, decode or a mixture; record batch, context and attention architecture.","Budget the live KV payload and append/write behavior separately from weight reads.","Keep imminent mutable state accessible; choose a policy for inactive prefixes, eviction and restoration.","Compare restore transfer time with recomputation and service deadlines under actual contention.","For split serving, trace KV over its physical links and validate model/version, ownership and completion before decode uses it."],"science":[{"title":"Conventional KV payload","equation":"MKV = 2 L Hkv d b Σs Ts","explanation":"Each stored token has key and value vectors for the specified layers and KV heads.","assumptions":"Full conventional KV representation; no sliding windows, compression or architecture-specific state reduction. Excludes paging overhead and padding."},{"title":"Restore or recompute","equation":"trestore ≥ Lpath + V/Bpath","explanation":"Compare this transfer lower bound with measured recomputation time and the request deadline.","assumptions":"One aggregate transfer; includes no queueing or validation cost and assumes the cache is valid."}],"specs":[{"label":"Decode","value":"Repeated reads and KV appends","context":"An active KV cache is not immutable pretrained weight storage."},{"label":"Prefix reuse","value":"Must retain semantic validity","context":"Model, token sequence and attention configuration matter."}],"interfaces":[{"name":"Prefill","direction":"in","signal":"Prompt computation and newly produced KV"},{"name":"Decode","direction":"out","signal":"Ready prior state, appended KV and output token"}],"tradeoff":"More cached context saves recomputation but occupies memory and creates movement when placed remotely; a cache hit still has a service cost.","whyAI":"Memory tiers determine concurrency and context support, while the restore path can affect first-token and inter-token latency.","prerequisites":["prefill-resource-trace","decode-bandwidth-trace","memory-working-set-placement"],"related":["hbf-latency-prefetch","disaggregated-serving","memory-to-hall-propagation"],"misconception":"Active decoding both reads old context and appends state; placement must match those access requirements.","check":{"question":"Does a large read-oriented tier automatically suit the active KV cache?","options":["No: append writes, access granularity and deadlines must be evaluated","Yes: capacity alone establishes suitability","Yes: KV never changes during generation"],"answer":0,"explanation":"Active decoding both reads old context and appends state; placement must match those access requirements."},"engineeringExample":"Synthetic conventional KV: 32 layers × 8 KV heads × 128 values × 2 bytes × 2 for K/V × 8,192 tokens is 1 GiB per sequence. Eight such sequences need 8 GiB before metadata and other model state.","evidenceNotes":"The equation is an explicitly scoped conventional attention model. It is not a specification of Astra. Any HBF KV placement is a scenario requiring controller/write behavior and serving measurements, not an asserted deployed configuration.","sources":[{"title":"NVIDIA — Mastering LLM Techniques: Inference Optimization","url":"https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/"},{"title":"Sandisk — HBF fact sheet","url":"https://documents.sandisk.com/content/dam/asset-library/en_us/assets/public/sandisk/collateral/company/Sandisk-HBF-Fact-Sheet.pdf"}],"learningObjective":"Place active and reusable inference state according to the execution phase and service deadline.","workstream":"W11","visualFamily":"model-workload","scale":"Prefill output, live decode state and tier-crossing transfers; abstraction","kind":"system","labId":"inference-placement","branch":"Software","subbranch":"Working-set placement across memory tiers","scope":"Place read-only weights and mutable KV across specified tiers and inspect transfer paths; no HBF shipping-system benchmark or universal memory semantics are implied.","investigationId":"execution","labScope":"Place read-only weights and mutable KV across specified tiers and inspect transfer paths; no HBF shipping-system benchmark or universal memory semantics are implied."}