{"id":"disaggregated-serving","level":"software","name":"Prefill/decode disaggregation and useful serving rate","shortName":"Prefill/decode disaggregation and useful serving rate","role":"Separates workload phases when independent scheduling benefits exceed the state-transfer cost.","description":"A serving system can run prefill and decode on different resource pools. Prefill produces KV state that must become accessible to the decode worker, creating a transfer and handoff boundary. Separating the phases may prevent long prompts from disrupting token-generation deadlines and allows different batching or hardware choices. It also adds network demand, queue coordination and failure cases. DistServe studies this design under explicit service objectives. PagedAttention addresses a different layer: efficient mapping and sharing of KV memory blocks. Combining allocation and scheduling methods requires measuring accepted requests that meet latency and quality targets, not just the largest raw token count.\n\nA prefill/decode split hands off sequence state as well as service work. The downstream worker needs compatible model, representation and ownership before it can consume transferred KV data. A partially transferred or stale state cannot be treated as a complete handoff merely because the sending request returned. Keeping the same request identity through queueing and transfer makes those failure boundaries visible.","specs":[{"label":"Goodput","value":"Accepted requests/s meeting stated service objectives","context":"Raw throughput includes outputs that may miss deadlines."},{"label":"Handoff state","value":"KV bytes and metadata per transferred request","context":"Transfer must preserve model version, layout and request identity."}],"interfaces":[{"name":"Prefill pool","direction":"in","signal":"Completed prompt state and request deadline"},{"name":"Decode pool","direction":"out","signal":"Continued token stream after verified KV handoff"}],"tradeoff":"Disaggregation can improve scheduling isolation and hardware fit, but large KV transfers or imbalanced pools can erase the gain.","whyAI":"Workload phases stress compute and memory differently; useful capacity depends on serving deadlines, arrival patterns and the cost of moving state.","related":["serving","kv-cache","prefill-resource-trace","decode-bandwidth-trace","runtime-device-dispatch","context-memory-provenance","event-queues-delivery"],"sources":[{"title":"DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving","url":"https://arxiv.org/abs/2401.09670"},{"title":"Efficient Memory Management for Large Language Model Serving with PagedAttention","url":"https://arxiv.org/abs/2309.06180"}],"scale":"Accelerator group to serving pool; routes schematic","kind":"workload","branch":"Model–machine co-design","subbranch":"Distributed work and serving choices","mechanism":["Classify the prompt work and generation work with separate latency objectives and resource estimates.","Allocate prefill and decode capacity according to the observed request mix and variability.","Transfer or expose the completed KV state with explicit ownership, layout and readiness metadata.","Admit the request to decode only when its state is accessible and scheduling deadlines remain feasible.","Measure goodput and handoff failures while varying context, batch and network load; compare with a colocated baseline.","The handoff declares which sequence/model state is transferred, who owns it and what confirms a complete usable copy.","Queue imbalance, retransmission and state conversion are additional to the simple payload transfer bound."],"misconception":"Paged KV allocation does not itself separate prefill from decode, and phase separation is not automatically beneficial for every workload.","check":{"question":"What new cost must a disaggregated design explicitly budget?","options":["Moving or exposing KV state between phase workers","Retraining the tokenizer for each token","Eliminating all decode weights"],"answer":0,"explanation":"The decode worker needs the prompt’s context state, making its handoff a capacity, bandwidth and correctness concern."},"science":[{"title":"KV handoff floor","equation":"Thandoff ≥ MKV / Blink","explanation":"MKV is bytes actually transferred and Blink useful bytes/s on the selected path; Thandoff is seconds.","assumptions":"A payload floor; queueing, setup, layout conversion and synchronization add latency, while reuse or shared access changes transferred bytes."}],"prerequisites":["prefill-resource-trace","decode-bandwidth-trace"],"learningObjective":"Judge phase separation through service objectives and state-transfer costs.","workstream":"W11","visualFamily":"model-workload","engineeringExample":"Synthetic handoff: 2 GB of KV at 25 GB/s requires at least 80 ms of payload transfer. That may be acceptable for one workload and dominate another’s first-token target; a fixed universal benefit is unjustified.\n\nA synthetic 2 GB KV payload over an effective 25 GB/s path has a payload-only lower bound of 0.08 s, or 80 ms. If a required state-conversion step takes another 10 ms and is serialized after transfer, the handoff path needs at least 90 ms under those assumptions. It cannot be added as free overlap without a dependency argument. An interrupted transfer also needs a rule for discarding or resuming partial state.","evidenceNotes":"DistServe and PagedAttention are distinct research methods. Their measured gains depend on workload and configurations and are not reproduced as unconditional defaults.\n\nThe handoff extension is worked timing and ownership analysis. It is not a measured disaggregated server, network protocol or real-model KV exchange.","checks":[{"question":"Can a decoder consume an incomplete handoff simply because some KV bytes arrived?","options":["Yes, all partial state is equivalent","No, its declared valid-state and ownership contract must be met","Yes, if the link is fast"],"answer":1,"explanation":"Arrival of some bytes does not establish a complete sequence representation suitable for decode."}],"labId":"library","scope":"No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.","investigationId":"execution","labScope":"No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions."}