The mechanism
A serving system can run prefill and decode on different resource pools. Prefill produces KV state that must become accessible to the decode worker, creating a transfer and handoff boundary. Separating the phases may prevent long prompts from disrupting token-generation deadlines and allows different batching or hardware choices. It also adds network demand, queue coordination and failure cases. DistServe studies this design under explicit service objectives. PagedAttention addresses a different layer: efficient mapping and sharing of KV memory blocks. Combining allocation and scheduling methods requires measuring accepted requests that meet latency and quality targets, not just the largest raw token count.
A prefill/decode split hands off sequence state as well as service work. The downstream worker needs compatible model, representation and ownership before it can consume transferred KV data. A partially transferred or stale state cannot be treated as a complete handoff merely because the sending request returned. Keeping the same request identity through queueing and transfer makes those failure boundaries visible.
- Classify the prompt work and generation work with separate latency objectives and resource estimates.
- Allocate prefill and decode capacity according to the observed request mix and variability.
- Transfer or expose the completed KV state with explicit ownership, layout and readiness metadata.
- Admit the request to decode only when its state is accessible and scheduling deadlines remain feasible.
- Measure goodput and handoff failures while varying context, batch and network load; compare with a colocated baseline.
- The handoff declares which sequence/model state is transferred, who owns it and what confirms a complete usable copy.
- Queue imbalance, retransmission and state conversion are additional to the simple payload transfer bound.
Science and assumptions
KV handoff floor
KV handoff takes at least transferred KV bytes divided by useful path bandwidth.
Symbols and units
- T_handoff
- KV handoff time lower bound
s
- M_KV
- KV bytes actually transferred
bytes
- B_link
- Useful bandwidth of the handoff path
bytes/s
MKV is bytes actually transferred and Blink useful bytes/s on the selected path; Thandoff is seconds.
Assumptions A payload floor; queueing, setup, layout conversion and synchronization add latency, while reuse or shared access changes transferred bytes.
Engineering example
Synthetic handoff: 2 GB of KV at 25 GB/s requires at least 80 ms of payload transfer. That may be acceptable for one workload and dominate another’s first-token target; a fixed universal benefit is unjustified.
A synthetic 2 GB KV payload over an effective 25 GB/s path has a payload-only lower bound of 0.08 s, or 80 ms. If a required state-conversion step takes another 10 ms and is serialized after transfer, the handoff path needs at least 90 ms under those assumptions. It cannot be added as free overlap without a dependency argument. An interrupted transfer also needs a rule for discarding or resuming partial state.
Distinctions and tradeoffs
Paged KV allocation does not itself separate prefill from decode, and phase separation is not automatically beneficial for every workload.
Disaggregation can improve scheduling isolation and hardware fit, but large KV transfers or imbalanced pools can erase the gain.
Workload phases stress compute and memory differently; useful capacity depends on serving deadlines, arrival patterns and the cost of moving state.
Specifications and interfaces
- Goodput
- Accepted requests/s meeting stated service objectives
Raw throughput includes outputs that may miss deadlines.
- Handoff state
- KV bytes and metadata per transferred request
Transfer must preserve model version, layout and request identity.
Interfaces
- Prefill pool Input
- Completed prompt state and request deadline
- Decode pool Output
- Continued token stream after verified KV handoff
Check your understanding
What new cost must a disaggregated design explicitly budget?
- Moving or exposing KV state between phase workers
- Retraining the tokenizer for each token
- Eliminating all decode weights
Answer and reasoning
Moving or exposing KV state between phase workers
The decode worker needs the prompt’s context state, making its handoff a capacity, bandwidth and correctness concern.
Can a decoder consume an incomplete handoff simply because some KV bytes arrived?
- Yes, all partial state is equivalent
- No, its declared valid-state and ownership contract must be met
- Yes, if the link is fast
Answer and reasoning
No, its declared valid-state and ownership contract must be met
Arrival of some bytes does not establish a complete sequence representation suitable for decode.
Evidence and scope
DistServe and PagedAttention are distinct research methods. Their measured gains depend on workload and configurations and are not reproduced as unconditional defaults.
The handoff extension is worked timing and ownership analysis. It is not a measured disaggregated server, network protocol or real-model KV exchange.
No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.