{
  "schemaVersion": 1,
  "id": "atlas-lesson-disaggregated-serving",
  "title": "Prefill/decode disaggregation and useful serving rate",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Separates workload phases when independent scheduling benefits exceed the state-transfer cost.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/disaggregated-serving/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Model–machine co-design",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "DistServe and PagedAttention are distinct research methods. Their measured gains depend on workload and configurations and are not reproduced as unconditional defaults.\n\nThe handoff extension is worked timing and ownership analysis. It is not a measured disaggregated server, network protocol or real-model KV exchange.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/disaggregated-serving-27974f91455cb4c0.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#disaggregated-serving",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "9d8119928ee2717222713ba4d38b7c264a5d0861677a8de85100dfba6cf2b1d4",
  "contentLength": 7040,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-disaggregated-serving.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-disaggregated-serving.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Prefill/decode disaggregation and useful serving rate",
      "start": 40,
      "end": 418
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 418,
      "end": 2342
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 2342,
      "end": 2370
    },
    {
      "id": "section-4",
      "title": "KV handoff floor",
      "start": 2370,
      "end": 2958
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2958,
      "end": 3609
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 3609,
      "end": 4059
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 4059,
      "end": 4345
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 4345,
      "end": 4496
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 4496,
      "end": 4525
    },
    {
      "id": "section-10",
      "title": "What new cost must a disaggregated design explicitly budget?",
      "start": 4525,
      "end": 4917
    },
    {
      "id": "section-11",
      "title": "Can a decoder consume an incomplete handoff simply because some KV bytes arrived?",
      "start": 4917,
      "end": 5328
    },
    {
      "id": "section-12",
      "title": "Evidence and scope",
      "start": 5328,
      "end": 5816
    },
    {
      "id": "section-13",
      "title": "Sources",
      "start": 5816,
      "end": 6288
    },
    {
      "id": "section-14",
      "title": "Continue exploring",
      "start": 6288,
      "end": 6311
    },
    {
      "id": "section-15",
      "title": "Foundations",
      "start": 6311,
      "end": 6538
    },
    {
      "id": "section-16",
      "title": "Related lessons",
      "start": 6538,
      "end": 7040
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#disaggregated-serving"
    },
    {
      "title": "DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving",
      "url": "https://arxiv.org/abs/2401.09670"
    },
    {
      "title": "Efficient Memory Management for Large Language Model Serving with PagedAttention",
      "url": "https://arxiv.org/abs/2309.06180"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/disaggregated-serving-27974f91455cb4c0.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Prefill and prompt processing",
      "url": "https://semiaifoundry.com/learn/atlas/prefill-resource-trace/"
    },
    {
      "title": "Autoregressive decode and weight traffic",
      "url": "https://semiaifoundry.com/learn/atlas/decode-bandwidth-trace/"
    },
    {
      "title": "Inference serving and request scheduling",
      "url": "https://semiaifoundry.com/learn/atlas/serving/"
    },
    {
      "title": "Key–value cache",
      "url": "https://semiaifoundry.com/learn/atlas/kv-cache/"
    },
    {
      "title": "Runtime, driver and device dispatch",
      "url": "https://semiaifoundry.com/learn/atlas/runtime-device-dispatch/"
    },
    {
      "title": "Context, retrieval and durable memory",
      "url": "https://semiaifoundry.com/learn/atlas/context-memory-provenance/"
    },
    {
      "title": "Events, queues and backpressure",
      "url": "https://semiaifoundry.com/learn/atlas/event-queues-delivery/"
    }
  ],
  "recordHash": "28fab066871fa713fc2b3bf2a61f8f5da75aab55aa09ed882ad627f5dd9951e2",
  "contentFormat": "text/markdown",
  "content": "Model–machine co-design · Atlas v1.3.0\n\n# Prefill/decode disaggregation and useful serving rate\n\nSeparates workload phases when independent scheduling benefits exceed the state-transfer cost.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#disaggregated-serving)Accelerator group to serving pool; routes schematic\n\nLearning objectiveJudge phase separation through service objectives and state-transfer costs.\n\n## The mechanism\n\nA serving system can run prefill and decode on different resource pools. Prefill produces KV state that must become accessible to the decode worker, creating a transfer and handoff boundary. Separating the phases may prevent long prompts from disrupting token-generation deadlines and allows different batching or hardware choices. It also adds network demand, queue coordination and failure cases. DistServe studies this design under explicit service objectives. PagedAttention addresses a different layer: efficient mapping and sharing of KV memory blocks. Combining allocation and scheduling methods requires measuring accepted requests that meet latency and quality targets, not just the largest raw token count.\n\nA prefill/decode split hands off sequence state as well as service work. The downstream worker needs compatible model, representation and ownership before it can consume transferred KV data. A partially transferred or stale state cannot be treated as a complete handoff merely because the sending request returned. Keeping the same request identity through queueing and transfer makes those failure boundaries visible.\n\n1.  Classify the prompt work and generation work with separate latency objectives and resource estimates.\n2.  Allocate prefill and decode capacity according to the observed request mix and variability.\n3.  Transfer or expose the completed KV state with explicit ownership, layout and readiness metadata.\n4.  Admit the request to decode only when its state is accessible and scheduling deadlines remain feasible.\n5.  Measure goodput and handoff failures while varying context, batch and network load; compare with a colocated baseline.\n6.  The handoff declares which sequence/model state is transferred, who owns it and what confirms a complete usable copy.\n7.  Queue imbalance, retransmission and state conversion are additional to the simple payload transfer bound.\n\n## Science and assumptions\n\n### KV handoff floor\n\n$$\nT_{\\mathrm{handoff}}\\ge\\frac{M_{KV}}{B_{\\mathrm{link}}}\n$$\n\nKV handoff takes at least transferred KV bytes divided by useful path bandwidth.\n\nSymbols and units\n\nT\\_handoff\n\nKV handoff time lower bound\n\ns\n\nM\\_KV\n\nKV bytes actually transferred\n\nbytes\n\nB\\_link\n\nUseful bandwidth of the handoff path\n\nbytes/s\n\nMKV is bytes actually transferred and Blink useful bytes/s on the selected path; Thandoff is seconds.\n\nAssumptions A payload floor; queueing, setup, layout conversion and synchronization add latency, while reuse or shared access changes transferred bytes.\n\n## Engineering example\n\nSynthetic handoff: 2 GB of KV at 25 GB/s requires at least 80 ms of payload transfer. That may be acceptable for one workload and dominate another’s first-token target; a fixed universal benefit is unjustified.\n\nA synthetic 2 GB KV payload over an effective 25 GB/s path has a payload-only lower bound of 0.08 s, or 80 ms. If a required state-conversion step takes another 10 ms and is serialized after transfer, the handoff path needs at least 90 ms under those assumptions. It cannot be added as free overlap without a dependency argument. An interrupted transfer also needs a rule for discarding or resuming partial state.\n\n## Distinctions and tradeoffs\n\nPaged KV allocation does not itself separate prefill from decode, and phase separation is not automatically beneficial for every workload.\n\nDisaggregation can improve scheduling isolation and hardware fit, but large KV transfers or imbalanced pools can erase the gain.\n\nWorkload phases stress compute and memory differently; useful capacity depends on serving deadlines, arrival patterns and the cost of moving state.\n\n## Specifications and interfaces\n\nGoodput\n\nAccepted requests/s meeting stated service objectives\n\nRaw throughput includes outputs that may miss deadlines.\n\nHandoff state\n\nKV bytes and metadata per transferred request\n\nTransfer must preserve model version, layout and request identity.\n\n### Interfaces\n\nPrefill pool Input\n\nCompleted prompt state and request deadline\n\nDecode pool Output\n\nContinued token stream after verified KV handoff\n\n## Check your understanding\n\n### What new cost must a disaggregated design explicitly budget?\n\n1.  Moving or exposing KV state between phase workers\n2.  Retraining the tokenizer for each token\n3.  Eliminating all decode weights\n\nAnswer and reasoning\n\nMoving or exposing KV state between phase workers\n\nThe decode worker needs the prompt’s context state, making its handoff a capacity, bandwidth and correctness concern.\n\n### Can a decoder consume an incomplete handoff simply because some KV bytes arrived?\n\n1.  Yes, all partial state is equivalent\n2.  No, its declared valid-state and ownership contract must be met\n3.  Yes, if the link is fast\n\nAnswer and reasoning\n\nNo, its declared valid-state and ownership contract must be met\n\nArrival of some bytes does not establish a complete sequence representation suitable for decode.\n\n## Evidence and scope\n\nDistServe and PagedAttention are distinct research methods. Their measured gains depend on workload and configurations and are not reproduced as unconditional defaults.\n\nThe handoff extension is worked timing and ownership analysis. It is not a measured disaggregated server, network protocol or real-model KV exchange.\n\nNo current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.\n\n## Sources\n\n-   [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://arxiv.org/abs/2401.09670)\n-   [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/disaggregated-serving-27974f91455cb4c0.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Prefill and prompt processing](https://semiaifoundry.com/learn/atlas/prefill-resource-trace/)\n-   [Autoregressive decode and weight traffic](https://semiaifoundry.com/learn/atlas/decode-bandwidth-trace/)\n\n### Related lessons\n\n-   [Inference serving and request scheduling](https://semiaifoundry.com/learn/atlas/serving/)\n-   [Key–value cache](https://semiaifoundry.com/learn/atlas/kv-cache/)\n-   [Runtime, driver and device dispatch](https://semiaifoundry.com/learn/atlas/runtime-device-dispatch/)\n-   [Context, retrieval and durable memory](https://semiaifoundry.com/learn/atlas/context-memory-provenance/)\n-   [Events, queues and backpressure](https://semiaifoundry.com/learn/atlas/event-queues-delivery/)"
}