{
  "schemaVersion": 1,
  "id": "atlas-lesson-serving",
  "title": "Inference serving and request scheduling",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Turns incoming model requests into scheduled prefill and generation work.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/serving/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Model deployment",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "The expression does not promise a quality or throughput target.\n\nThis additive trace is worked synthetic timing. It does not extend the existing resource calculator into a measured queueing engine or claim current vendor service rates.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/serving-ac7cef5ec8fe6856.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#serving",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "78afb00ebb54a79008ec665146bab10e02e464a681363e8e8ff4342a0cc3f4e7",
  "contentLength": 7205,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-serving.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-serving.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Inference serving and request scheduling",
      "start": 33,
      "end": 324
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 324,
      "end": 2076
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 2076,
      "end": 2104
    },
    {
      "id": "section-4",
      "title": "Generation latency decomposition",
      "start": 2104,
      "end": 2954
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2954,
      "end": 3471
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 3471,
      "end": 3854
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3854,
      "end": 4194
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 4194,
      "end": 4338
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 4338,
      "end": 4367
    },
    {
      "id": "section-10",
      "title": "What does continuous batching allow?",
      "start": 4367,
      "end": 4788
    },
    {
      "id": "section-11",
      "title": "A request arrives at 0 ms and returns its first token at 80 ms after 30 ms queueing. What is TTFT?",
      "start": 4788,
      "end": 5031
    },
    {
      "id": "section-12",
      "title": "Historical context",
      "start": 5031,
      "end": 5054
    },
    {
      "id": "section-13",
      "title": "2023 · PagedAttention connects memory management to LLM serving",
      "start": 5054,
      "end": 5378
    },
    {
      "id": "section-14",
      "title": "Evidence and scope",
      "start": 5378,
      "end": 5823
    },
    {
      "id": "section-15",
      "title": "Sources",
      "start": 5823,
      "end": 6206
    },
    {
      "id": "section-16",
      "title": "Continue exploring",
      "start": 6206,
      "end": 6229
    },
    {
      "id": "section-17",
      "title": "Foundations",
      "start": 6229,
      "end": 6501
    },
    {
      "id": "section-18",
      "title": "Related lessons",
      "start": 6501,
      "end": 7205
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#serving"
    },
    {
      "title": "Kwon et al. — PagedAttention",
      "url": "https://arxiv.org/abs/2309.06180"
    },
    {
      "title": "vLLM — Inference and Serving Documentation",
      "url": "https://docs.vllm.ai/en/latest/"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/serving-ac7cef5ec8fe6856.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Cluster scheduler and resource allocation",
      "url": "https://semiaifoundry.com/learn/atlas/scheduler/"
    },
    {
      "title": "GPU kernel and thread hierarchy",
      "url": "https://semiaifoundry.com/learn/atlas/kernel/"
    },
    {
      "title": "Key–value cache",
      "url": "https://semiaifoundry.com/learn/atlas/kv-cache/"
    },
    {
      "title": "CUDA programming and runtime layer",
      "url": "https://semiaifoundry.com/learn/atlas/cuda/"
    },
    {
      "title": "Quantization and numerical representation",
      "url": "https://semiaifoundry.com/learn/atlas/quantization/"
    },
    {
      "title": "Dataset and checkpoint storage hierarchy",
      "url": "https://semiaifoundry.com/learn/atlas/storage/"
    },
    {
      "title": "Output head & decoding",
      "url": "https://semiaifoundry.com/learn/atlas/output-head/"
    },
    {
      "title": "Events, queues and backpressure",
      "url": "https://semiaifoundry.com/learn/atlas/event-queues-delivery/"
    },
    {
      "title": "Traces connect cost, latency and results",
      "url": "https://semiaifoundry.com/learn/atlas/telemetry-traces-evaluation/"
    },
    {
      "title": "Runtime, driver and device dispatch",
      "url": "https://semiaifoundry.com/learn/atlas/runtime-device-dispatch/"
    }
  ],
  "recordHash": "4a96b3cabad8814073e2cb1e33b06216e80243dbbcd3762443fa4a2a8096a117",
  "contentFormat": "text/markdown",
  "content": "Model deployment · Atlas v1.3.0\n\n# Inference serving and request scheduling\n\nTurns incoming model requests into scheduled prefill and generation work.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#serving)Request-to-device software pipeline\n\nLearning objectiveSeparate time-to-first-token from per-token latency.\n\n## The mechanism\n\nA serving engine manages the many requests that share an inference deployment. It receives inputs, schedules prompt processing, keeps track of generation state, and returns outputs while managing memory and latency goals. Continuous batching can admit and retire requests as generation progresses instead of waiting for one fixed batch to finish. Paged KV-cache management organizes attention state in blocks to reduce wasted allocation and support flexible scheduling. The atlas shows requests with different lengths sharing device execution while retaining distinct state. Serving optimization changes resource use and response timing, not the meaning of trained weights. Throughput, time to first token, and time between later tokens must be considered separately because users experience all three differently.\n\nA request-level explanation should retain arrival, admission, service and completion as separate states. Resource bounds for a decode step do not by themselves determine which queued request runs next. A scheduling policy also owns live sequence state and must release or preserve it correctly after completion, cancellation and handoff. A useful comparison holds the request trace and service assumptions fixed while changing only the scheduling policy.\n\n1.  Accept requests and schedule their prompt processing.\n2.  Maintain each request’s state while batching compatible generation work.\n3.  Stream results and release resources as requests finish.\n4.  Request timestamps and state transitions distinguish queue wait, first-token delay and later inter-token intervals.\n5.  A canceled or completed request releases its owned state under an explicit policy; a workload bound alone does not simulate that transition.\n\n## Science and assumptions\n\n### Generation latency decomposition\n\n$$\nT_{\\mathrm{response}}\\approx\\mathrm{TTFT}+(N_{\\mathrm{output}}-1)\\overline{\\mathrm{ITL}}\n$$\n\nResponse time is approximately time to first token plus the remaining token gaps times their mean inter-token latency.\n\nSymbols and units\n\nT\\_response\n\nTime through the final output token of one streamed request\n\ns\n\nTTFT\n\nTime to first token\n\ns\n\nN\\_output\n\nNumber of output tokens, at least one\n\ntokens\n\nmean ITL\n\nMean gap between successive output tokens\n\ns/token gap\n\nTime to first token covers the initial wait and work; later tokens accumulate inter-token latency. Two systems with equal total throughput can offer very different interactive responsiveness.\n\nAssumptions Simplified mean-latency decomposition for one streamed request with at least one output token; actual token gaps and network delivery vary.\n\n## Engineering example\n\nFor 200 output tokens, response time is TTFT plus 199 inter-token intervals; batching changes both terms.\n\nFor a hypothetical request that arrives at 0 ms, is admitted at 30 ms and returns its first token at 80 ms, queue wait is 30 ms and time to first token is 80 ms. If 100 later inter-token intervals each take 5 ms, its final token arrives at 580 ms. Summing a separate device enqueue duration into every interval without identifying a dependency would double-count or misplace overhead.\n\n## Distinctions and tradeoffs\n\nHigher tokens-per-second throughput does not automatically mean lower latency for one user’s request.\n\nBatching can improve throughput. Queueing and memory pressure can worsen individual request latency.\n\nServing makes trained models available to people and applications. It translates hardware capacity into practical responsiveness and concurrency.\n\n## Specifications and interfaces\n\nExecution phases\n\nPrefill and decode\n\nPrompt processing followed by iterative generation.\n\nLatency measures\n\nTime to first token and inter-token latency\n\nDifferent parts of perceived responsiveness.\n\nExample techniques\n\nContinuous batching and paged KV cache\n\nDocumented in vLLM; implementation-specific.\n\n### Interfaces\n\nInference requests Input\n\nPrompts, settings, and request state\n\nGenerated outputs Output\n\nStreamed tokens or completed results\n\n## Check your understanding\n\n### What does continuous batching allow?\n\n1.  All users must finish at the same time\n2.  Requests can enter and leave the running workload as generation progresses\n3.  Every token bypasses the model\n\nAnswer and reasoning\n\nRequests can enter and leave the running workload as generation progresses\n\nThe batch can evolve across iterations, helping use capacity despite requests having different arrival times and lengths.\n\n### A request arrives at 0 ms and returns its first token at 80 ms after 30 ms queueing. What is TTFT?\n\n1.  30 ms\n2.  50 ms\n3.  80 ms\n\nAnswer and reasoning\n\n80 ms\n\nTTFT is measured from arrival to first token and includes the queue interval.\n\n## Historical context\n\n### 2023 · PagedAttention connects memory management to LLM serving\n\nThe vLLM paper showed how block-based KV-cache management and flexible sharing can improve the use of memory during batched generation. Reported gains depend on workloads and comparisons.\n\n[Kwon et al. — PagedAttention](https://arxiv.org/abs/2309.06180)\n\n## Evidence and scope\n\nThe expression does not promise a quality or throughput target.\n\nThis additive trace is worked synthetic timing. It does not extend the existing resource calculator into a measured queueing engine or claim current vendor service rates.\n\nFollow a modeled batched decode critical path and energy budget; prefill, expert routing and scheduling differences are comparison context rather than separately simulated algorithms.\n\n## Sources\n\n-   [vLLM — Inference and Serving Documentation](https://docs.vllm.ai/en/latest/)\n-   [Kwon et al. — Efficient Memory Management with PagedAttention](https://arxiv.org/abs/2309.06180)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/serving-ac7cef5ec8fe6856.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Cluster scheduler and resource allocation](https://semiaifoundry.com/learn/atlas/scheduler/)\n-   [GPU kernel and thread hierarchy](https://semiaifoundry.com/learn/atlas/kernel/)\n-   [Key–value cache](https://semiaifoundry.com/learn/atlas/kv-cache/)\n\n### Related lessons\n\n-   [CUDA programming and runtime layer](https://semiaifoundry.com/learn/atlas/cuda/)\n-   [Quantization and numerical representation](https://semiaifoundry.com/learn/atlas/quantization/)\n-   [Dataset and checkpoint storage hierarchy](https://semiaifoundry.com/learn/atlas/storage/)\n-   [Output head & decoding](https://semiaifoundry.com/learn/atlas/output-head/)\n-   [Events, queues and backpressure](https://semiaifoundry.com/learn/atlas/event-queues-delivery/)\n-   [Traces connect cost, latency and results](https://semiaifoundry.com/learn/atlas/telemetry-traces-evaluation/)\n-   [Runtime, driver and device dispatch](https://semiaifoundry.com/learn/atlas/runtime-device-dispatch/)"
}