{"id":"serving","level":"software","name":"Inference serving and request scheduling","shortName":"Serving engine","role":"Turns incoming model requests into scheduled prefill and generation work.","description":"A serving engine manages the many requests that share an inference deployment. It receives inputs, schedules prompt processing, keeps track of generation state, and returns outputs while managing memory and latency goals. Continuous batching can admit and retire requests as generation progresses instead of waiting for one fixed batch to finish. Paged KV-cache management organizes attention state in blocks to reduce wasted allocation and support flexible scheduling. The atlas shows requests with different lengths sharing device execution while retaining distinct state. Serving optimization changes resource use and response timing, not the meaning of trained weights. Throughput, time to first token, and time between later tokens must be considered separately because users experience all three differently.\n\nA request-level explanation should retain arrival, admission, service and completion as separate states. Resource bounds for a decode step do not by themselves determine which queued request runs next. A scheduling policy also owns live sequence state and must release or preserve it correctly after completion, cancellation and handoff. A useful comparison holds the request trace and service assumptions fixed while changing only the scheduling policy.","specs":[{"label":"Execution phases","value":"Prefill and decode","context":"Prompt processing followed by iterative generation."},{"label":"Latency measures","value":"Time to first token and inter-token latency","context":"Different parts of perceived responsiveness."},{"label":"Example techniques","value":"Continuous batching and paged KV cache","context":"Documented in vLLM; implementation-specific."}],"interfaces":[{"name":"Inference requests","direction":"in","signal":"Prompts, settings, and request state"},{"name":"Generated outputs","direction":"out","signal":"Streamed tokens or completed results"}],"tradeoff":"Batching can improve throughput. Queueing and memory pressure can worsen individual request latency.","whyAI":"Serving makes trained models available to people and applications. It translates hardware capacity into practical responsiveness and concurrency.","related":["cuda","kernel","quantization","storage","scheduler","output-head","event-queues-delivery","telemetry-traces-evaluation","runtime-device-dispatch"],"sources":[{"title":"vLLM — Inference and Serving Documentation","url":"https://docs.vllm.ai/en/latest/"},{"title":"Kwon et al. — Efficient Memory Management with PagedAttention","url":"https://arxiv.org/abs/2309.06180"}],"scale":"Request-to-device software pipeline","kind":"serving","mechanism":["Accept requests and schedule their prompt processing.","Maintain each request’s state while batching compatible generation work.","Stream results and release resources as requests finish.","Request timestamps and state transitions distinguish queue wait, first-token delay and later inter-token intervals.","A canceled or completed request releases its owned state under an explicit policy; a workload bound alone does not simulate that transition."],"misconception":"Higher tokens-per-second throughput does not automatically mean lower latency for one user’s request.","check":{"question":"What does continuous batching allow?","options":["All users must finish at the same time","Requests can enter and leave the running workload as generation progresses","Every token bypasses the model"],"answer":1,"explanation":"The batch can evolve across iterations, helping use capacity despite requests having different arrival times and lengths."},"branch":"Model deployment","subbranch":"Request execution","science":[{"title":"Generation latency decomposition","equation":"T_response ≈ TTFT + (N_output − 1) × mean_ITL","explanation":"Time to first token covers the initial wait and work; later tokens accumulate inter-token latency. Two systems with equal total throughput can offer very different interactive responsiveness.","assumptions":"Simplified mean-latency decomposition for one streamed request with at least one output token; actual token gaps and network delivery vary."}],"history":[{"year":2023,"title":"PagedAttention connects memory management to LLM serving","significance":"The vLLM paper showed how block-based KV-cache management and flexible sharing can improve the use of memory during batched generation. Reported gains depend on workloads and comparisons.","source":{"title":"Kwon et al. — PagedAttention","url":"https://arxiv.org/abs/2309.06180"}}],"learningObjective":"Separate time-to-first-token from per-token latency.","engineeringExample":"For 200 output tokens, response time is TTFT plus 199 inter-token intervals; batching changes both terms.\n\nFor a hypothetical request that arrives at 0 ms, is admitted at 30 ms and returns its first token at 80 ms, queue wait is 30 ms and time to first token is 80 ms. If 100 later inter-token intervals each take 5 ms, its final token arrives at 580 ms. Summing a separate device enqueue duration into every interval without identifying a dependency would double-count or misplace overhead.","evidenceNotes":"The expression does not promise a quality or throughput target.\n\nThis additive trace is worked synthetic timing. It does not extend the existing resource calculator into a measured queueing engine or claim current vendor service rates.","visualFamily":"model-workload","prerequisites":["scheduler","kernel","kv-cache"],"checks":[{"question":"A request arrives at 0 ms and returns its first token at 80 ms after 30 ms queueing. What is TTFT?","options":["30 ms","50 ms","80 ms"],"answer":2,"explanation":"TTFT is measured from arrival to first token and includes the queue interval."}],"labId":"token-system-path","scope":"Follow a modeled batched decode critical path and energy budget; prefill, expert routing and scheduling differences are comparison context rather than separately simulated algorithms.","investigationId":"execution","labScope":"Follow a modeled batched decode critical path and energy budget; prefill, expert routing and scheduling differences are comparison context rather than separately simulated algorithms."}