Inference serving and request scheduling

The mechanism

A serving engine manages the many requests that share an inference deployment. It receives inputs, schedules prompt processing, keeps track of generation state, and returns outputs while managing memory and latency goals. Continuous batching can admit and retire requests as generation progresses instead of waiting for one fixed batch to finish. Paged KV-cache management organizes attention state in blocks to reduce wasted allocation and support flexible scheduling. The atlas shows requests with different lengths sharing device execution while retaining distinct state. Serving optimization changes resource use and response timing, not the meaning of trained weights. Throughput, time to first token, and time between later tokens must be considered separately because users experience all three differently.

A request-level explanation should retain arrival, admission, service and completion as separate states. Resource bounds for a decode step do not by themselves determine which queued request runs next. A scheduling policy also owns live sequence state and must release or preserve it correctly after completion, cancellation and handoff. A useful comparison holds the request trace and service assumptions fixed while changing only the scheduling policy.

1. Accept requests and schedule their prompt processing.
2. Maintain each request’s state while batching compatible generation work.
3. Stream results and release resources as requests finish.
4. Request timestamps and state transitions distinguish queue wait, first-token delay and later inter-token intervals.
5. A canceled or completed request releases its owned state under an explicit policy; a workload bound alone does not simulate that transition.

Science and assumptions

Engineering example

For 200 output tokens, response time is TTFT plus 199 inter-token intervals; batching changes both terms.

For a hypothetical request that arrives at 0 ms, is admitted at 30 ms and returns its first token at 80 ms, queue wait is 30 ms and time to first token is 80 ms. If 100 later inter-token intervals each take 5 ms, its final token arrives at 580 ms. Summing a separate device enqueue duration into every interval without identifying a dependency would double-count or misplace overhead.

Distinctions and tradeoffs

Higher tokens-per-second throughput does not automatically mean lower latency for one user’s request.

Batching can improve throughput. Queueing and memory pressure can worsen individual request latency.

Serving makes trained models available to people and applications. It translates hardware capacity into practical responsiveness and concurrency.

Specifications and interfaces

Execution phases

Prefill and decode

Prompt processing followed by iterative generation.

Latency measures

Time to first token and inter-token latency

Different parts of perceived responsiveness.

Example techniques

Continuous batching and paged KV cache

Documented in vLLM; implementation-specific.

Interfaces

Inference requests Input

Prompts, settings, and request state

Generated outputs Output

Streamed tokens or completed results

Check your understanding

Historical context

2023 · PagedAttention connects memory management to LLM serving

The vLLM paper showed how block-based KV-cache management and flexible sharing can improve the use of memory during batched generation. Reported gains depend on workloads and comparisons.

Kwon et al. — PagedAttention

Evidence and scope

The expression does not promise a quality or throughput target.

This additive trace is worked synthetic timing. It does not extend the existing resource calculator into a measured queueing engine or claim current vendor service rates.

Follow a modeled batched decode critical path and energy budget; prefill, expert routing and scheduling differences are comparison context rather than separately simulated algorithms.

Sources

• vLLM — Inference and Serving Documentation
• Kwon et al. — Efficient Memory Management with PagedAttention

Published lesson data · Atlas releases and licensing

Continue exploring

Foundations

• Cluster scheduler and resource allocation
• GPU kernel and thread hierarchy
• Key–value cache

Related lessons

• CUDA programming and runtime layer
• Quantization and numerical representation
• Dataset and checkpoint storage hierarchy
• Output head & decoding
• Events, queues and backpressure
• Traces connect cost, latency and results
• Runtime, driver and device dispatch