The mechanism
Autoregressive decode generates a token using the preceding context, then uses the selected token to continue. For one request, each projection may have little weight reuse within a step, making memory traffic expensive relative to arithmetic. Serving several requests together can reuse weight tiles across their new-token vectors, but raises KV capacity and scheduling demands. Longer contexts also increase attention reads from the KV cache. Capacity, bandwidth and token dependency are separate constraints: a model may fit yet move weights too slowly, or have ample bandwidth but wait on a serial chain of computation and communication.
- Take the current request tokens and gather the context state needed by each layer.
- Apply projections, reusing weights across concurrent requests where the batch permits it.
- Read prior keys and values for attention and append the newly produced cache entries.
- Complete logits and token selection before the next dependent step can proceed for that request.
- Measure generated-token latency, weight traffic, KV traffic and queueing while varying batch and context independently.
Science and assumptions
Weight-streaming decode floor
Decode-step time is at least actually read weight bytes divided by sustainable memory bandwidth.
Symbols and units
- T_step
- Decode-step time lower bound from weight traffic
s
- W_read
- Weight bytes actually read during the step
bytes
- B_memory
- Sustainable useful memory bandwidth
bytes/s
Wread is weight bytes actually read per step and Bmemory is sustainable bytes/s; Tstep is seconds.
Assumptions A bandwidth lower bound; cached weights reduce Wread and KV traffic, arithmetic and synchronization can add time.
Engineering example
Synthetic weights: reading 20 GB each step through 1 TB/s requires at least 20 ms before other costs, or at most 50 steps/s. Batching four requests can share that weight pass, but does not make each request’s token dependency disappear.
Distinctions and tradeoffs
One active request does not generally generate all its future tokens independently; more peak matrix FLOPS alone cannot remove autoregressive dependence.
Batching improves weight reuse but consumes more cache capacity and can increase latency; reducing precision saves bytes only if kernels and quality support the representation.
Decode makes the difference between nominal arithmetic capacity and delivered responses visible, particularly at low concurrency and strict per-token deadlines.
Specifications and interfaces
- Throughput units
- Generated tokens/s versus requests/s
State batch and accepted output criteria; these are not prompt tokens/s.
- KV scaling
- Requests × context × stored head dimensions
Grouped-query attention and cache precision change the byte count.
Interfaces
- Weights, new token and KV Input
- Repeated parameter reads and context-dependent attention state
- Next token and appended KV Output
- New output plus persistent request state
Check your understanding
Which change can improve weight reuse but increase KV memory demand?
- Serving more requests together
- Doubling only the clock label
- Deleting the runtime dependency rules
Answer and reasoning
Serving more requests together
A batch shares weight use across requests, but each request needs its own context state unless specific prefixes can be shared.
Evidence and scope
Example rates are invented and assume one weight pass. Real caching, quantization, expert activation and tensor partitioning change the bytes and must be profiled.
Follow a modeled batched decode critical path and energy budget; prefill, expert routing and scheduling differences are comparison context rather than separately simulated algorithms.