# Autoregressive decode and weight traffic

Canonical: https://semiaifoundry.com/learn/atlas/decode-bandwidth-trace/
Revision: 2026-10-07
Rights: https://semiaifoundry.com/ai-atlas/licenses.html

Model–machine co-design · Atlas v1.3.0

# Autoregressive decode and weight traffic

Produces dependent new tokens while repeatedly accessing weights and growing KV state.

[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#decode-bandwidth-trace)One model step to accelerator memory hierarchy

Learning objectiveIdentify why low-batch decode may be bandwidth-limited even when weights fit.

## The mechanism

Autoregressive decode generates a token using the preceding context, then uses the selected token to continue. For one request, each projection may have little weight reuse within a step, making memory traffic expensive relative to arithmetic. Serving several requests together can reuse weight tiles across their new-token vectors, but raises KV capacity and scheduling demands. Longer contexts also increase attention reads from the KV cache. Capacity, bandwidth and token dependency are separate constraints: a model may fit yet move weights too slowly, or have ample bandwidth but wait on a serial chain of computation and communication.

1.  Take the current request tokens and gather the context state needed by each layer.
2.  Apply projections, reusing weights across concurrent requests where the batch permits it.
3.  Read prior keys and values for attention and append the newly produced cache entries.
4.  Complete logits and token selection before the next dependent step can proceed for that request.
5.  Measure generated-token latency, weight traffic, KV traffic and queueing while varying batch and context independently.

## Science and assumptions

### Weight-streaming decode floor

$$
T_{\mathrm{step}}\ge\frac{W_{\mathrm{read}}}{B_{\mathrm{memory}}}
$$

Decode-step time is at least actually read weight bytes divided by sustainable memory bandwidth.

Symbols and units

T\_step

Decode-step time lower bound from weight traffic

s

W\_read

Weight bytes actually read during the step

bytes

B\_memory

Sustainable useful memory bandwidth

bytes/s

Wread is weight bytes actually read per step and Bmemory is sustainable bytes/s; Tstep is seconds.

Assumptions A bandwidth lower bound; cached weights reduce Wread and KV traffic, arithmetic and synchronization can add time.

## Engineering example

Synthetic weights: reading 20 GB each step through 1 TB/s requires at least 20 ms before other costs, or at most 50 steps/s. Batching four requests can share that weight pass, but does not make each request’s token dependency disappear.

## Distinctions and tradeoffs

One active request does not generally generate all its future tokens independently; more peak matrix FLOPS alone cannot remove autoregressive dependence.

Batching improves weight reuse but consumes more cache capacity and can increase latency; reducing precision saves bytes only if kernels and quality support the representation.

Decode makes the difference between nominal arithmetic capacity and delivered responses visible, particularly at low concurrency and strict per-token deadlines.

## Specifications and interfaces

Throughput units

Generated tokens/s versus requests/s

State batch and accepted output criteria; these are not prompt tokens/s.

KV scaling

Requests × context × stored head dimensions

Grouped-query attention and cache precision change the byte count.

### Interfaces

Weights, new token and KV Input

Repeated parameter reads and context-dependent attention state

Next token and appended KV Output

New output plus persistent request state

## Check your understanding

### Which change can improve weight reuse but increase KV memory demand?

1.  Serving more requests together
2.  Doubling only the clock label
3.  Deleting the runtime dependency rules

Answer and reasoning

Serving more requests together

A batch shares weight use across requests, but each request needs its own context state unless specific prefixes can be shared.

## Evidence and scope

Example rates are invented and assume one weight pass. Real caching, quantization, expert activation and tensor partitioning change the bytes and must be profiled.

Follow a modeled batched decode critical path and energy budget; prefill, expert routing and scheduling differences are comparison context rather than separately simulated algorithms.

## Sources

-   [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180)
-   [Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multicore Architectures](https://www2.eecs.berkeley.edu/Pubs/TechRpts/2008/EECS-2008-134.html)

[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/decode-bandwidth-trace-543a70b8ef7fc754.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)

## Continue exploring

### Foundations

-   [Key–value cache](https://semiaifoundry.com/learn/atlas/kv-cache/)
-   [Historical memory-wall evidence](https://semiaifoundry.com/learn/atlas/memory-wall-trends/)

### Related lessons

-   [Inference serving and request scheduling](https://semiaifoundry.com/learn/atlas/serving/)
-   [Prefill and prompt processing](https://semiaifoundry.com/learn/atlas/prefill-resource-trace/)