{"id":"decode-bandwidth-trace","level":"software","name":"Autoregressive decode and weight traffic","shortName":"Autoregressive decode and weight traffic","role":"Produces dependent new tokens while repeatedly accessing weights and growing KV state.","description":"Autoregressive decode generates a token using the preceding context, then uses the selected token to continue. For one request, each projection may have little weight reuse within a step, making memory traffic expensive relative to arithmetic. Serving several requests together can reuse weight tiles across their new-token vectors, but raises KV capacity and scheduling demands. Longer contexts also increase attention reads from the KV cache. Capacity, bandwidth and token dependency are separate constraints: a model may fit yet move weights too slowly, or have ample bandwidth but wait on a serial chain of computation and communication.","specs":[{"label":"Throughput units","value":"Generated tokens/s versus requests/s","context":"State batch and accepted output criteria; these are not prompt tokens/s."},{"label":"KV scaling","value":"Requests × context × stored head dimensions","context":"Grouped-query attention and cache precision change the byte count."}],"interfaces":[{"name":"Weights, new token and KV","direction":"in","signal":"Repeated parameter reads and context-dependent attention state"},{"name":"Next token and appended KV","direction":"out","signal":"New output plus persistent request state"}],"tradeoff":"Batching improves weight reuse but consumes more cache capacity and can increase latency; reducing precision saves bytes only if kernels and quality support the representation.","whyAI":"Decode makes the difference between nominal arithmetic capacity and delivered responses visible, particularly at low concurrency and strict per-token deadlines.","related":["kv-cache","serving","prefill-resource-trace"],"sources":[{"title":"Efficient Memory Management for Large Language Model Serving with PagedAttention","url":"https://arxiv.org/abs/2309.06180"},{"title":"Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multicore Architectures","url":"https://www2.eecs.berkeley.edu/Pubs/TechRpts/2008/EECS-2008-134.html"}],"scale":"One model step to accelerator memory hierarchy","kind":"workload","branch":"Model–machine co-design","subbranch":"Training, prefill and decode","mechanism":["Take the current request tokens and gather the context state needed by each layer.","Apply projections, reusing weights across concurrent requests where the batch permits it.","Read prior keys and values for attention and append the newly produced cache entries.","Complete logits and token selection before the next dependent step can proceed for that request.","Measure generated-token latency, weight traffic, KV traffic and queueing while varying batch and context independently."],"misconception":"One active request does not generally generate all its future tokens independently; more peak matrix FLOPS alone cannot remove autoregressive dependence.","check":{"question":"Which change can improve weight reuse but increase KV memory demand?","options":["Serving more requests together","Doubling only the clock label","Deleting the runtime dependency rules"],"answer":0,"explanation":"A batch shares weight use across requests, but each request needs its own context state unless specific prefixes can be shared."},"science":[{"title":"Weight-streaming decode floor","equation":"Tstep ≥ Wread / Bmemory","explanation":"Wread is weight bytes actually read per step and Bmemory is sustainable bytes/s; Tstep is seconds.","assumptions":"A bandwidth lower bound; cached weights reduce Wread and KV traffic, arithmetic and synchronization can add time."}],"prerequisites":["kv-cache","memory-wall-trends"],"learningObjective":"Identify why low-batch decode may be bandwidth-limited even when weights fit.","workstream":"W11","visualFamily":"model-workload","engineeringExample":"Synthetic weights: reading 20 GB each step through 1 TB/s requires at least 20 ms before other costs, or at most 50 steps/s. Batching four requests can share that weight pass, but does not make each request’s token dependency disappear.","evidenceNotes":"Example rates are invented and assume one weight pass. Real caching, quantization, expert activation and tensor partitioning change the bytes and must be profiled.","labId":"token-system-path","scope":"Follow a modeled batched decode critical path and energy budget; prefill, expert routing and scheduling differences are comparison context rather than separately simulated algorithms.","investigationId":"execution","labScope":"Follow a modeled batched decode critical path and energy budget; prefill, expert routing and scheduling differences are comparison context rather than separately simulated algorithms."}