{
  "schemaVersion": 1,
  "id": "atlas-lesson-decode-bandwidth-trace",
  "title": "Autoregressive decode and weight traffic",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Produces dependent new tokens while repeatedly accessing weights and growing KV state.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/decode-bandwidth-trace/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Model–machine co-design",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Example rates are invented and assume one weight pass. Real caching, quantization, expert activation and tensor partitioning change the bytes and must be profiled.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/decode-bandwidth-trace-543a70b8ef7fc754.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#decode-bandwidth-trace",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "704f73bd60f11ebb2a0a234782ef0e84177669f27d220a381b53992315446465",
  "contentLength": 5189,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-decode-bandwidth-trace.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-decode-bandwidth-trace.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Autoregressive decode and weight traffic",
      "start": 40,
      "end": 395
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 395,
      "end": 1553
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1553,
      "end": 1581
    },
    {
      "id": "section-4",
      "title": "Weight-streaming decode floor",
      "start": 1581,
      "end": 2212
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2212,
      "end": 2474
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2474,
      "end": 3000
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3000,
      "end": 3289
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3289,
      "end": 3479
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 3479,
      "end": 3508
    },
    {
      "id": "section-10",
      "title": "Which change can improve weight reuse but increase KV memory demand?",
      "start": 3508,
      "end": 3877
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 3877,
      "end": 4250
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 4250,
      "end": 4766
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 4766,
      "end": 4789
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 4789,
      "end": 4975
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 4975,
      "end": 5189
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#decode-bandwidth-trace"
    },
    {
      "title": "Efficient Memory Management for Large Language Model Serving with PagedAttention",
      "url": "https://arxiv.org/abs/2309.06180"
    },
    {
      "title": "Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multicore Architectures",
      "url": "https://www2.eecs.berkeley.edu/Pubs/TechRpts/2008/EECS-2008-134.html"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/decode-bandwidth-trace-543a70b8ef7fc754.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Key–value cache",
      "url": "https://semiaifoundry.com/learn/atlas/kv-cache/"
    },
    {
      "title": "Historical memory-wall evidence",
      "url": "https://semiaifoundry.com/learn/atlas/memory-wall-trends/"
    },
    {
      "title": "Inference serving and request scheduling",
      "url": "https://semiaifoundry.com/learn/atlas/serving/"
    },
    {
      "title": "Prefill and prompt processing",
      "url": "https://semiaifoundry.com/learn/atlas/prefill-resource-trace/"
    }
  ],
  "recordHash": "60774316b9ca98e42c147bed2d012591214120a93e248ce797a5629f23b542c5",
  "contentFormat": "text/markdown",
  "content": "Model–machine co-design · Atlas v1.3.0\n\n# Autoregressive decode and weight traffic\n\nProduces dependent new tokens while repeatedly accessing weights and growing KV state.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#decode-bandwidth-trace)One model step to accelerator memory hierarchy\n\nLearning objectiveIdentify why low-batch decode may be bandwidth-limited even when weights fit.\n\n## The mechanism\n\nAutoregressive decode generates a token using the preceding context, then uses the selected token to continue. For one request, each projection may have little weight reuse within a step, making memory traffic expensive relative to arithmetic. Serving several requests together can reuse weight tiles across their new-token vectors, but raises KV capacity and scheduling demands. Longer contexts also increase attention reads from the KV cache. Capacity, bandwidth and token dependency are separate constraints: a model may fit yet move weights too slowly, or have ample bandwidth but wait on a serial chain of computation and communication.\n\n1.  Take the current request tokens and gather the context state needed by each layer.\n2.  Apply projections, reusing weights across concurrent requests where the batch permits it.\n3.  Read prior keys and values for attention and append the newly produced cache entries.\n4.  Complete logits and token selection before the next dependent step can proceed for that request.\n5.  Measure generated-token latency, weight traffic, KV traffic and queueing while varying batch and context independently.\n\n## Science and assumptions\n\n### Weight-streaming decode floor\n\n$$\nT_{\\mathrm{step}}\\ge\\frac{W_{\\mathrm{read}}}{B_{\\mathrm{memory}}}\n$$\n\nDecode-step time is at least actually read weight bytes divided by sustainable memory bandwidth.\n\nSymbols and units\n\nT\\_step\n\nDecode-step time lower bound from weight traffic\n\ns\n\nW\\_read\n\nWeight bytes actually read during the step\n\nbytes\n\nB\\_memory\n\nSustainable useful memory bandwidth\n\nbytes/s\n\nWread is weight bytes actually read per step and Bmemory is sustainable bytes/s; Tstep is seconds.\n\nAssumptions A bandwidth lower bound; cached weights reduce Wread and KV traffic, arithmetic and synchronization can add time.\n\n## Engineering example\n\nSynthetic weights: reading 20 GB each step through 1 TB/s requires at least 20 ms before other costs, or at most 50 steps/s. Batching four requests can share that weight pass, but does not make each request’s token dependency disappear.\n\n## Distinctions and tradeoffs\n\nOne active request does not generally generate all its future tokens independently; more peak matrix FLOPS alone cannot remove autoregressive dependence.\n\nBatching improves weight reuse but consumes more cache capacity and can increase latency; reducing precision saves bytes only if kernels and quality support the representation.\n\nDecode makes the difference between nominal arithmetic capacity and delivered responses visible, particularly at low concurrency and strict per-token deadlines.\n\n## Specifications and interfaces\n\nThroughput units\n\nGenerated tokens/s versus requests/s\n\nState batch and accepted output criteria; these are not prompt tokens/s.\n\nKV scaling\n\nRequests × context × stored head dimensions\n\nGrouped-query attention and cache precision change the byte count.\n\n### Interfaces\n\nWeights, new token and KV Input\n\nRepeated parameter reads and context-dependent attention state\n\nNext token and appended KV Output\n\nNew output plus persistent request state\n\n## Check your understanding\n\n### Which change can improve weight reuse but increase KV memory demand?\n\n1.  Serving more requests together\n2.  Doubling only the clock label\n3.  Deleting the runtime dependency rules\n\nAnswer and reasoning\n\nServing more requests together\n\nA batch shares weight use across requests, but each request needs its own context state unless specific prefixes can be shared.\n\n## Evidence and scope\n\nExample rates are invented and assume one weight pass. Real caching, quantization, expert activation and tensor partitioning change the bytes and must be profiled.\n\nFollow a modeled batched decode critical path and energy budget; prefill, expert routing and scheduling differences are comparison context rather than separately simulated algorithms.\n\n## Sources\n\n-   [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180)\n-   [Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multicore Architectures](https://www2.eecs.berkeley.edu/Pubs/TechRpts/2008/EECS-2008-134.html)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/decode-bandwidth-trace-543a70b8ef7fc754.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Key–value cache](https://semiaifoundry.com/learn/atlas/kv-cache/)\n-   [Historical memory-wall evidence](https://semiaifoundry.com/learn/atlas/memory-wall-trends/)\n\n### Related lessons\n\n-   [Inference serving and request scheduling](https://semiaifoundry.com/learn/atlas/serving/)\n-   [Prefill and prompt processing](https://semiaifoundry.com/learn/atlas/prefill-resource-trace/)"
}