{
  "schemaVersion": 1,
  "id": "atlas-lesson-kv-cache",
  "title": "Key–value cache",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Reuse earlier attention state during generation.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/kv-cache/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Generation",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "The formula excludes allocator fragmentation and quantized-cache variants.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/kv-cache-0a723b30e74dbe69.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#kv-cache",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "7adbca0b39793013d27561824c6eb03cd63ffd2744c5846922affff52e8fe5eb",
  "contentLength": 4837,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-kv-cache.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-kv-cache.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Key–value cache",
      "start": 27,
      "end": 274
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 274,
      "end": 948
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 948,
      "end": 976
    },
    {
      "id": "section-4",
      "title": "The cache grows with every retained position",
      "start": 976,
      "end": 2390
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2390,
      "end": 2556
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2556,
      "end": 2854
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 2854,
      "end": 3155
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3155,
      "end": 3273
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 3273,
      "end": 3302
    },
    {
      "id": "section-10",
      "title": "Does adding a token to the KV cache retrain the model?",
      "start": 3302,
      "end": 3493
    },
    {
      "id": "section-11",
      "title": "Historical context",
      "start": 3493,
      "end": 3516
    },
    {
      "id": "section-12",
      "title": "2023 · PagedAttention reorganizes inference memory",
      "start": 3516,
      "end": 3887
    },
    {
      "id": "section-13",
      "title": "Evidence and scope",
      "start": 3887,
      "end": 4115
    },
    {
      "id": "section-14",
      "title": "Sources",
      "start": 4115,
      "end": 4445
    },
    {
      "id": "section-15",
      "title": "Continue exploring",
      "start": 4445,
      "end": 4468
    },
    {
      "id": "section-16",
      "title": "Foundations",
      "start": 4468,
      "end": 4557
    },
    {
      "id": "section-17",
      "title": "Related lessons",
      "start": 4557,
      "end": 4837
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#kv-cache"
    },
    {
      "title": "Kwon et al. — Efficient Memory Management for Large Language Model Serving with PagedAttention",
      "url": "https://arxiv.org/abs/2309.06180"
    },
    {
      "title": "Ainslie et al. — GQA",
      "url": "https://arxiv.org/abs/2305.13245"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/kv-cache-0a723b30e74dbe69.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Self-attention",
      "url": "https://semiaifoundry.com/learn/atlas/attention/"
    },
    {
      "title": "High-bandwidth memory stack",
      "url": "https://semiaifoundry.com/learn/atlas/hbm-stack/"
    },
    {
      "title": "Inference serving and request scheduling",
      "url": "https://semiaifoundry.com/learn/atlas/serving/"
    },
    {
      "title": "Output head & decoding",
      "url": "https://semiaifoundry.com/learn/atlas/output-head/"
    }
  ],
  "recordHash": "9b9a5acddba13d8240242cf0bd651c2b17c7cf0a83192825a867340298faf28a",
  "contentFormat": "text/markdown",
  "content": "Generation · Atlas v1.3.0\n\n# Key–value cache\n\nReuse earlier attention state during generation.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#kv-cache)Logical structure · mapped onto physical hardware\n\nLearning objectiveCalculate KV growth per sequence and layer.\n\n## The mechanism\n\nAn autoregressive transformer would waste work if it recomputed every previous key and value at each step. A KV cache stores these intermediate arrays for later attention. It is distinct from model weights and from a hardware cache such as L2. Its size grows with the active sequence and batch, and depends strongly on the number of key–value heads and precision. Serving systems manage these arrays across changing requests, often with block-based allocation.\n\n1.  Store each new token’s keys and values for every relevant layer.\n2.  Read the past cache when computing the next token’s attention.\n3.  Release or reuse allocation when a request finishes.\n\n## Science and assumptions\n\n### The cache grows with every retained position\n\n$$\n\\begin{aligned}M_{KV}&=2BLTH_{KV}D_hs\\\\\\Delta M_{\\mathrm{per\\ token,per\\ sequence}}&=2LH_{KV}D_hs\\end{aligned}\n$$\n\nRaw KV bytes equal two for keys and values times sequences, cached layers, retained positions, KV heads, head width and bytes per scalar. One additional token in one sequence adds the same product without sequence count or context length.\n\nSymbols and units\n\nM\\_KV\n\nRaw key/value cache payload\n\nbytes\n\nB\n\nConcurrent sequences\n\nsequences\n\nL\n\nCached attention layers\n\nlayers\n\nT\n\nRetained positions per sequence\n\ntokens/sequence\n\nH\\_KV\n\nKey/value heads per layer\n\nheads/layer\n\nD\\_h\n\nValues per head for each key or value\n\nvalues/head\n\ns\n\nStored size per scalar value\n\nbytes/value\n\nΔM\\_per token,per sequence\n\nAdditional raw KV payload for one retained token in one sequence\n\nbytes\n\nThe factor two counts keys and values. Multiply sequences B, cached layers L, retained positions T, KV heads Hkv, head width Dh, and bytes per value s. Sharing one KV head among several query heads reduces this payload without requiring the same reduction in query heads.\n\nAssumptions This uniform full-attention estimate assumes equal key/value widths and sequence lengths, no prefix sharing, no replication, and no compression or allocator overhead. Sliding-window and hybrid models need a layer-by-layer sum. It is not an estimate of Astra’s hidden architecture.\n\n## Engineering example\n\nDoubling context length doubles KV storage at fixed batch, layer count, head count and head dimension; it does not double parameter storage.\n\n## Distinctions and tradeoffs\n\nThe KV cache is a software-managed tensor, not the chip’s hardware cache.\n\nMore context and concurrent requests consume memory. Quantization, grouped-query attention, and allocation policies change the cost.\n\nKV memory often limits long-context serving concurrency.\n\n## Specifications and interfaces\n\nPayload estimate\n\n2 × B × L × T × Hkv × Dh × s\n\nUniform uncompressed full-attention payload: batch B, layers L, tokens T, KV heads Hkv, head width Dh, and s bytes/value; excludes allocation overhead.\n\nScope\n\nRequest state\n\nNot a permanent update to trained weights.\n\n### Interfaces\n\nAttention layers Input\n\nNew key and value vectors\n\nLater decoding Output\n\nReusable attention history\n\n## Check your understanding\n\n### Does adding a token to the KV cache retrain the model?\n\n1.  Yes\n2.  Only at night\n3.  No\n\nAnswer and reasoning\n\nNo\n\nIt stores inference intermediates; the learned weights are unchanged.\n\n## Historical context\n\n### 2023 · PagedAttention reorganizes inference memory\n\nKwon and colleagues described a paged KV-cache layout that reduces fragmentation and supports sharing, connecting language-model serving to operating-system memory-management ideas.\n\n[Kwon et al. — Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180)\n\n## Evidence and scope\n\nThe formula excludes allocator fragmentation and quantized-cache variants.\n\nChange token count, concurrency and KV-head count to measure their raw cache consequence; the tokenizer itself is not executed.\n\n## Sources\n\n-   [Kwon et al. — PagedAttention](https://arxiv.org/abs/2309.06180)\n-   [Ainslie et al. — GQA](https://arxiv.org/abs/2305.13245)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/kv-cache-0a723b30e74dbe69.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Self-attention](https://semiaifoundry.com/learn/atlas/attention/)\n\n### Related lessons\n\n-   [High-bandwidth memory stack](https://semiaifoundry.com/learn/atlas/hbm-stack/)\n-   [Inference serving and request scheduling](https://semiaifoundry.com/learn/atlas/serving/)\n-   [Output head & decoding](https://semiaifoundry.com/learn/atlas/output-head/)"
}