{"id":"kv-cache","level":"model","name":"Key–value cache","shortName":"Key–value cache","role":"Reuse earlier attention state during generation.","description":"An autoregressive transformer would waste work if it recomputed every previous key and value at each step. A KV cache stores these intermediate arrays for later attention. It is distinct from model weights and from a hardware cache such as L2. Its size grows with the active sequence and batch, and depends strongly on the number of key–value heads and precision. Serving systems manage these arrays across changing requests, often with block-based allocation.","mechanism":["Store each new token’s keys and values for every relevant layer.","Read the past cache when computing the next token’s attention.","Release or reuse allocation when a request finishes."],"specs":[{"label":"Payload estimate","value":"2 × B × L × T × Hkv × Dh × s","context":"Uniform uncompressed full-attention payload: batch B, layers L, tokens T, KV heads Hkv, head width Dh, and s bytes/value; excludes allocation overhead."},{"label":"Scope","value":"Request state","context":"Not a permanent update to trained weights."}],"interfaces":[{"name":"Attention layers","direction":"in","signal":"New key and value vectors"},{"name":"Later decoding","direction":"out","signal":"Reusable attention history"}],"tradeoff":"More context and concurrent requests consume memory. Quantization, grouped-query attention, and allocation policies change the cost.","whyAI":"KV memory often limits long-context serving concurrency.","related":["attention","hbm-stack","serving","output-head"],"sources":[{"title":"Kwon et al. — PagedAttention","url":"https://arxiv.org/abs/2309.06180"},{"title":"Ainslie et al. — GQA","url":"https://arxiv.org/abs/2305.13245"}],"scale":"Logical structure · mapped onto physical hardware","kind":"kv-cache","misconception":"The KV cache is a software-managed tensor, not the chip’s hardware cache.","check":{"question":"Does adding a token to the KV cache retrain the model?","options":["Yes","Only at night","No"],"answer":2,"explanation":"It stores inference intermediates; the learned weights are unchanged."},"branch":"Generation","subbranch":"Attention state and memory","science":[{"title":"The cache grows with every retained position","equation":"M_KV = 2·B·L·T·Hkv·Dh·s bytes; ΔM per token per sequence = 2·L·Hkv·Dh·s","explanation":"The factor two counts keys and values. Multiply sequences B, cached layers L, retained positions T, KV heads Hkv, head width Dh, and bytes per value s. Sharing one KV head among several query heads reduces this payload without requiring the same reduction in query heads.","assumptions":"This uniform full-attention estimate assumes equal key/value widths and sequence lengths, no prefix sharing, no replication, and no compression or allocator overhead. Sliding-window and hybrid models need a layer-by-layer sum. It is not an estimate of Astra’s hidden architecture."}],"history":[{"year":2023,"title":"PagedAttention reorganizes inference memory","significance":"Kwon and colleagues described a paged KV-cache layout that reduces fragmentation and supports sharing, connecting language-model serving to operating-system memory-management ideas.","source":{"title":"Kwon et al. — Efficient Memory Management for Large Language Model Serving with PagedAttention","url":"https://arxiv.org/abs/2309.06180"}}],"learningObjective":"Calculate KV growth per sequence and layer.","engineeringExample":"Doubling context length doubles KV storage at fixed batch, layer count, head count and head dimension; it does not double parameter storage.","evidenceNotes":"The formula excludes allocator fragmentation and quantized-cache variants.","visualFamily":"model-workload","prerequisites":["attention"],"labId":"kv","scope":"Change token count, concurrency and KV-head count to measure their raw cache consequence; the tokenizer itself is not executed.","investigationId":"execution","labScope":"Change token count, concurrency and KV-head count to measure their raw cache consequence; the tokenizer itself is not executed."}