# Key–value cache

Canonical: https://semiaifoundry.com/learn/atlas/kv-cache/
Revision: 2026-10-07
Rights: https://semiaifoundry.com/ai-atlas/licenses.html

Generation · Atlas v1.3.0

# Key–value cache

Reuse earlier attention state during generation.

[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#kv-cache)Logical structure · mapped onto physical hardware

Learning objectiveCalculate KV growth per sequence and layer.

## The mechanism

An autoregressive transformer would waste work if it recomputed every previous key and value at each step. A KV cache stores these intermediate arrays for later attention. It is distinct from model weights and from a hardware cache such as L2. Its size grows with the active sequence and batch, and depends strongly on the number of key–value heads and precision. Serving systems manage these arrays across changing requests, often with block-based allocation.

1.  Store each new token’s keys and values for every relevant layer.
2.  Read the past cache when computing the next token’s attention.
3.  Release or reuse allocation when a request finishes.

## Science and assumptions

### The cache grows with every retained position

$$
\begin{aligned}M_{KV}&=2BLTH_{KV}D_hs\\\Delta M_{\mathrm{per\ token,per\ sequence}}&=2LH_{KV}D_hs\end{aligned}
$$

Raw KV bytes equal two for keys and values times sequences, cached layers, retained positions, KV heads, head width and bytes per scalar. One additional token in one sequence adds the same product without sequence count or context length.

Symbols and units

M\_KV

Raw key/value cache payload

bytes

B

Concurrent sequences

sequences

L

Cached attention layers

layers

T

Retained positions per sequence

tokens/sequence

H\_KV

Key/value heads per layer

heads/layer

D\_h

Values per head for each key or value

values/head

s

Stored size per scalar value

bytes/value

ΔM\_per token,per sequence

Additional raw KV payload for one retained token in one sequence

bytes

The factor two counts keys and values. Multiply sequences B, cached layers L, retained positions T, KV heads Hkv, head width Dh, and bytes per value s. Sharing one KV head among several query heads reduces this payload without requiring the same reduction in query heads.

Assumptions This uniform full-attention estimate assumes equal key/value widths and sequence lengths, no prefix sharing, no replication, and no compression or allocator overhead. Sliding-window and hybrid models need a layer-by-layer sum. It is not an estimate of Astra’s hidden architecture.

## Engineering example

Doubling context length doubles KV storage at fixed batch, layer count, head count and head dimension; it does not double parameter storage.

## Distinctions and tradeoffs

The KV cache is a software-managed tensor, not the chip’s hardware cache.

More context and concurrent requests consume memory. Quantization, grouped-query attention, and allocation policies change the cost.

KV memory often limits long-context serving concurrency.

## Specifications and interfaces

Payload estimate

2 × B × L × T × Hkv × Dh × s

Uniform uncompressed full-attention payload: batch B, layers L, tokens T, KV heads Hkv, head width Dh, and s bytes/value; excludes allocation overhead.

Scope

Request state

Not a permanent update to trained weights.

### Interfaces

Attention layers Input

New key and value vectors

Later decoding Output

Reusable attention history

## Check your understanding

### Does adding a token to the KV cache retrain the model?

1.  Yes
2.  Only at night
3.  No

Answer and reasoning

No

It stores inference intermediates; the learned weights are unchanged.

## Historical context

### 2023 · PagedAttention reorganizes inference memory

Kwon and colleagues described a paged KV-cache layout that reduces fragmentation and supports sharing, connecting language-model serving to operating-system memory-management ideas.

[Kwon et al. — Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180)

## Evidence and scope

The formula excludes allocator fragmentation and quantized-cache variants.

Change token count, concurrency and KV-head count to measure their raw cache consequence; the tokenizer itself is not executed.

## Sources

-   [Kwon et al. — PagedAttention](https://arxiv.org/abs/2309.06180)
-   [Ainslie et al. — GQA](https://arxiv.org/abs/2305.13245)

[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/kv-cache-0a723b30e74dbe69.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)

## Continue exploring

### Foundations

-   [Self-attention](https://semiaifoundry.com/learn/atlas/attention/)

### Related lessons

-   [High-bandwidth memory stack](https://semiaifoundry.com/learn/atlas/hbm-stack/)
-   [Inference serving and request scheduling](https://semiaifoundry.com/learn/atlas/serving/)
-   [Output head & decoding](https://semiaifoundry.com/learn/atlas/output-head/)