Key–value cache

The mechanism

An autoregressive transformer would waste work if it recomputed every previous key and value at each step. A KV cache stores these intermediate arrays for later attention. It is distinct from model weights and from a hardware cache such as L2. Its size grows with the active sequence and batch, and depends strongly on the number of key–value heads and precision. Serving systems manage these arrays across changing requests, often with block-based allocation.

Store each new token’s keys and values for every relevant layer.
Read the past cache when computing the next token’s attention.
Release or reuse allocation when a request finishes.

Science and assumptions

The cache grows with every retained position

\begin{aligned}M_{KV}&=2BLTH_{KV}D_hs\\\Delta M_{\mathrm{per\ token,per\ sequence}}&=2LH_{KV}D_hs\end{aligned}

Raw KV bytes equal two for keys and values times sequences, cached layers, retained positions, KV heads, head width and bytes per scalar. One additional token in one sequence adds the same product without sequence count or context length.

Symbols and units

M_KV: Raw key/value cache payload
bytes
B: Concurrent sequences
sequences
L: Cached attention layers
layers
T: Retained positions per sequence
tokens/sequence
H_KV: Key/value heads per layer
heads/layer
D_h: Values per head for each key or value
values/head
s: Stored size per scalar value
bytes/value
ΔM_per token,per sequence: Additional raw KV payload for one retained token in one sequence
bytes

The factor two counts keys and values. Multiply sequences B, cached layers L, retained positions T, KV heads Hkv, head width Dh, and bytes per value s. Sharing one KV head among several query heads reduces this payload without requiring the same reduction in query heads.

Assumptions This uniform full-attention estimate assumes equal key/value widths and sequence lengths, no prefix sharing, no replication, and no compression or allocator overhead. Sliding-window and hybrid models need a layer-by-layer sum. It is not an estimate of Astra’s hidden architecture.

Engineering example

Doubling context length doubles KV storage at fixed batch, layer count, head count and head dimension; it does not double parameter storage.

Distinctions and tradeoffs

The KV cache is a software-managed tensor, not the chip’s hardware cache.

More context and concurrent requests consume memory. Quantization, grouped-query attention, and allocation policies change the cost.

KV memory often limits long-context serving concurrency.

Specifications and interfaces

Payload estimate: 2 × B × L × T × Hkv × Dh × s
Uniform uncompressed full-attention payload: batch B, layers L, tokens T, KV heads Hkv, head width Dh, and s bytes/value; excludes allocation overhead.
Scope: Request state
Not a permanent update to trained weights.

Interfaces

Attention layers Input: New key and value vectors
Later decoding Output: Reusable attention history

Check your understanding

Does adding a token to the KV cache retrain the model?

Yes
Only at night
No

Answer and reasoning

It stores inference intermediates; the learned weights are unchanged.

Historical context

2023 · PagedAttention reorganizes inference memory

Kwon and colleagues described a paged KV-cache layout that reduces fragmentation and supports sharing, connecting language-model serving to operating-system memory-management ideas.

Kwon et al. — Efficient Memory Management for Large Language Model Serving with PagedAttention

Evidence and scope

The formula excludes allocator fragmentation and quantized-cache variants.

Change token count, concurrency and KV-head count to measure their raw cache consequence; the tokenizer itself is not executed.