Prefill, decode and the moving KV working set

The mechanism

Prefill processes an input sequence and creates state for subsequent generation. Decode repeatedly generates new tokens while reading prior context and appending new KV state. Weights are reused across requests, while KV belongs to a model/version, sequence position and attention configuration. A reusable prefix can be retained or offloaded, but restoring it competes with recomputing it. Moving large KV payloads between prefill and decode nodes adds network work and coordination. Larger near-compute capacity can help only if the state is available before the next dependent operation and the placement preserves correctness.

Identify whether the workload is prefill, decode or a mixture; record batch, context and attention architecture.
Budget the live KV payload and append/write behavior separately from weight reads.
Keep imminent mutable state accessible; choose a policy for inactive prefixes, eviction and restoration.
Compare restore transfer time with recomputation and service deadlines under actual contention.
For split serving, trace KV over its physical links and validate model/version, ownership and completion before decode uses it.

Science and assumptions

Conventional KV payload

M_{\mathrm{KV}}=2LH_{\mathrm{KV}}db\sum_s T_s

MKV = 2 L Hkv d b Σs Ts

Symbols and units

MKV: Raw live KV payload
bytes
L: Layers retaining KV
count
Hkv: KV heads per layer
count
d: Values per KV head
count
b: Bytes per stored value
bytes
Ts: Stored tokens in sequence s
tokens

Each stored token has key and value vectors for the specified layers and KV heads.

Assumptions Full conventional KV representation; no sliding windows, compression or architecture-specific state reduction. Excludes paging overhead and padding.

Restore or recompute

t_{\mathrm{restore}}\geq L_{\mathrm{path}}+\frac{V}{B_{\mathrm{path}}}

trestore ≥ Lpath + V/Bpath

Symbols and units

trestore: State-restoration time
s
Lpath: Path setup/completion delay
s
V: Bytes restored
bytes
Bpath: Sustained path payload bandwidth
bytes/s

Compare this transfer lower bound with measured recomputation time and the request deadline.

Assumptions One aggregate transfer; includes no queueing or validation cost and assumes the cache is valid.

Engineering example

Synthetic conventional KV: 32 layers × 8 KV heads × 128 values × 2 bytes × 2 for K/V × 8,192 tokens is 1 GiB per sequence. Eight such sequences need 8 GiB before metadata and other model state.

Distinctions and tradeoffs

Active decoding both reads old context and appends state; placement must match those access requirements.

More cached context saves recomputation but occupies memory and creates movement when placed remotely; a cache hit still has a service cost.

Memory tiers determine concurrency and context support, while the restore path can affect first-token and inter-token latency.

Specifications and interfaces

Decode: Repeated reads and KV appends
An active KV cache is not immutable pretrained weight storage.
Prefix reuse: Must retain semantic validity
Model, token sequence and attention configuration matter.

Interfaces

Prefill Input: Prompt computation and newly produced KV
Decode Output: Ready prior state, appended KV and output token

Check your understanding

Does a large read-oriented tier automatically suit the active KV cache?

No: append writes, access granularity and deadlines must be evaluated
Yes: capacity alone establishes suitability
Yes: KV never changes during generation

Answer and reasoning

No: append writes, access granularity and deadlines must be evaluated

Active decoding both reads old context and appends state; placement must match those access requirements.

Evidence and scope

The equation is an explicitly scoped conventional attention model. It is not a specification of Astra. Any HBF KV placement is a scenario requiring controller/write behavior and serving measurements, not an asserted deployed configuration.

Place read-only weights and mutable KV across specified tiers and inspect transfer paths; no HBF shipping-system benchmark or universal memory semantics are implied.