The mechanism
Prefill evaluates the input prompt and builds the state needed to start generation. Because the prompt tokens are already known, many projections can be expressed as matrix–matrix work, allowing a weight tile to contribute to multiple token positions. Attention has causal dependencies, but implementations need not generate prompt tokens one at a time. Longer prompts raise both arithmetic and attention-memory demands; fused attention can avoid materializing a full score matrix. Prefill often uses arithmetic units more effectively than low-batch decode, but short prompts, poor shapes, host overhead and memory traffic can still dominate. Time to first token includes queueing as well as this phase.
- Prepare prompt embeddings for the known token positions and schedule the relevant model layers.
- Group token activations into matrix operands so loaded weights can be reused across multiple positions.
- Compute causal attention with a legal blocked schedule and maintain the key/value state required by future tokens.
- Manage temporary activations and workspaces so long prompts fit without unintended offload.
- Measure queueing, prompt execution and first-token delivery separately to locate the service bottleneck.
Science and assumptions
Projection work
A dense projection uses approximately two times token positions times input width times output width floating-point operations.
Symbols and units
- F_projection
- Dense projection arithmetic count
FLOP
- T
- Processed token positions
tokens
- d_in
- Input feature width
features/token
- d_out
- Output feature width
features/token
T is processed token positions, din and dout are feature dimensions, and F counts multiply and add separately.
Assumptions Dense linear projection only; attention, normalization, embeddings and output generation are additional work.
Engineering example
Synthetic comparison: processing 128 known token vectors together can reuse the same weight tile across those rows. Processing one new vector offers much less reuse unless requests are batched; neither case changes the stored parameter count.
Distinctions and tradeoffs
Causal attention does not require the already-known prompt to be processed as serial token generation.
Batching prompts improves reuse but can increase queueing or interfere with decode deadlines; chunking limits a single prompt’s scheduling footprint.
Comparing prefill and decode requires fixing prompt length, batch, model, precision and cache behavior because they expose different matrix shapes and traffic.
Specifications and interfaces
- Work unit
- Known prompt tokens
Separate prompt tokens from generated tokens and the number of requests.
- Service metric
- Time to first token, seconds
Includes admission and scheduling effects when measured end to end.
Interfaces
- Prompt tensors and weights Input
- Token activations and model parameters
- Decode-ready state Output
- KV cache and final prompt-position result
Check your understanding
Why can prefill reuse a weight tile more effectively than one-request decode?
- Many prompt positions are available together
- Prompt tokens require no model weights
- Prefill has no memory accesses
Answer and reasoning
Many prompt positions are available together
Known token activations form a matrix, allowing one weight tile to contribute to multiple rows during the same phase.
Evidence and scope
Prefill is not universally compute-bound. The resource trace is illustrative and does not claim an implementation speedup without specified shapes and measured traffic.
No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.