{"id":"prefill-resource-trace","level":"software","name":"Prefill and prompt processing","shortName":"Prefill and prompt processing","role":"Processes many known prompt tokens with substantial potential for weight reuse.","description":"Prefill evaluates the input prompt and builds the state needed to start generation. Because the prompt tokens are already known, many projections can be expressed as matrix–matrix work, allowing a weight tile to contribute to multiple token positions. Attention has causal dependencies, but implementations need not generate prompt tokens one at a time. Longer prompts raise both arithmetic and attention-memory demands; fused attention can avoid materializing a full score matrix. Prefill often uses arithmetic units more effectively than low-batch decode, but short prompts, poor shapes, host overhead and memory traffic can still dominate. Time to first token includes queueing as well as this phase.","specs":[{"label":"Work unit","value":"Known prompt tokens","context":"Separate prompt tokens from generated tokens and the number of requests."},{"label":"Service metric","value":"Time to first token, seconds","context":"Includes admission and scheduling effects when measured end to end."}],"interfaces":[{"name":"Prompt tensors and weights","direction":"in","signal":"Token activations and model parameters"},{"name":"Decode-ready state","direction":"out","signal":"KV cache and final prompt-position result"}],"tradeoff":"Batching prompts improves reuse but can increase queueing or interfere with decode deadlines; chunking limits a single prompt’s scheduling footprint.","whyAI":"Comparing prefill and decode requires fixing prompt length, batch, model, precision and cache behavior because they expose different matrix shapes and traffic.","related":["attention","kv-cache","decode-bandwidth-trace"],"sources":[{"title":"FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness","url":"https://arxiv.org/abs/2205.14135"},{"title":"DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving","url":"https://arxiv.org/abs/2401.09670"}],"scale":"One model step to accelerator memory hierarchy","kind":"workload","branch":"Model–machine co-design","subbranch":"Training, prefill and decode","mechanism":["Prepare prompt embeddings for the known token positions and schedule the relevant model layers.","Group token activations into matrix operands so loaded weights can be reused across multiple positions.","Compute causal attention with a legal blocked schedule and maintain the key/value state required by future tokens.","Manage temporary activations and workspaces so long prompts fit without unintended offload.","Measure queueing, prompt execution and first-token delivery separately to locate the service bottleneck."],"misconception":"Causal attention does not require the already-known prompt to be processed as serial token generation.","check":{"question":"Why can prefill reuse a weight tile more effectively than one-request decode?","options":["Many prompt positions are available together","Prompt tokens require no model weights","Prefill has no memory accesses"],"answer":0,"explanation":"Known token activations form a matrix, allowing one weight tile to contribute to multiple rows during the same phase."},"science":[{"title":"Projection work","equation":"Fprojection ≈ 2Tdin dout","explanation":"T is processed token positions, din and dout are feature dimensions, and F counts multiply and add separately.","assumptions":"Dense linear projection only; attention, normalization, embeddings and output generation are additional work."}],"prerequisites":["attention","memory-wall-trends"],"learningObjective":"Explain prompt-phase reuse and separate its compute, memory and service-time constraints.","workstream":"W11","visualFamily":"model-workload","engineeringExample":"Synthetic comparison: processing 128 known token vectors together can reuse the same weight tile across those rows. Processing one new vector offers much less reuse unless requests are batched; neither case changes the stored parameter count.","evidenceNotes":"Prefill is not universally compute-bound. The resource trace is illustrative and does not claim an implementation speedup without specified shapes and measured traffic.","labId":"library","scope":"No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.","investigationId":"execution","labScope":"No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions."}