{
  "schemaVersion": 1,
  "id": "atlas-lesson-prefill-resource-trace",
  "title": "Prefill and prompt processing",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Processes many known prompt tokens with substantial potential for weight reuse.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/prefill-resource-trace/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Model–machine co-design",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Prefill is not universally compute-bound. The resource trace is illustrative and does not claim an implementation speedup without specified shapes and measured traffic.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/prefill-resource-trace-a76d037833bd69f7.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#prefill-resource-trace",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "51cc844e24447080c27f7fe81021546588ac145990965ea6050980f0f3ac1b82",
  "contentLength": 5105,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-prefill-resource-trace.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-prefill-resource-trace.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Prefill and prompt processing",
      "start": 40,
      "end": 389
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 389,
      "end": 1645
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1645,
      "end": 1673
    },
    {
      "id": "section-4",
      "title": "Projection work",
      "start": 1673,
      "end": 2333
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2333,
      "end": 2601
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2601,
      "end": 3048
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3048,
      "end": 3303
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3303,
      "end": 3463
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 3463,
      "end": 3492
    },
    {
      "id": "section-10",
      "title": "Why can prefill reuse a weight tile more effectively than one-request decode?",
      "start": 3492,
      "end": 3890
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 3890,
      "end": 4227
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 4227,
      "end": 4695
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 4695,
      "end": 4718
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 4718,
      "end": 4904
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 4904,
      "end": 5105
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#prefill-resource-trace"
    },
    {
      "title": "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness",
      "url": "https://arxiv.org/abs/2205.14135"
    },
    {
      "title": "DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving",
      "url": "https://arxiv.org/abs/2401.09670"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/prefill-resource-trace-a76d037833bd69f7.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Self-attention",
      "url": "https://semiaifoundry.com/learn/atlas/attention/"
    },
    {
      "title": "Historical memory-wall evidence",
      "url": "https://semiaifoundry.com/learn/atlas/memory-wall-trends/"
    },
    {
      "title": "Key–value cache",
      "url": "https://semiaifoundry.com/learn/atlas/kv-cache/"
    },
    {
      "title": "Autoregressive decode and weight traffic",
      "url": "https://semiaifoundry.com/learn/atlas/decode-bandwidth-trace/"
    }
  ],
  "recordHash": "cb7fc32f7d32b416998be6612075db81d2b0686c4c2d0cbd3d314a4d00ed07e5",
  "contentFormat": "text/markdown",
  "content": "Model–machine co-design · Atlas v1.3.0\n\n# Prefill and prompt processing\n\nProcesses many known prompt tokens with substantial potential for weight reuse.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#prefill-resource-trace)One model step to accelerator memory hierarchy\n\nLearning objectiveExplain prompt-phase reuse and separate its compute, memory and service-time constraints.\n\n## The mechanism\n\nPrefill evaluates the input prompt and builds the state needed to start generation. Because the prompt tokens are already known, many projections can be expressed as matrix–matrix work, allowing a weight tile to contribute to multiple token positions. Attention has causal dependencies, but implementations need not generate prompt tokens one at a time. Longer prompts raise both arithmetic and attention-memory demands; fused attention can avoid materializing a full score matrix. Prefill often uses arithmetic units more effectively than low-batch decode, but short prompts, poor shapes, host overhead and memory traffic can still dominate. Time to first token includes queueing as well as this phase.\n\n1.  Prepare prompt embeddings for the known token positions and schedule the relevant model layers.\n2.  Group token activations into matrix operands so loaded weights can be reused across multiple positions.\n3.  Compute causal attention with a legal blocked schedule and maintain the key/value state required by future tokens.\n4.  Manage temporary activations and workspaces so long prompts fit without unintended offload.\n5.  Measure queueing, prompt execution and first-token delivery separately to locate the service bottleneck.\n\n## Science and assumptions\n\n### Projection work\n\n$$\nF_{\\mathrm{projection}}\\approx2T d_{\\mathrm{in}}d_{\\mathrm{out}}\n$$\n\nA dense projection uses approximately two times token positions times input width times output width floating-point operations.\n\nSymbols and units\n\nF\\_projection\n\nDense projection arithmetic count\n\nFLOP\n\nT\n\nProcessed token positions\n\ntokens\n\nd\\_in\n\nInput feature width\n\nfeatures/token\n\nd\\_out\n\nOutput feature width\n\nfeatures/token\n\nT is processed token positions, din and dout are feature dimensions, and F counts multiply and add separately.\n\nAssumptions Dense linear projection only; attention, normalization, embeddings and output generation are additional work.\n\n## Engineering example\n\nSynthetic comparison: processing 128 known token vectors together can reuse the same weight tile across those rows. Processing one new vector offers much less reuse unless requests are batched; neither case changes the stored parameter count.\n\n## Distinctions and tradeoffs\n\nCausal attention does not require the already-known prompt to be processed as serial token generation.\n\nBatching prompts improves reuse but can increase queueing or interfere with decode deadlines; chunking limits a single prompt’s scheduling footprint.\n\nComparing prefill and decode requires fixing prompt length, batch, model, precision and cache behavior because they expose different matrix shapes and traffic.\n\n## Specifications and interfaces\n\nWork unit\n\nKnown prompt tokens\n\nSeparate prompt tokens from generated tokens and the number of requests.\n\nService metric\n\nTime to first token, seconds\n\nIncludes admission and scheduling effects when measured end to end.\n\n### Interfaces\n\nPrompt tensors and weights Input\n\nToken activations and model parameters\n\nDecode-ready state Output\n\nKV cache and final prompt-position result\n\n## Check your understanding\n\n### Why can prefill reuse a weight tile more effectively than one-request decode?\n\n1.  Many prompt positions are available together\n2.  Prompt tokens require no model weights\n3.  Prefill has no memory accesses\n\nAnswer and reasoning\n\nMany prompt positions are available together\n\nKnown token activations form a matrix, allowing one weight tile to contribute to multiple rows during the same phase.\n\n## Evidence and scope\n\nPrefill is not universally compute-bound. The resource trace is illustrative and does not claim an implementation speedup without specified shapes and measured traffic.\n\nNo current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.\n\n## Sources\n\n-   [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://arxiv.org/abs/2205.14135)\n-   [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://arxiv.org/abs/2401.09670)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/prefill-resource-trace-a76d037833bd69f7.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Self-attention](https://semiaifoundry.com/learn/atlas/attention/)\n-   [Historical memory-wall evidence](https://semiaifoundry.com/learn/atlas/memory-wall-trends/)\n\n### Related lessons\n\n-   [Key–value cache](https://semiaifoundry.com/learn/atlas/kv-cache/)\n-   [Autoregressive decode and weight traffic](https://semiaifoundry.com/learn/atlas/decode-bandwidth-trace/)"
}