{ "schemaVersion": 1, "id": "atlas-lesson-speculative-decoding-verification", "title": "Speculative decoding: proposals and verified output", "category": "learning", "format": "Atlas lesson", "description": "Account for target-distributed output tokens while separating draft work, verification and actual elapsed time.", "canonicalUrl": "https://semiaifoundry.com/learn/atlas/speculative-decoding-verification/", "publishedDate": null, "revisionDate": "2026-10-07", "author": null, "topic": "Executable software", "series": null, "license": null, "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html", "version": "1.3.0", "status": "Published educational lesson", "evidenceScope": "The cited ICML paper establishes the strict correction algorithm and the explicitly iid output-count formula. The toy probabilities, output-count distribution and all millisecond values are authored. No private model, serving implementation, benchmark speedup or quality improvement is asserted.", "contentCoverage": "published-lesson", "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/speculative-decoding-verification-f6066fe7fb36a164.json", "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#speculative-decoding-verification", "related": [], "supersedes": [], "supersededBy": [], "contentHash": "911b1bb7541f981e066bc41707b9fe78bdbce7439e344ff6d5c12a8df928f4e0", "contentLength": 9732, "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-speculative-decoding-verification.json", "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-speculative-decoding-verification.md", "sections": [ { "id": "section-1", "title": "Speculative decoding: proposals and verified output", "start": 36, "end": 585 }, { "id": "section-2", "title": "The mechanism", "start": 585, "end": 2547 }, { "id": "section-3", "title": "Science and assumptions", "start": 2547, "end": 2575 }, { "id": "section-4", "title": "Acceptance and rejection correction", "start": 2575, "end": 3672 }, { "id": "section-5", "title": "Output count in the iid acceptance scenario", "start": 3672, "end": 4939 }, { "id": "section-6", "title": "Engineering example", "start": 4939, "end": 5920 }, { "id": "section-7", "title": "Distinctions and tradeoffs", "start": 5920, "end": 6517 }, { "id": "section-8", "title": "Specifications and interfaces", "start": 6517, "end": 6960 }, { "id": "section-9", "title": "Interfaces", "start": 6960, "end": 7167 }, { "id": "section-10", "title": "Check your understanding", "start": 7167, "end": 7196 }, { "id": "section-11", "title": "In the stated toy distributions, a rejected proposal must be corrected from which distribution?", "start": 7196, "end": 7577 }, { "id": "section-12", "title": "Under iid α=0.6 with two proposals, what is the expected output count including correction or bonus?", "start": 7577, "end": 7778 }, { "id": "section-13", "title": "Does preserving p establish that sampled statements are true?", "start": 7778, "end": 7987 }, { "id": "section-14", "title": "Evidence and scope", "start": 7987, "end": 8504 }, { "id": "section-15", "title": "Sources", "start": 8504, "end": 8878 }, { "id": "section-16", "title": "Continue exploring", "start": 8878, "end": 8901 }, { "id": "section-17", "title": "Foundations", "start": 8901, "end": 9268 }, { "id": "section-18", "title": "Related lessons", "start": 9268, "end": 9732 } ], "citations": [ { "title": "Explore in Atlas", "url": "https://semiaifoundry.com/ai-atlas/#speculative-decoding-verification" }, { "title": "Leviathan, Kalman and Matias — Fast Inference from Transformers via Speculative Decoding", "url": "https://proceedings.mlr.press/v202/leviathan23a.html" }, { "title": "Published lesson data", "url": "https://semiaifoundry.com/ai-atlas/lesson-data/speculative-decoding-verification-f6066fe7fb36a164.json" }, { "title": "Atlas releases and licensing", "url": "https://semiaifoundry.com/ai-atlas/licenses.html" }, { "title": "Output head & decoding", "url": "https://semiaifoundry.com/learn/atlas/output-head/" }, { "title": "Autoregressive decode and weight traffic", "url": "https://semiaifoundry.com/learn/atlas/decode-bandwidth-trace/" }, { "title": "Key–value cache", "url": "https://semiaifoundry.com/learn/atlas/kv-cache/" }, { "title": "Evaluation & the AGI frontier", "url": "https://semiaifoundry.com/learn/atlas/evaluation/" }, { "title": "Prefill and prompt processing", "url": "https://semiaifoundry.com/learn/atlas/prefill-resource-trace/" }, { "title": "Prefill/decode disaggregation and useful serving rate", "url": "https://semiaifoundry.com/learn/atlas/disaggregated-serving/" }, { "title": "Runtime, driver and device dispatch", "url": "https://semiaifoundry.com/learn/atlas/runtime-device-dispatch/" }, { "title": "Place the working set, not just the model", "url": "https://semiaifoundry.com/learn/atlas/memory-working-set-placement/" } ], "recordHash": "28975e99ab09262e51ed0ff37d964c0343844aed4e8a7da3f818f74fefdc163e", "contentFormat": "text/markdown", "content": "Executable software · Atlas v1.3.0\n\n# Speculative decoding: proposals and verified output\n\nAccount for target-distributed output tokens while separating draft work, verification and actual elapsed time.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#speculative-decoding-verification)Logical values and distributed ownership; no physical scale or vendor timing\n\nLearning objectiveRecover the target distribution by exact probability accounting, then calculate output-per-round and a conditional throughput without treating proposals or assumptions as measured useful output.\n\n## The mechanism\n\nA draft model can propose several next tokens cheaply, but proposals are not yet accepted output. In strict speculative sampling, the target model evaluates the relevant conditional distributions and verifies the draft sequence in order. A proposal is accepted with a probability determined by the target-to-draft probability ratio. The first rejected position is corrected using the positive residual between the two distributions, and later proposals from that speculative branch are discarded. When all proposals are accepted, one additional target token can be emitted. This procedure preserves the target sampling distribution under its stated assumptions; it does not establish factual truth or force the same sampled text from the same random seed. The benefit can come from evaluating several target positions together and reusing data movement, while draft work, rejected work and additional state still consume resources. The exact three-symbol example below checks the sampling correction before considering a separately assumed timing budget.\n\n1. Use target p and draft q distributions over the same token events and the same valid prefix, including any declared sampling transforms.\n2. Sample a proposal x from q; when q(x)>0, accept it with probability min(1,p(x)/q(x)). A symbol with q(x)=0 cannot be proposed.\n3. After the first rejection, sample a correction from normalized max(p−q,0) at that prefix and discard the later speculative branch.\n4. If all γ proposals are accepted, draw the extra token from the target distribution conditioned on the full accepted prefix.\n5. Count accepted prefix tokens plus the one correction or bonus token as output; count draft proposals and rejected target work separately.\n6. Measure draft, target-verification, synchronization and state-management time on the actual workload before claiming a speedup. Distributional equivalence remains separate from factual or task correctness.\n\n## Science and assumptions\n\n### Acceptance and rejection correction\n\n$$\na(x)=\\min\\!\\left(1,\\frac{p(x)}{q(x)}\\right),\\qquad r(x)=\\frac{[p(x)-q(x)]_+}{\\sum_y[p(y)-q(y)]_+},\\qquad \\alpha=\\sum_x\\min(p(x),q(x))\n$$\n\nAccept a proposed symbol using the target-to-draft ratio capped at one. After rejection, sample from the normalized positive target-minus-draft residual. Acceptance mass sums the minimum of each pair of probabilities.\n\nSymbols and units\n\np(x), q(x)\n\nTarget and draft probabilities at the same prefix\n\nprobability\n\na(x)\n\nConditional acceptance probability for q(x)>0\n\nprobability\n\nr(x)\n\nCorrection probability when residual mass is positive\n\nprobability\n\n\\[z\\]+\n\nPositive part max(z,0)\n\nvalue\n\nα\n\nTotal proposal-acceptance probability at this prefix\n\nprobability\n\nFor a proposed symbol, accept with a(x). After a rejection, sample from residual r. Total acceptance probability is α=Σx min(p(x),q(x)).\n\nAssumptions Finite normalized distributions with q(x)>0 for a proposed x. Use r only when its denominator is positive; if p=q no rejection occurs. Exact probability arithmetic and consistent prefixes are assumed.\n\n### Output count in the iid acceptance scenario\n\n$$\n\\mathbb{E}[K]=\\sum_{j=0}^{\\gamma}\\alpha^j,\\qquad R_{\\mathrm{scenario}}=\\frac{\\mathbb{E}[K]}{T_{\\mathrm{draft}}+T_{\\mathrm{verify}}+T_{\\mathrm{overhead}}}\n$$\n\nWith iid acceptance and no early stop, expected output count is one plus powers of acceptance probability through the proposal count. Divide by the separately assumed total round time for a scenario token rate.\n\nSymbols and units\n\nK\n\nOutput tokens, including the correction or bonus token\n\ntokens/round\n\nγ\n\nMaximum draft proposals in a verification round\n\ncount\n\nα\n\nAssumed identical independent acceptance probability\n\nprobability\n\nTdraft\n\nTotal draft time for the round\n\ns/round\n\nTverify\n\nTarget verification time for the round\n\ns/round\n\nToverhead\n\nOther nonoverlapped round time\n\ns/round\n\nRscenario\n\nOutput rate under the time assumptions\n\ntokens/s\n\nK counts output tokens, including the correction or bonus. With iid acceptance probability α and no early stopping, the probability of accepting at least j proposals is αʲ.\n\nAssumptions The iid assumption is synthetic; actual acceptance varies with prefix. Time terms are separately declared seconds per verification round; no measured speedup, energy reduction or universal latency scaling is implied.\n\n## Engineering example\n\nFor symbols A/B/C, choose target p=\\[0.6,0.3,0.1\\] and draft q=\\[0.2,0.3,0.5\\]. Conditional acceptance is \\[1,1,0.2\\], so accepted probability mass is \\[0.2,0.3,0.1\\] and total α=0.6. The rejected mass is 0.4; normalized positive residual p−q is \\[1,0,0\\], so every correction in this toy case is A. The final output distribution is \\[0.2,0.3,0.1\\]+0.4×\\[1,0,0\\]=\\[0.6,0.3,0.1\\], exactly the target distribution. If γ=2 and acceptance events are independently 0.6 at both positions, output counts 1,2,3 have probabilities 0.4,0.24,0.36. Thus E\\[K\\]=1.96 tokens. Separately assume the two draft steps together take 2 ms, target verification 6 ms and overhead 1 ms: 1.96/0.009≈217.8 output tokens/s. A 6 ms single-token baseline would be 166.7 tokens/s under those assumptions. At 12 ms verification, the speculative scenario instead gives 130.7 tokens/s. Neither timing is measured or taken from the paper; acceptance alone does not establish acceleration.\n\n## Distinctions and tradeoffs\n\nAccepting a draft token validates it against a sampling distribution, not against external facts; higher acceptance does not by itself prove lower latency.\n\nMore proposals can expose useful parallel verification while increasing draft computation, temporary state and wasted work. Prefix-dependent acceptance and workload-specific target cost determine the result.\n\nUseful serving throughput counts the output that survives the verification algorithm and meets the task’s quality requirements. This connects decoding semantics, memory reuse and latency accounting.\n\n## Specifications and interfaces\n\nPreserved quantity\n\nTarget sampling distribution\n\nIdeal strict correction; not a guarantee of the same random sample or true statements.\n\nOutput per verification\n\n1 through γ+1 tokens before stopping rules\n\nAccepted prefix plus a correction or a target bonus.\n\nResource boundary\n\nDraft and verification both cost work\n\nBatch verification latency is an implementation measurement, not implied by token count.\n\n### Interfaces\n\nAligned target and draft distributions Input\n\nPrefix-conditioned token probabilities and proposal sequence\n\nVerified continuation Output\n\nAccepted prefix and corrected or bonus target token\n\n## Check your understanding\n\n### In the stated toy distributions, a rejected proposal must be corrected from which distribution?\n\n1. The original draft \\[0.2,0.3,0.5\\]\n2. The residual \\[1,0,0\\]\n3. Any uniform distribution\n\nAnswer and reasoning\n\nThe residual \\[1,0,0\\]\n\nThe positive residual is \\[0.4,0,0\\], which normalizes to \\[1,0,0\\]. Adding its weighted contribution recovers the target probabilities.\n\n### Under iid α=0.6 with two proposals, what is the expected output count including correction or bonus?\n\n1. 1.2\n2. 1.96\n3. 3\n\nAnswer and reasoning\n\n1.96\n\nThe count expectation is 1+0.6+0.36=1.96.\n\n### Does preserving p establish that sampled statements are true?\n\n1. Yes\n2. No\n3. Only when α=1\n\nAnswer and reasoning\n\nNo\n\nThe target distribution can assign probability to factually wrong continuations.\n\n## Evidence and scope\n\nThe cited ICML paper establishes the strict correction algorithm and the explicitly iid output-count formula. The toy probabilities, output-count distribution and all millisecond values are authored. No private model, serving implementation, benchmark speedup or quality improvement is asserted.\n\nExplore the KV storage consequence of token length and concurrency. Draft/target acceptance and corrected sampling are worked explicitly in this lesson, not executed by the cache-size calculator.\n\n## Sources\n\n- [Leviathan, Kalman and Matias — Fast Inference from Transformers via Speculative Decoding](https://proceedings.mlr.press/v202/leviathan23a.html)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/speculative-decoding-verification-f6066fe7fb36a164.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n- [Output head & decoding](https://semiaifoundry.com/learn/atlas/output-head/)\n- [Autoregressive decode and weight traffic](https://semiaifoundry.com/learn/atlas/decode-bandwidth-trace/)\n- [Key–value cache](https://semiaifoundry.com/learn/atlas/kv-cache/)\n- [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/)\n\n### Related lessons\n\n- [Prefill and prompt processing](https://semiaifoundry.com/learn/atlas/prefill-resource-trace/)\n- [Prefill/decode disaggregation and useful serving rate](https://semiaifoundry.com/learn/atlas/disaggregated-serving/)\n- [Runtime, driver and device dispatch](https://semiaifoundry.com/learn/atlas/runtime-device-dispatch/)\n- [Place the working set, not just the model](https://semiaifoundry.com/learn/atlas/memory-working-set-placement/)" }