{
  "schemaVersion": 1,
  "id": "atlas-lesson-test-time-search-verification",
  "title": "Sampling more candidates versus selecting correct ones",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Distinguish finding a correct candidate from selecting one under a finite verification budget.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/test-time-search-verification/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "model-lifecycle-evaluation",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Mode: worked example. Independent identically distributed success events for the probability calculation. Correlated samples and imperfect selectors invalidate interpreting this as final accuracy. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/test-time-search-verification-44ee0a218a7c83b3.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#test-time-search-verification",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "2be8b62b4e5510c792e1a759ffb29ddc3e6626834bb67c0e50892fd3cb92f769",
  "contentLength": 6678,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-test-time-search-verification.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-test-time-search-verification.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Sampling more candidates versus selecting correct ones",
      "start": 43,
      "end": 481
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 481,
      "end": 1555
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1555,
      "end": 1583
    },
    {
      "id": "section-4",
      "title": "Sampling more candidates versus selecting correct ones · worked relation",
      "start": 1583,
      "end": 2616
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2616,
      "end": 3065
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 3065,
      "end": 3536
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3536,
      "end": 3990
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3990,
      "end": 4235
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 4235,
      "end": 4264
    },
    {
      "id": "section-10",
      "title": "Can pass-at-4 be successful while the selected answer is wrong?",
      "start": 4264,
      "end": 4817
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4817,
      "end": 5458
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5458,
      "end": 5840
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 5840,
      "end": 5863
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 5863,
      "end": 6181
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 6181,
      "end": 6678
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#test-time-search-verification"
    },
    {
      "title": "DeepSeek-R1",
      "url": "https://arxiv.org/html/2501.12948v1"
    },
    {
      "title": "Tülu 3: Pushing Frontiers in Open Language Model Post-Training",
      "url": "https://arxiv.org/html/2411.15124v1"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/test-time-search-verification-44ee0a218a7c83b3.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Output head & decoding",
      "url": "https://semiaifoundry.com/learn/atlas/output-head/"
    },
    {
      "title": "Evaluation & the AGI frontier",
      "url": "https://semiaifoundry.com/learn/atlas/evaluation/"
    },
    {
      "title": "Speculative decoding: proposals and verified output",
      "url": "https://semiaifoundry.com/learn/atlas/speculative-decoding-verification/"
    },
    {
      "title": "Confidence, errors and refusal to guess",
      "url": "https://semiaifoundry.com/learn/atlas/probability-calibration-abstention/"
    },
    {
      "title": "Human and model judgments under a rubric",
      "url": "https://semiaifoundry.com/learn/atlas/judge-rubric-agreement/"
    },
    {
      "title": "From lexical retrieval to model answers with evidence",
      "url": "https://semiaifoundry.com/learn/atlas/retrieval-supported-generation/"
    },
    {
      "title": "Version the entire evaluated system",
      "url": "https://semiaifoundry.com/learn/atlas/model-system-regression-contract/"
    }
  ],
  "recordHash": "4bbb6b0acf6b1efd99cc97fcda4361e2bd6c9b0aed06b57ae38035d38eec26fd",
  "contentFormat": "text/markdown",
  "content": "model-lifecycle-evaluation · Atlas v1.3.0\n\n# Sampling more candidates versus selecting correct ones\n\nDistinguish finding a correct candidate from selecting one under a finite verification budget.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#test-time-search-verification)Mathematical, data and behavior space; logical dimensions are not physical distance\n\nLearning objectiveDistinguish finding a correct candidate from selecting one under a finite verification budget.\n\n## The mechanism\n\nInference can spend additional computation on multiple candidates or search without changing the model’s weights. A verifier then selects or rejects candidates under a task-specific rule. Pass-at-k asks whether a correct candidate exists in a sampled set; deployed selected-answer accuracy asks whether the system actually identifies one. A faulty verifier can discard the correct candidate or choose an attractive wrong one. Candidate dependence also matters: repeated near-identical samples do not provide independent chances. This differs from speculative decoding, whose acceptance correction preserves a sampling distribution rather than verifying answer truth.\n\n1.  Fix generation and checking budgets and retain all candidates, including failures.\n2.  Score candidate correctness independently of the selector when a reference exists.\n3.  Report both candidate-set success and selected-answer success.\n4.  Count model calls, generated tokens and verification calls separately; do not convert them into measured latency without a timing experiment.\n\n## Science and assumptions\n\n### Sampling more candidates versus selecting correct ones · worked relation\n\n$$\nP(\\text{at least one success})=1-(1-p)^k\n$$\n\nDistinguish finding a correct candidate from selecting one under a finite verification budget.\n\nSymbols and units\n\np\n\nSingle-candidate success probability under the assumed distribution\n\ndimensionless\n\nk\n\nIndependent candidates\n\ncandidates\n\nP\n\nCandidate-set success probability\n\ndimensionless\n\nIf independent candidates each succeed with probability 0.25, the probability at least one of four succeeds is 1−0.75^4=0.683594. This is an independence calculation, not an observed model result. In a concrete four-candidate set \\[wrong, correct, wrong, wrong\\], pass-at-4 is one, but a selector choosing the first candidate has zero final correctness. A perfect existence score therefore can coexist with a failed answer.\n\nAssumptions Independent identically distributed success events for the probability calculation. Correlated samples and imperfect selectors invalidate interpreting this as final accuracy.\n\n## Engineering example\n\nIf independent candidates each succeed with probability 0.25, the probability at least one of four succeeds is 1−0.75^4=0.683594. This is an independence calculation, not an observed model result. In a concrete four-candidate set \\[wrong, correct, wrong, wrong\\], pass-at-4 is one, but a selector choosing the first candidate has zero final correctness. A perfect existence score therefore can coexist with a failed answer.\n\n## Distinctions and tradeoffs\n\nIt is a mistake to conclude: No; pass-at-4 proves the selector is correct.\n\nExtra inference can improve the chance of finding useful candidates, but consumes computation and increases reliance on the verifier. It is not post-training unless parameters are updated.\n\nMeasure quality, uncertainty and failure under a stated system/task contract. Distinguish finding a correct candidate from selecting one under a finite verification budget.\n\n## Specifications and interfaces\n\nLearning mode\n\nworked example\n\nThe executable decoder has 423 parameters; worked extensions do not imply an additional simulator.\n\nWorked contract\n\nDistinguish finding a correct candidate from selecting one under a finite verification budget.\n\nIndependent identically distributed success events for the probability calculation. Correlated samples and imperfect selectors invalidate interpreting this as final accuracy.\n\n### Interfaces\n\nInputs and state Input\n\nFix generation and checking budgets and retain all candidates, including failures.\n\nResult and check Output\n\nDistinguish finding a correct candidate from selecting one under a finite verification budget.\n\n## Check your understanding\n\n### Can pass-at-4 be successful while the selected answer is wrong?\n\n1.  No; pass-at-4 proves the selector is correct.\n2.  Only if model weights were updated during sampling.\n3.  Yes; the correct candidate may exist but not be selected.\n\nAnswer and reasoning\n\nYes; the correct candidate may exist but not be selected.\n\nYes; the correct candidate may exist but not be selected. Independent identically distributed success events for the probability calculation. Correlated samples and imperfect selectors invalidate interpreting this as final accuracy.\n\n## Evidence and scope\n\nMode: worked example. Independent identically distributed success events for the probability calculation. Correlated samples and imperfect selectors invalidate interpreting this as final accuracy. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nModel lifecycle studio: worked example. Independent identically distributed success events for the probability calculation. Correlated samples and imperfect selectors invalidate interpreting this as final accuracy.\n\n## Sources\n\n-   [DeepSeek-R1](https://arxiv.org/html/2501.12948v1)\n-   [Tülu 3: Pushing Frontiers in Open Language Model Post-Training](https://arxiv.org/html/2411.15124v1)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/test-time-search-verification-44ee0a218a7c83b3.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Output head & decoding](https://semiaifoundry.com/learn/atlas/output-head/)\n-   [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/)\n-   [Speculative decoding: proposals and verified output](https://semiaifoundry.com/learn/atlas/speculative-decoding-verification/)\n\n### Related lessons\n\n-   [Confidence, errors and refusal to guess](https://semiaifoundry.com/learn/atlas/probability-calibration-abstention/)\n-   [Human and model judgments under a rubric](https://semiaifoundry.com/learn/atlas/judge-rubric-agreement/)\n-   [From lexical retrieval to model answers with evidence](https://semiaifoundry.com/learn/atlas/retrieval-supported-generation/)\n-   [Version the entire evaluated system](https://semiaifoundry.com/learn/atlas/model-system-regression-contract/)"
}