{"id":"test-time-search-verification","name":"Sampling more candidates versus selecting correct ones","shortName":"Sampling more candidates versus selecting correct ones","role":"Distinguish finding a correct candidate from selecting one under a finite verification budget.","description":"Inference can spend additional computation on multiple candidates or search without changing the model’s weights. A verifier then selects or rejects candidates under a task-specific rule. Pass-at-k asks whether a correct candidate exists in a sampled set; deployed selected-answer accuracy asks whether the system actually identifies one. A faulty verifier can discard the correct candidate or choose an attractive wrong one. Candidate dependence also matters: repeated near-identical samples do not provide independent chances. This differs from speculative decoding, whose acceptance correction preserves a sampling distribution rather than verifying answer truth.","mechanism":["Fix generation and checking budgets and retain all candidates, including failures.","Score candidate correctness independently of the selector when a reference exists.","Report both candidate-set success and selected-answer success.","Count model calls, generated tokens and verification calls separately; do not convert them into measured latency without a timing experiment."],"learningObjective":"Distinguish finding a correct candidate from selecting one under a finite verification budget.","engineeringExample":"If independent candidates each succeed with probability 0.25, the probability at least one of four succeeds is 1−0.75^4=0.683594. This is an independence calculation, not an observed model result. In a concrete four-candidate set [wrong, correct, wrong, wrong], pass-at-4 is one, but a selector choosing the first candidate has zero final correctness. A perfect existence score therefore can coexist with a failed answer.","evidenceNotes":"Mode: worked example. Independent identically distributed success events for the probability calculation. Correlated samples and imperfect selectors invalidate interpreting this as final accuracy. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","specs":[{"label":"Learning mode","value":"worked example","context":"The executable decoder has 423 parameters; worked extensions do not imply an additional simulator."},{"label":"Worked contract","value":"Distinguish finding a correct candidate from selecting one under a finite verification budget.","context":"Independent identically distributed success events for the probability calculation. Correlated samples and imperfect selectors invalidate interpreting this as final accuracy."}],"interfaces":[{"name":"Inputs and state","direction":"in","signal":"Fix generation and checking budgets and retain all candidates, including failures."},{"name":"Result and check","direction":"out","signal":"Distinguish finding a correct candidate from selecting one under a finite verification budget."}],"tradeoff":"Extra inference can improve the chance of finding useful candidates, but consumes computation and increases reliance on the verifier. It is not post-training unless parameters are updated.","whyAI":"Measure quality, uncertainty and failure under a stated system/task contract. Distinguish finding a correct candidate from selecting one under a finite verification budget.","misconception":"It is a mistake to conclude: No; pass-at-4 proves the selector is correct.","check":{"question":"Can pass-at-4 be successful while the selected answer is wrong?","options":["No; pass-at-4 proves the selector is correct.","Only if model weights were updated during sampling.","Yes; the correct candidate may exist but not be selected."],"answer":2,"explanation":"Yes; the correct candidate may exist but not be selected. Independent identically distributed success events for the probability calculation. Correlated samples and imperfect selectors invalidate interpreting this as final accuracy."},"prerequisites":["output-head","evaluation","speculative-decoding-verification"],"related":["output-head","evaluation","speculative-decoding-verification","probability-calibration-abstention","judge-rubric-agreement","retrieval-supported-generation","model-system-regression-contract"],"sources":[{"title":"DeepSeek-R1","url":"https://arxiv.org/html/2501.12948v1"},{"title":"Tülu 3: Pushing Frontiers in Open Language Model Post-Training","url":"https://arxiv.org/html/2411.15124v1"}],"level":"model","scale":"Mathematical, data and behavior space; logical dimensions are not physical distance","kind":"model","branch":"model-lifecycle-evaluation","subbranch":"Sampling more candidates versus selecting correct ones","workstream":"W11","visualFamily":"mathematical","labId":"library","labScope":"Model lifecycle studio: worked example. Independent identically distributed success events for the probability calculation. Correlated samples and imperfect selectors invalidate interpreting this as final accuracy.","science":[{"title":"Sampling more candidates versus selecting correct ones · worked relation","equation":"P(\\text{at least one success})=1-(1-p)^k","explanation":"If independent candidates each succeed with probability 0.25, the probability at least one of four succeeds is 1−0.75^4=0.683594. This is an independence calculation, not an observed model result. In a concrete four-candidate set [wrong, correct, wrong, wrong], pass-at-4 is one, but a selector choosing the first candidate has zero final correctness. A perfect existence score therefore can coexist with a failed answer.","assumptions":"Independent identically distributed success events for the probability calculation. Correlated samples and imperfect selectors invalidate interpreting this as final accuracy."}],"history":[]}