Sampling more candidates versus selecting correct ones

The mechanism

Inference can spend additional computation on multiple candidates or search without changing the model’s weights. A verifier then selects or rejects candidates under a task-specific rule. Pass-at-k asks whether a correct candidate exists in a sampled set; deployed selected-answer accuracy asks whether the system actually identifies one. A faulty verifier can discard the correct candidate or choose an attractive wrong one. Candidate dependence also matters: repeated near-identical samples do not provide independent chances. This differs from speculative decoding, whose acceptance correction preserves a sampling distribution rather than verifying answer truth.

Fix generation and checking budgets and retain all candidates, including failures.
Score candidate correctness independently of the selector when a reference exists.
Report both candidate-set success and selected-answer success.
Count model calls, generated tokens and verification calls separately; do not convert them into measured latency without a timing experiment.

Science and assumptions

Sampling more candidates versus selecting correct ones · worked relation

P(\text{at least one success})=1-(1-p)^k

Distinguish finding a correct candidate from selecting one under a finite verification budget.

Symbols and units

p: Single-candidate success probability under the assumed distribution
dimensionless
k: Independent candidates
candidates
P: Candidate-set success probability
dimensionless

If independent candidates each succeed with probability 0.25, the probability at least one of four succeeds is 1−0.75^4=0.683594. This is an independence calculation, not an observed model result. In a concrete four-candidate set [wrong, correct, wrong, wrong], pass-at-4 is one, but a selector choosing the first candidate has zero final correctness. A perfect existence score therefore can coexist with a failed answer.

Assumptions Independent identically distributed success events for the probability calculation. Correlated samples and imperfect selectors invalidate interpreting this as final accuracy.

Engineering example

Distinctions and tradeoffs

It is a mistake to conclude: No; pass-at-4 proves the selector is correct.

Extra inference can improve the chance of finding useful candidates, but consumes computation and increases reliance on the verifier. It is not post-training unless parameters are updated.

Measure quality, uncertainty and failure under a stated system/task contract. Distinguish finding a correct candidate from selecting one under a finite verification budget.

Specifications and interfaces

Learning mode: worked example
The executable decoder has 423 parameters; worked extensions do not imply an additional simulator.
Worked contract: Distinguish finding a correct candidate from selecting one under a finite verification budget.
Independent identically distributed success events for the probability calculation. Correlated samples and imperfect selectors invalidate interpreting this as final accuracy.

Interfaces

Inputs and state Input: Fix generation and checking budgets and retain all candidates, including failures.
Result and check Output: Distinguish finding a correct candidate from selecting one under a finite verification budget.

Check your understanding

Can pass-at-4 be successful while the selected answer is wrong?

No; pass-at-4 proves the selector is correct.
Only if model weights were updated during sampling.
Yes; the correct candidate may exist but not be selected.

Answer and reasoning

Yes; the correct candidate may exist but not be selected.

Yes; the correct candidate may exist but not be selected. Independent identically distributed success events for the probability calculation. Correlated samples and imperfect selectors invalidate interpreting this as final accuracy.

Evidence and scope

Mode: worked example. Independent identically distributed success events for the probability calculation. Correlated samples and imperfect selectors invalidate interpreting this as final accuracy. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.

Model lifecycle studio: worked example. Independent identically distributed success events for the probability calculation. Correlated samples and imperfect selectors invalidate interpreting this as final accuracy.