# Synthetic supervision and selection loops Canonical: https://semiaifoundry.com/learn/atlas/synthetic-data-selection-feedback/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html model-lifecycle-data · Atlas v1.3.0 # Synthetic supervision and selection loops Track a generated example through verification and selection without converting acceptance into ground truth. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#synthetic-data-selection-feedback)Mathematical, data and behavior space; logical dimensions are not physical distance Learning objectiveTrack a generated example through verification and selection without converting acceptance into ground truth. ## The mechanism Synthetic supervision has a lineage: a generator produces a candidate, a verifier or judge assigns a signal, and a selection rule decides whether to train on it. Each stage can introduce a different bias. Repeating a generator’s answer does not create independent evidence, and selecting only accepted outputs can hide the verifier’s blind spots. Generation seeds, prompts, model identities and verifier versions therefore belong to the dataset record. Final evaluation needs independently specified cases and scoring rules, rather than the same filter that selected training material. 1. Retain rejected candidates as an audit sample alongside accepted examples. 2. Evaluate an independent reference on a bounded problem where the answer is actually computable. 3. Separate selection rate, precision among accepted examples and coverage of problem types. 4. Compare the trained model on a reserved task family; selection statistics are not model-quality measurements. ## Science and assumptions ### Synthetic supervision and selection loops · worked relation $$ \operatorname{precision}_{\rm selected}=\frac{N_{\rm correct\,and\,selected}}{N_{\rm selected}},\qquad \operatorname{selection\ rate}=\frac{N_{\rm selected}}{N_{\rm generated}} $$ Track a generated example through verification and selection without converting acceptance into ground truth. Symbols and units N Counts under the stated candidate or unique-pair boundary examples precision\_selected Independent correctness among accepted candidates dimensionless Four authored candidates answer 2+2: 4, 5, 4 and 6. A faulty verifier accepts even numbers, so it accepts three candidates, including the wrong 6. Selection rate is 3/4; correctness among accepted candidates is 2/3. Deduplicating identical problem/answer pairs leaves {4,6}, whose correctness is 1/2. This denominator change reveals why duplicate correct answers can make a selected corpus look more reliable. Assumptions Explicit arithmetic reference and deterministic faulty filter; positive denominators. This is a worked selection audit, not a synthetic-data training simulator. ## Engineering example Four authored candidates answer 2+2: 4, 5, 4 and 6. A faulty verifier accepts even numbers, so it accepts three candidates, including the wrong 6. Selection rate is 3/4; correctness among accepted candidates is 2/3. Deduplicating identical problem/answer pairs leaves {4,6}, whose correctness is 1/2. This denominator change reveals why duplicate correct answers can make a selected corpus look more reliable. ## Distinctions and tradeoffs It is a mistake to conclude: Deterministic verification guarantees mathematical truth. A restrictive verifier can improve accepted precision while reducing diversity. A permissive or biased verifier can amplify generator errors through repeated training. Make the effective training distribution inspectable rather than treating tokens as a homogeneous commodity. Track a generated example through verification and selection without converting acceptance into ground truth. ## Specifications and interfaces Learning mode worked example The executable decoder has 423 parameters; worked extensions do not imply an additional simulator. Worked contract Track a generated example through verification and selection without converting acceptance into ground truth. Explicit arithmetic reference and deterministic faulty filter; positive denominators. This is a worked selection audit, not a synthetic-data training simulator. ### Interfaces Inputs and state Input Retain rejected candidates as an audit sample alongside accepted examples. Result and check Output Track a generated example through verification and selection without converting acceptance into ground truth. ## Check your understanding ### What does the accepted answer 6 demonstrate? 1. The filter accepts some incorrect answers, even though its rule is deterministic. 2. Deterministic verification guarantees mathematical truth. 3. A high selection rate proves generalization. Answer and reasoning The filter accepts some incorrect answers, even though its rule is deterministic. The filter accepts some incorrect answers, even though its rule is deterministic. Explicit arithmetic reference and deterministic faulty filter; positive denominators. This is a worked selection audit, not a synthetic-data training simulator. ## Evidence and scope Mode: worked example. Explicit arithmetic reference and deterministic faulty filter; positive denominators. This is a worked selection audit, not a synthetic-data training simulator. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied. Model lifecycle studio: worked example. Explicit arithmetic reference and deterministic faulty filter; positive denominators. This is a worked selection audit, not a synthetic-data training simulator. ## Sources - [Tülu 3: Pushing Frontiers in Open Language Model Post-Training](https://arxiv.org/html/2411.15124v1) - [DeepSeek-R1](https://arxiv.org/html/2501.12948v1) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/synthetic-data-selection-feedback-c54f0b92559cc4f9.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Foundations - [Training data pipeline](https://semiaifoundry.com/learn/atlas/training-data/) - [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/) ### Related lessons - [Filtering, duplication and contamination](https://semiaifoundry.com/learn/atlas/data-quality-dedup-audit/) - [Mixtures, epochs and exposure](https://semiaifoundry.com/learn/atlas/data-mixture-sampling/) - [Packing documents and masking valid targets](https://semiaifoundry.com/learn/atlas/sequence-packing-loss-denominators/) - [Coverage across language and source slices](https://semiaifoundry.com/learn/atlas/data-coverage-language-slices/)