{
  "schemaVersion": 1,
  "id": "atlas-lesson-synthetic-data-selection-feedback",
  "title": "Synthetic supervision and selection loops",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Track a generated example through verification and selection without converting acceptance into ground truth.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/synthetic-data-selection-feedback/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "model-lifecycle-data",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Mode: worked example. Explicit arithmetic reference and deterministic faulty filter; positive denominators. This is a worked selection audit, not a synthetic-data training simulator. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/synthetic-data-selection-feedback-c54f0b92559cc4f9.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#synthetic-data-selection-feedback",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "a798e6681183d3e67045e8e1206b8256d21bb79ea861afd2420b87ccdf393d54",
  "contentLength": 6600,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-synthetic-data-selection-feedback.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-synthetic-data-selection-feedback.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Synthetic supervision and selection loops",
      "start": 37,
      "end": 496
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 496,
      "end": 1489
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1489,
      "end": 1517
    },
    {
      "id": "section-4",
      "title": "Synthetic supervision and selection loops · worked relation",
      "start": 1517,
      "end": 2640
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2640,
      "end": 3075
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 3075,
      "end": 3583
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3583,
      "end": 4038
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 4038,
      "end": 4290
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 4290,
      "end": 4319
    },
    {
      "id": "section-10",
      "title": "What does the accepted answer 6 demonstrate?",
      "start": 4319,
      "end": 4916
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4916,
      "end": 5529
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5529,
      "end": 5915
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 5915,
      "end": 5938
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 5938,
      "end": 6126
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 6126,
      "end": 6600
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#synthetic-data-selection-feedback"
    },
    {
      "title": "Tülu 3: Pushing Frontiers in Open Language Model Post-Training",
      "url": "https://arxiv.org/html/2411.15124v1"
    },
    {
      "title": "DeepSeek-R1",
      "url": "https://arxiv.org/html/2501.12948v1"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/synthetic-data-selection-feedback-c54f0b92559cc4f9.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Training data pipeline",
      "url": "https://semiaifoundry.com/learn/atlas/training-data/"
    },
    {
      "title": "Evaluation & the AGI frontier",
      "url": "https://semiaifoundry.com/learn/atlas/evaluation/"
    },
    {
      "title": "Filtering, duplication and contamination",
      "url": "https://semiaifoundry.com/learn/atlas/data-quality-dedup-audit/"
    },
    {
      "title": "Mixtures, epochs and exposure",
      "url": "https://semiaifoundry.com/learn/atlas/data-mixture-sampling/"
    },
    {
      "title": "Packing documents and masking valid targets",
      "url": "https://semiaifoundry.com/learn/atlas/sequence-packing-loss-denominators/"
    },
    {
      "title": "Coverage across language and source slices",
      "url": "https://semiaifoundry.com/learn/atlas/data-coverage-language-slices/"
    }
  ],
  "recordHash": "62943f9c9c84ca0362ceb065bd7faba8351be7b9f9ec8e682faab5e1c99d3df6",
  "contentFormat": "text/markdown",
  "content": "model-lifecycle-data · Atlas v1.3.0\n\n# Synthetic supervision and selection loops\n\nTrack a generated example through verification and selection without converting acceptance into ground truth.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#synthetic-data-selection-feedback)Mathematical, data and behavior space; logical dimensions are not physical distance\n\nLearning objectiveTrack a generated example through verification and selection without converting acceptance into ground truth.\n\n## The mechanism\n\nSynthetic supervision has a lineage: a generator produces a candidate, a verifier or judge assigns a signal, and a selection rule decides whether to train on it. Each stage can introduce a different bias. Repeating a generator’s answer does not create independent evidence, and selecting only accepted outputs can hide the verifier’s blind spots. Generation seeds, prompts, model identities and verifier versions therefore belong to the dataset record. Final evaluation needs independently specified cases and scoring rules, rather than the same filter that selected training material.\n\n1.  Retain rejected candidates as an audit sample alongside accepted examples.\n2.  Evaluate an independent reference on a bounded problem where the answer is actually computable.\n3.  Separate selection rate, precision among accepted examples and coverage of problem types.\n4.  Compare the trained model on a reserved task family; selection statistics are not model-quality measurements.\n\n## Science and assumptions\n\n### Synthetic supervision and selection loops · worked relation\n\n$$\n\\operatorname{precision}_{\\rm selected}=\\frac{N_{\\rm correct\\,and\\,selected}}{N_{\\rm selected}},\\qquad \\operatorname{selection\\ rate}=\\frac{N_{\\rm selected}}{N_{\\rm generated}}\n$$\n\nTrack a generated example through verification and selection without converting acceptance into ground truth.\n\nSymbols and units\n\nN\n\nCounts under the stated candidate or unique-pair boundary\n\nexamples\n\nprecision\\_selected\n\nIndependent correctness among accepted candidates\n\ndimensionless\n\nFour authored candidates answer 2+2: 4, 5, 4 and 6. A faulty verifier accepts even numbers, so it accepts three candidates, including the wrong 6. Selection rate is 3/4; correctness among accepted candidates is 2/3. Deduplicating identical problem/answer pairs leaves {4,6}, whose correctness is 1/2. This denominator change reveals why duplicate correct answers can make a selected corpus look more reliable.\n\nAssumptions Explicit arithmetic reference and deterministic faulty filter; positive denominators. This is a worked selection audit, not a synthetic-data training simulator.\n\n## Engineering example\n\nFour authored candidates answer 2+2: 4, 5, 4 and 6. A faulty verifier accepts even numbers, so it accepts three candidates, including the wrong 6. Selection rate is 3/4; correctness among accepted candidates is 2/3. Deduplicating identical problem/answer pairs leaves {4,6}, whose correctness is 1/2. This denominator change reveals why duplicate correct answers can make a selected corpus look more reliable.\n\n## Distinctions and tradeoffs\n\nIt is a mistake to conclude: Deterministic verification guarantees mathematical truth.\n\nA restrictive verifier can improve accepted precision while reducing diversity. A permissive or biased verifier can amplify generator errors through repeated training.\n\nMake the effective training distribution inspectable rather than treating tokens as a homogeneous commodity. Track a generated example through verification and selection without converting acceptance into ground truth.\n\n## Specifications and interfaces\n\nLearning mode\n\nworked example\n\nThe executable decoder has 423 parameters; worked extensions do not imply an additional simulator.\n\nWorked contract\n\nTrack a generated example through verification and selection without converting acceptance into ground truth.\n\nExplicit arithmetic reference and deterministic faulty filter; positive denominators. This is a worked selection audit, not a synthetic-data training simulator.\n\n### Interfaces\n\nInputs and state Input\n\nRetain rejected candidates as an audit sample alongside accepted examples.\n\nResult and check Output\n\nTrack a generated example through verification and selection without converting acceptance into ground truth.\n\n## Check your understanding\n\n### What does the accepted answer 6 demonstrate?\n\n1.  The filter accepts some incorrect answers, even though its rule is deterministic.\n2.  Deterministic verification guarantees mathematical truth.\n3.  A high selection rate proves generalization.\n\nAnswer and reasoning\n\nThe filter accepts some incorrect answers, even though its rule is deterministic.\n\nThe filter accepts some incorrect answers, even though its rule is deterministic. Explicit arithmetic reference and deterministic faulty filter; positive denominators. This is a worked selection audit, not a synthetic-data training simulator.\n\n## Evidence and scope\n\nMode: worked example. Explicit arithmetic reference and deterministic faulty filter; positive denominators. This is a worked selection audit, not a synthetic-data training simulator. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nModel lifecycle studio: worked example. Explicit arithmetic reference and deterministic faulty filter; positive denominators. This is a worked selection audit, not a synthetic-data training simulator.\n\n## Sources\n\n-   [Tülu 3: Pushing Frontiers in Open Language Model Post-Training](https://arxiv.org/html/2411.15124v1)\n-   [DeepSeek-R1](https://arxiv.org/html/2501.12948v1)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/synthetic-data-selection-feedback-c54f0b92559cc4f9.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Training data pipeline](https://semiaifoundry.com/learn/atlas/training-data/)\n-   [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/)\n\n### Related lessons\n\n-   [Filtering, duplication and contamination](https://semiaifoundry.com/learn/atlas/data-quality-dedup-audit/)\n-   [Mixtures, epochs and exposure](https://semiaifoundry.com/learn/atlas/data-mixture-sampling/)\n-   [Packing documents and masking valid targets](https://semiaifoundry.com/learn/atlas/sequence-packing-loss-denominators/)\n-   [Coverage across language and source slices](https://semiaifoundry.com/learn/atlas/data-coverage-language-slices/)"
}