{
  "schemaVersion": 1,
  "id": "atlas-lesson-training-data",
  "title": "Training data pipeline",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Convert a source collection into useful learning examples.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/training-data/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Learning",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "The fixed-mixture expectation is the relation stated in the lesson. Values are synthetic; deduplication, representativeness, contamination and provenance require independent checks and cannot be inferred from this weighted average.\n\nThe sensor datasets and numeric normalization values are authored. They establish a leakage/artifact mechanism, not field data quality or generalization on real equipment.\n\nMode: executable corpus lineage. Two intentionally conflicting domains share prompt families within the same split, never across splits. No external dataset license or privacy review is implied. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nThese are computed relationships in twelve authored documents across seven known families. No language-model quality, memorization rate or representative real-corpus coverage is measured.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/training-data-0781ba332be25aa0.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#training-data",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "c202dd53365327e4f7f4bc50725630bddcfcc4dd699f03504d0bfdafa5ac9a13",
  "contentLength": 12997,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-training-data.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-training-data.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Training data pipeline",
      "start": 25,
      "end": 354
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 354,
      "end": 3156
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 3156,
      "end": 3184
    },
    {
      "id": "section-4",
      "title": "The mixture defines the expected learning signal",
      "start": 3184,
      "end": 4476
    },
    {
      "id": "section-5",
      "title": "Dataset lineage and reproducible splits · worked relation",
      "start": 4476,
      "end": 5373
    },
    {
      "id": "section-6",
      "title": "Engineering example",
      "start": 5373,
      "end": 6714
    },
    {
      "id": "section-7",
      "title": "Distinctions and tradeoffs",
      "start": 6714,
      "end": 7024
    },
    {
      "id": "section-8",
      "title": "Specifications and interfaces",
      "start": 7024,
      "end": 7219
    },
    {
      "id": "section-9",
      "title": "Interfaces",
      "start": 7219,
      "end": 7325
    },
    {
      "id": "section-10",
      "title": "Check your understanding",
      "start": 7325,
      "end": 7354
    },
    {
      "id": "section-11",
      "title": "Does a larger raw corpus guarantee a better model?",
      "start": 7354,
      "end": 7640
    },
    {
      "id": "section-12",
      "title": "Should held-out evaluation labels or future observations fit a preprocessing statistic used to claim an earlier forecast?",
      "start": 7640,
      "end": 8078
    },
    {
      "id": "section-13",
      "title": "Why are there only 24 distinct prompt families across the 48 records?",
      "start": 8078,
      "end": 8654
    },
    {
      "id": "section-14",
      "title": "A mirrored document appears in training and evaluation after individual-document splitting. What does this establish?",
      "start": 8654,
      "end": 9275
    },
    {
      "id": "section-15",
      "title": "Historical context",
      "start": 9275,
      "end": 9298
    },
    {
      "id": "section-16",
      "title": "2021 · Deduplication is measured as a training intervention",
      "start": 9298,
      "end": 9686
    },
    {
      "id": "section-17",
      "title": "Evidence and scope",
      "start": 9686,
      "end": 10857
    },
    {
      "id": "section-18",
      "title": "Sources",
      "start": 10857,
      "end": 11580
    },
    {
      "id": "section-19",
      "title": "Continue exploring",
      "start": 11580,
      "end": 11603
    },
    {
      "id": "section-20",
      "title": "Foundations",
      "start": 11603,
      "end": 11687
    },
    {
      "id": "section-21",
      "title": "Related lessons",
      "start": 11687,
      "end": 12997
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#training-data"
    },
    {
      "title": "Lee et al. — Deduplicating Training Data Makes Language Models Better",
      "url": "https://arxiv.org/abs/2107.06499"
    },
    {
      "title": "Hoffmann et al. — Compute-Optimal Training",
      "url": "https://arxiv.org/abs/2203.15556"
    },
    {
      "title": "GPT-4 Technical Report",
      "url": "https://arxiv.org/abs/2303.08774"
    },
    {
      "title": "scikit-learn · Common pitfalls and data leakage",
      "url": "https://scikit-learn.org/stable/common_pitfalls.html#data-leakage"
    },
    {
      "title": "The FineWeb Datasets",
      "url": "https://arxiv.org/html/2406.17557v1"
    },
    {
      "title": "Deduplicating Training Data Makes Language Models Better",
      "url": "https://arxiv.org/html/2107.06499v1"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/training-data-0781ba332be25aa0.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Tokenizer",
      "url": "https://semiaifoundry.com/learn/atlas/tokenizer/"
    },
    {
      "title": "Loss, gradients & optimizer",
      "url": "https://semiaifoundry.com/learn/atlas/loss-optimizer/"
    },
    {
      "title": "Dataset and checkpoint storage hierarchy",
      "url": "https://semiaifoundry.com/learn/atlas/storage/"
    },
    {
      "title": "Scaling laws & resource allocation",
      "url": "https://semiaifoundry.com/learn/atlas/scaling-laws/"
    },
    {
      "title": "Model artifacts, rollout and drift",
      "url": "https://semiaifoundry.com/learn/atlas/model-artifact-rollout-drift/"
    },
    {
      "title": "Forecasts are tested forward in time",
      "url": "https://semiaifoundry.com/learn/atlas/temporal-forecast-validation/"
    },
    {
      "title": "Filtering, duplication and contamination",
      "url": "https://semiaifoundry.com/learn/atlas/data-quality-dedup-audit/"
    },
    {
      "title": "Mixtures, epochs and exposure",
      "url": "https://semiaifoundry.com/learn/atlas/data-mixture-sampling/"
    },
    {
      "title": "Packing documents and masking valid targets",
      "url": "https://semiaifoundry.com/learn/atlas/sequence-packing-loss-denominators/"
    },
    {
      "title": "Synthetic supervision and selection loops",
      "url": "https://semiaifoundry.com/learn/atlas/synthetic-data-selection-feedback/"
    },
    {
      "title": "Coverage across language and source slices",
      "url": "https://semiaifoundry.com/learn/atlas/data-coverage-language-slices/"
    },
    {
      "title": "Human judgments, disagreement and dataset lineage",
      "url": "https://semiaifoundry.com/learn/atlas/dataset-human-judgment-lineage/"
    },
    {
      "title": "Evaluation & the AGI frontier",
      "url": "https://semiaifoundry.com/learn/atlas/evaluation/"
    }
  ],
  "recordHash": "5987ab6da17a12bafdfcb4d8db8937c8369fc8b1eca05ada4ac4b8a359e445a7",
  "contentFormat": "text/markdown",
  "content": "Learning · Atlas v1.3.0\n\n# Training data pipeline\n\nConvert a source collection into useful learning examples.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#training-data)Logical structure · mapped onto physical hardware\n\nLearning objectiveCalculate a mixture-weighted loss and explain why changing the data mix changes the learning objective.\n\n## The mechanism\n\nA training pipeline may collect, filter, deduplicate, format, tokenize, and batch data. The mixture controls which patterns the optimizer repeatedly encounters. Quality is not a single scalar: correctness, coverage, provenance, licensing, contamination, and representativeness all matter. Training, validation, and benchmark splits need careful separation and overlap checks. More bytes alone do not guarantee better learning; repeated low-value or contaminated examples can waste compute and distort evaluation.\n\nTraining examples also have an availability and transformation history. A label may arrive after an input was first observed, and preprocessing fitted on future or held-out cases can leak information across the evaluation boundary. A deployed artifact should retain which split supplied each learned statistic so later predictions can be reproduced without silently refitting on new data.\n\nAdvanced lifecycle practice retains source identity, transformation history, tokenizer version and document-family split assignment together. A train/development/final separation is meaningful only if near-identical families do not cross it. The executable symbol corpus records these fields, while its source is original synthetic material rather than a crawl or human conversation. Development answers support iteration; the first final report records the evaluated checkpoint and cannot be silently rewritten after another update.\n\nThe corpus lineage experience now executes a small construction chain: original document families are assigned to splits before normalization; exact training copies can be excluded with retained source pointers; a tokenizer fits only retained training fields; deterministic document sampling and packing produce inspectable arrays.\n\n1.  Prepare and document the source mixture.\n2.  Filter, deduplicate, tokenize, and form batches.\n3.  Track validation and contamination separately from optimization.\n4.  Feature construction and normalizer fitting use only the allowed training information.\n5.  Evaluation and later monitoring retain their own sample and label-availability boundaries rather than being folded back into the training statistic.\n6.  Preserve raw-source and transformed-example identities.\n7.  Assign whole document families to splits before tokenization-dependent sampling.\n8.  Record final exposure as part of the experimental history, including after importing an older checkpoint.\n9.  Compare exact transformed prompt/response pairs, three-word-shingle similarity and known-family relationships without treating any single diagnostic as semantic truth.\n10.  Record split seed, transformation, excluded IDs, tokenizer fitting IDs, merge order, sampler cursor and packed indices so the same declared batch can be reconstructed.\n\n## Science and assumptions\n\n### The mixture defines the expected learning signal\n\n$$\n\\mathcal L_{\\mathrm{mix}}(\\theta)=\\sum_j w_j\\,\\mathbb E_{x\\sim D_j}[\\ell(\\theta;x)],\\qquad w_j\\ge0,\\quad\\sum_jw_j=1\n$$\n\nMixture loss is the weighted sum of expected per-example loss for each source. Sampling weights are nonnegative and sum to one.\n\nSymbols and units\n\nℒ\\_mix\n\nExpected loss under the selected source mixture\n\nloss units\n\nθ\n\nModel parameter vector\n\nmodel coordinates\n\nwⱼ\n\nSampling weight of source j\n\nprobability fraction\n\nDⱼ\n\nData distribution for source j\n\nprobability distribution\n\nx\n\nExample sampled from Dⱼ\n\nexample\n\nℓ(θ,x)\n\nPer-example loss\n\nsame loss units as ℒ\\_mix\n\nj\n\nSource index\n\ninteger index\n\nEach data source $D_{j}$ contributes according to its sampling weight $w_{j}$. Changing those weights changes which errors the optimizer is asked to reduce most often. Repeating near-duplicates can therefore increase their influence without adding comparable diversity. The same nominal token budget can represent very different learning opportunities.\n\nAssumptions This expectation assumes a fixed mixture and loss definition. Curriculum schedules, example weighting, repeated epochs, and adaptive sampling modify the effective weights. Source quality and legal provenance cannot be inferred from the equation.\n\n### Dataset lineage and reproducible splits · worked relation\n\n$$\nN_{records}=\\sum_s N_s,\\qquad N_{families}=|\\{\\operatorname{family}(x):x\\in D\\}|\n$$\n\nAudit data lineage at the family and split boundaries.\n\nSymbols and units\n\nN\\_records\n\nStored example rows\n\nexamples\n\nN\\_s\n\nRows in split s\n\nexamples\n\nN\\_families\n\nDistinct prompt-family identities\n\nfamilies\n\nD\n\nDeclared corpus\n\ndataset\n\nThe executable corpus has 12 training, 8 development and 4 final examples per domain. Across copy and swap this is 24/16/8 example records, but only 12/8/4 distinct prompt families because each domain assigns its own answer to the same family. Treating all 48 records as independent prompt families would double-count the evidence.\n\nAssumptions Two intentionally conflicting domains share prompt families within the same split, never across splits. No external dataset license or privacy review is implied.\n\n## Engineering example\n\nSynthetic source losses are 2 and 4 under the same loss definition. Sampling weights 0.75 and 0.25 give mixture loss 2.5; changing to 0.5 and 0.5 gives 3. The calculation changes source influence even when total example count is unchanged.\n\nIf a training-only normalizer has mean 60 and standard deviation 10, input 70 maps to 1. Recomputing a mean of 70 using a later population maps the same input to 0. Comparing model predictions before and after that change as if only the weights mattered hides a preprocessing change. The new artifact lesson gives a logistic example where the resulting threshold decision reverses.\n\nThe executable corpus has 12 training, 8 development and 4 final examples per domain. Across copy and swap this is 24/16/8 example records, but only 12/8/4 distinct prompt families because each domain assigns its own answer to the same family. Treating all 48 records as independent prompt families would double-count the evidence.\n\nAt split seed17 in the original maintenance fixture, splitting individual documents places fan7-original in training and its exact mirror in evaluation. Whole-family assignment removes that flagged cross-split overlap. At default seed7, exact-copy exclusion removes three training records while preserving their original texts and retained-document pointers.\n\n## Distinctions and tradeoffs\n\nTraining tokens are not the same thing as a count of unique source documents.\n\nCleaning can remove useful rare content as well as noise. Data composition must match intended capabilities and evaluation goals.\n\nData supplies the examples that shape the model’s learned behavior.\n\n## Specifications and interfaces\n\nBudget\n\nTokens actually trained on\n\nDifferent from raw corpus size.\n\nData quality\n\nMultidimensional\n\nCoverage, correctness, provenance, and duplication matter.\n\n### Interfaces\n\nStorage Input\n\nPrepared examples\n\nTraining loop Output\n\nBatches of token IDs and targets\n\n## Check your understanding\n\n### Does a larger raw corpus guarantee a better model?\n\n1.  No; quality and training use matter\n2.  Yes, without exception\n3.  Only if it contains images\n\nAnswer and reasoning\n\nNo; quality and training use matter\n\nSize alone misses duplication, usefulness, coverage, and optimization.\n\n### Should held-out evaluation labels or future observations fit a preprocessing statistic used to claim an earlier forecast?\n\n1.  Yes, more data always improves the experiment\n2.  No, that violates the stated availability boundary\n3.  Only if the resulting accuracy increases\n\nAnswer and reasoning\n\nNo, that violates the stated availability boundary\n\nThe comparison must use information that was allowed at training or prediction time.\n\n### Why are there only 24 distinct prompt families across the 48 records?\n\n1.  Each prompt family is represented once in copy and once in swap.\n2.  Half the records are secretly final-test examples.\n3.  Every source record is necessarily an independent task.\n\nAnswer and reasoning\n\nEach prompt family is represented once in copy and once in swap.\n\nEach prompt family is represented once in copy and once in swap. Two intentionally conflicting domains share prompt families within the same split, never across splits. No external dataset license or privacy review is implied.\n\n### A mirrored document appears in training and evaluation after individual-document splitting. What does this establish?\n\n1.  The model must have memorized it\n2.  The evaluation has observable overlap that must be disclosed; model impact needs separate measurement\n3.  The mirror becomes independent because its ID differs\n\nAnswer and reasoning\n\nThe evaluation has observable overlap that must be disclosed; model impact needs separate measurement\n\nAn ID difference does not erase shared content or lineage. The data relationship is measurable before training; a causal model-quality claim requires another experiment.\n\n## Historical context\n\n### 2021 · Deduplication is measured as a training intervention\n\nLee and colleagues showed that removing repeated training material could reduce memorized output and train–test overlap while preserving or improving measured accuracy. Their preprint preceded ACL 2022 publication.\n\n[Lee et al. — Deduplicating Training Data Makes Language Models Better](https://arxiv.org/abs/2107.06499)\n\n## Evidence and scope\n\nThe fixed-mixture expectation is the relation stated in the lesson. Values are synthetic; deduplication, representativeness, contamination and provenance require independent checks and cannot be inferred from this weighted average.\n\nThe sensor datasets and numeric normalization values are authored. They establish a leakage/artifact mechanism, not field data quality or generalization on real equipment.\n\nMode: executable corpus lineage. Two intentionally conflicting domains share prompt families within the same split, never across splits. No external dataset license or privacy review is implied. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nThese are computed relationships in twelve authored documents across seven known families. No language-model quality, memorization rate or representative real-corpus coverage is measured.\n\nUse the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism.\n\n## Sources\n\n-   [Hoffmann et al. — Compute-Optimal Training](https://arxiv.org/abs/2203.15556)\n-   [GPT-4 Technical Report](https://arxiv.org/abs/2303.08774)\n-   [Lee et al. · duplicates and train-test overlap](https://arxiv.org/abs/2107.06499)\n-   [scikit-learn · Common pitfalls and data leakage](https://scikit-learn.org/stable/common_pitfalls.html#data-leakage)\n-   [The FineWeb Datasets](https://arxiv.org/html/2406.17557v1)\n-   [Deduplicating Training Data Makes Language Models Better](https://arxiv.org/html/2107.06499v1)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/training-data-0781ba332be25aa0.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Tokenizer](https://semiaifoundry.com/learn/atlas/tokenizer/)\n\n### Related lessons\n\n-   [Loss, gradients & optimizer](https://semiaifoundry.com/learn/atlas/loss-optimizer/)\n-   [Dataset and checkpoint storage hierarchy](https://semiaifoundry.com/learn/atlas/storage/)\n-   [Scaling laws & resource allocation](https://semiaifoundry.com/learn/atlas/scaling-laws/)\n-   [Model artifacts, rollout and drift](https://semiaifoundry.com/learn/atlas/model-artifact-rollout-drift/)\n-   [Forecasts are tested forward in time](https://semiaifoundry.com/learn/atlas/temporal-forecast-validation/)\n-   [Filtering, duplication and contamination](https://semiaifoundry.com/learn/atlas/data-quality-dedup-audit/)\n-   [Mixtures, epochs and exposure](https://semiaifoundry.com/learn/atlas/data-mixture-sampling/)\n-   [Packing documents and masking valid targets](https://semiaifoundry.com/learn/atlas/sequence-packing-loss-denominators/)\n-   [Synthetic supervision and selection loops](https://semiaifoundry.com/learn/atlas/synthetic-data-selection-feedback/)\n-   [Coverage across language and source slices](https://semiaifoundry.com/learn/atlas/data-coverage-language-slices/)\n-   [Human judgments, disagreement and dataset lineage](https://semiaifoundry.com/learn/atlas/dataset-human-judgment-lineage/)\n-   [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/)"
}