# Training data pipeline Canonical: https://semiaifoundry.com/learn/atlas/training-data/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html Learning · Atlas v1.3.0 # Training data pipeline Convert a source collection into useful learning examples. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#training-data)Logical structure · mapped onto physical hardware Learning objectiveCalculate a mixture-weighted loss and explain why changing the data mix changes the learning objective. ## The mechanism A training pipeline may collect, filter, deduplicate, format, tokenize, and batch data. The mixture controls which patterns the optimizer repeatedly encounters. Quality is not a single scalar: correctness, coverage, provenance, licensing, contamination, and representativeness all matter. Training, validation, and benchmark splits need careful separation and overlap checks. More bytes alone do not guarantee better learning; repeated low-value or contaminated examples can waste compute and distort evaluation. Training examples also have an availability and transformation history. A label may arrive after an input was first observed, and preprocessing fitted on future or held-out cases can leak information across the evaluation boundary. A deployed artifact should retain which split supplied each learned statistic so later predictions can be reproduced without silently refitting on new data. Advanced lifecycle practice retains source identity, transformation history, tokenizer version and document-family split assignment together. A train/development/final separation is meaningful only if near-identical families do not cross it. The executable symbol corpus records these fields, while its source is original synthetic material rather than a crawl or human conversation. Development answers support iteration; the first final report records the evaluated checkpoint and cannot be silently rewritten after another update. The corpus lineage experience now executes a small construction chain: original document families are assigned to splits before normalization; exact training copies can be excluded with retained source pointers; a tokenizer fits only retained training fields; deterministic document sampling and packing produce inspectable arrays. 1. Prepare and document the source mixture. 2. Filter, deduplicate, tokenize, and form batches. 3. Track validation and contamination separately from optimization. 4. Feature construction and normalizer fitting use only the allowed training information. 5. Evaluation and later monitoring retain their own sample and label-availability boundaries rather than being folded back into the training statistic. 6. Preserve raw-source and transformed-example identities. 7. Assign whole document families to splits before tokenization-dependent sampling. 8. Record final exposure as part of the experimental history, including after importing an older checkpoint. 9. Compare exact transformed prompt/response pairs, three-word-shingle similarity and known-family relationships without treating any single diagnostic as semantic truth. 10. Record split seed, transformation, excluded IDs, tokenizer fitting IDs, merge order, sampler cursor and packed indices so the same declared batch can be reconstructed. ## Science and assumptions ### The mixture defines the expected learning signal $$ \mathcal L_{\mathrm{mix}}(\theta)=\sum_j w_j\,\mathbb E_{x\sim D_j}[\ell(\theta;x)],\qquad w_j\ge0,\quad\sum_jw_j=1 $$ Mixture loss is the weighted sum of expected per-example loss for each source. Sampling weights are nonnegative and sum to one. Symbols and units ℒ\_mix Expected loss under the selected source mixture loss units θ Model parameter vector model coordinates wⱼ Sampling weight of source j probability fraction Dⱼ Data distribution for source j probability distribution x Example sampled from Dⱼ example ℓ(θ,x) Per-example loss same loss units as ℒ\_mix j Source index integer index Each data source $D_{j}$ contributes according to its sampling weight $w_{j}$. Changing those weights changes which errors the optimizer is asked to reduce most often. Repeating near-duplicates can therefore increase their influence without adding comparable diversity. The same nominal token budget can represent very different learning opportunities. Assumptions This expectation assumes a fixed mixture and loss definition. Curriculum schedules, example weighting, repeated epochs, and adaptive sampling modify the effective weights. Source quality and legal provenance cannot be inferred from the equation. ### Dataset lineage and reproducible splits · worked relation $$ N_{records}=\sum_s N_s,\qquad N_{families}=|\{\operatorname{family}(x):x\in D\}| $$ Audit data lineage at the family and split boundaries. Symbols and units N\_records Stored example rows examples N\_s Rows in split s examples N\_families Distinct prompt-family identities families D Declared corpus dataset The executable corpus has 12 training, 8 development and 4 final examples per domain. Across copy and swap this is 24/16/8 example records, but only 12/8/4 distinct prompt families because each domain assigns its own answer to the same family. Treating all 48 records as independent prompt families would double-count the evidence. Assumptions Two intentionally conflicting domains share prompt families within the same split, never across splits. No external dataset license or privacy review is implied. ## Engineering example Synthetic source losses are 2 and 4 under the same loss definition. Sampling weights 0.75 and 0.25 give mixture loss 2.5; changing to 0.5 and 0.5 gives 3. The calculation changes source influence even when total example count is unchanged. If a training-only normalizer has mean 60 and standard deviation 10, input 70 maps to 1. Recomputing a mean of 70 using a later population maps the same input to 0. Comparing model predictions before and after that change as if only the weights mattered hides a preprocessing change. The new artifact lesson gives a logistic example where the resulting threshold decision reverses. The executable corpus has 12 training, 8 development and 4 final examples per domain. Across copy and swap this is 24/16/8 example records, but only 12/8/4 distinct prompt families because each domain assigns its own answer to the same family. Treating all 48 records as independent prompt families would double-count the evidence. At split seed17 in the original maintenance fixture, splitting individual documents places fan7-original in training and its exact mirror in evaluation. Whole-family assignment removes that flagged cross-split overlap. At default seed7, exact-copy exclusion removes three training records while preserving their original texts and retained-document pointers. ## Distinctions and tradeoffs Training tokens are not the same thing as a count of unique source documents. Cleaning can remove useful rare content as well as noise. Data composition must match intended capabilities and evaluation goals. Data supplies the examples that shape the model’s learned behavior. ## Specifications and interfaces Budget Tokens actually trained on Different from raw corpus size. Data quality Multidimensional Coverage, correctness, provenance, and duplication matter. ### Interfaces Storage Input Prepared examples Training loop Output Batches of token IDs and targets ## Check your understanding ### Does a larger raw corpus guarantee a better model? 1. No; quality and training use matter 2. Yes, without exception 3. Only if it contains images Answer and reasoning No; quality and training use matter Size alone misses duplication, usefulness, coverage, and optimization. ### Should held-out evaluation labels or future observations fit a preprocessing statistic used to claim an earlier forecast? 1. Yes, more data always improves the experiment 2. No, that violates the stated availability boundary 3. Only if the resulting accuracy increases Answer and reasoning No, that violates the stated availability boundary The comparison must use information that was allowed at training or prediction time. ### Why are there only 24 distinct prompt families across the 48 records? 1. Each prompt family is represented once in copy and once in swap. 2. Half the records are secretly final-test examples. 3. Every source record is necessarily an independent task. Answer and reasoning Each prompt family is represented once in copy and once in swap. Each prompt family is represented once in copy and once in swap. Two intentionally conflicting domains share prompt families within the same split, never across splits. No external dataset license or privacy review is implied. ### A mirrored document appears in training and evaluation after individual-document splitting. What does this establish? 1. The model must have memorized it 2. The evaluation has observable overlap that must be disclosed; model impact needs separate measurement 3. The mirror becomes independent because its ID differs Answer and reasoning The evaluation has observable overlap that must be disclosed; model impact needs separate measurement An ID difference does not erase shared content or lineage. The data relationship is measurable before training; a causal model-quality claim requires another experiment. ## Historical context ### 2021 · Deduplication is measured as a training intervention Lee and colleagues showed that removing repeated training material could reduce memorized output and train–test overlap while preserving or improving measured accuracy. Their preprint preceded ACL 2022 publication. [Lee et al. — Deduplicating Training Data Makes Language Models Better](https://arxiv.org/abs/2107.06499) ## Evidence and scope The fixed-mixture expectation is the relation stated in the lesson. Values are synthetic; deduplication, representativeness, contamination and provenance require independent checks and cannot be inferred from this weighted average. The sensor datasets and numeric normalization values are authored. They establish a leakage/artifact mechanism, not field data quality or generalization on real equipment. Mode: executable corpus lineage. Two intentionally conflicting domains share prompt families within the same split, never across splits. No external dataset license or privacy review is implied. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied. These are computed relationships in twelve authored documents across seven known families. No language-model quality, memorization rate or representative real-corpus coverage is measured. Use the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism. ## Sources - [Hoffmann et al. — Compute-Optimal Training](https://arxiv.org/abs/2203.15556) - [GPT-4 Technical Report](https://arxiv.org/abs/2303.08774) - [Lee et al. · duplicates and train-test overlap](https://arxiv.org/abs/2107.06499) - [scikit-learn · Common pitfalls and data leakage](https://scikit-learn.org/stable/common_pitfalls.html#data-leakage) - [The FineWeb Datasets](https://arxiv.org/html/2406.17557v1) - [Deduplicating Training Data Makes Language Models Better](https://arxiv.org/html/2107.06499v1) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/training-data-0781ba332be25aa0.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Foundations - [Tokenizer](https://semiaifoundry.com/learn/atlas/tokenizer/) ### Related lessons - [Loss, gradients & optimizer](https://semiaifoundry.com/learn/atlas/loss-optimizer/) - [Dataset and checkpoint storage hierarchy](https://semiaifoundry.com/learn/atlas/storage/) - [Scaling laws & resource allocation](https://semiaifoundry.com/learn/atlas/scaling-laws/) - [Model artifacts, rollout and drift](https://semiaifoundry.com/learn/atlas/model-artifact-rollout-drift/) - [Forecasts are tested forward in time](https://semiaifoundry.com/learn/atlas/temporal-forecast-validation/) - [Filtering, duplication and contamination](https://semiaifoundry.com/learn/atlas/data-quality-dedup-audit/) - [Mixtures, epochs and exposure](https://semiaifoundry.com/learn/atlas/data-mixture-sampling/) - [Packing documents and masking valid targets](https://semiaifoundry.com/learn/atlas/sequence-packing-loss-denominators/) - [Synthetic supervision and selection loops](https://semiaifoundry.com/learn/atlas/synthetic-data-selection-feedback/) - [Coverage across language and source slices](https://semiaifoundry.com/learn/atlas/data-coverage-language-slices/) - [Human judgments, disagreement and dataset lineage](https://semiaifoundry.com/learn/atlas/dataset-human-judgment-lineage/) - [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/)