{"id":"training-data","level":"model","name":"Training data pipeline","shortName":"Training data pipeline","role":"Convert a source collection into useful learning examples.","description":"A training pipeline may collect, filter, deduplicate, format, tokenize, and batch data. The mixture controls which patterns the optimizer repeatedly encounters. Quality is not a single scalar: correctness, coverage, provenance, licensing, contamination, and representativeness all matter. Training, validation, and benchmark splits need careful separation and overlap checks. More bytes alone do not guarantee better learning; repeated low-value or contaminated examples can waste compute and distort evaluation.\n\nTraining examples also have an availability and transformation history. A label may arrive after an input was first observed, and preprocessing fitted on future or held-out cases can leak information across the evaluation boundary. A deployed artifact should retain which split supplied each learned statistic so later predictions can be reproduced without silently refitting on new data.\n\nAdvanced lifecycle practice retains source identity, transformation history, tokenizer version and document-family split assignment together. A train/development/final separation is meaningful only if near-identical families do not cross it. The executable symbol corpus records these fields, while its source is original synthetic material rather than a crawl or human conversation. Development answers support iteration; the first final report records the evaluated checkpoint and cannot be silently rewritten after another update.\n\nThe corpus lineage experience now executes a small construction chain: original document families are assigned to splits before normalization; exact training copies can be excluded with retained source pointers; a tokenizer fits only retained training fields; deterministic document sampling and packing produce inspectable arrays.","mechanism":["Prepare and document the source mixture.","Filter, deduplicate, tokenize, and form batches.","Track validation and contamination separately from optimization.","Feature construction and normalizer fitting use only the allowed training information.","Evaluation and later monitoring retain their own sample and label-availability boundaries rather than being folded back into the training statistic.","Preserve raw-source and transformed-example identities.","Assign whole document families to splits before tokenization-dependent sampling.","Record final exposure as part of the experimental history, including after importing an older checkpoint.","Compare exact transformed prompt/response pairs, three-word-shingle similarity and known-family relationships without treating any single diagnostic as semantic truth.","Record split seed, transformation, excluded IDs, tokenizer fitting IDs, merge order, sampler cursor and packed indices so the same declared batch can be reconstructed."],"specs":[{"label":"Budget","value":"Tokens actually trained on","context":"Different from raw corpus size."},{"label":"Data quality","value":"Multidimensional","context":"Coverage, correctness, provenance, and duplication matter."}],"interfaces":[{"name":"Storage","direction":"in","signal":"Prepared examples"},{"name":"Training loop","direction":"out","signal":"Batches of token IDs and targets"}],"tradeoff":"Cleaning can remove useful rare content as well as noise. Data composition must match intended capabilities and evaluation goals.","whyAI":"Data supplies the examples that shape the model’s learned behavior.","related":["tokenizer","loss-optimizer","storage","scaling-laws","model-artifact-rollout-drift","temporal-forecast-validation","data-quality-dedup-audit","data-mixture-sampling","sequence-packing-loss-denominators","synthetic-data-selection-feedback","data-coverage-language-slices","dataset-human-judgment-lineage","evaluation"],"sources":[{"title":"Hoffmann et al. — Compute-Optimal Training","url":"https://arxiv.org/abs/2203.15556"},{"title":"GPT-4 Technical Report","url":"https://arxiv.org/abs/2303.08774"},{"title":"Lee et al. · duplicates and train-test overlap","url":"https://arxiv.org/abs/2107.06499"},{"title":"scikit-learn · Common pitfalls and data leakage","url":"https://scikit-learn.org/stable/common_pitfalls.html#data-leakage"},{"title":"The FineWeb Datasets","url":"https://arxiv.org/html/2406.17557v1"},{"title":"Deduplicating Training Data Makes Language Models Better","url":"https://arxiv.org/html/2107.06499v1"}],"scale":"Logical structure · mapped onto physical hardware","kind":"training-data","misconception":"Training tokens are not the same thing as a count of unique source documents.","check":{"question":"Does a larger raw corpus guarantee a better model?","options":["No; quality and training use matter","Yes, without exception","Only if it contains images"],"answer":0,"explanation":"Size alone misses duplication, usefulness, coverage, and optimization."},"branch":"Learning","subbranch":"Data mixture and coverage","science":[{"title":"The mixture defines the expected learning signal","equation":"ℒ_mix(θ) = Σⱼ w_j·E[x∼D_j][ℓ(θ;x)]; w_j ≥ 0; Σⱼw_j = 1","explanation":"Each data source D_j contributes according to its sampling weight w_j. Changing those weights changes which errors the optimizer is asked to reduce most often. Repeating near-duplicates can therefore increase their influence without adding comparable diversity. The same nominal token budget can represent very different learning opportunities.","assumptions":"This expectation assumes a fixed mixture and loss definition. Curriculum schedules, example weighting, repeated epochs, and adaptive sampling modify the effective weights. Source quality and legal provenance cannot be inferred from the equation."},{"title":"Dataset lineage and reproducible splits · worked relation","equation":"N_{records}=\\sum_s N_s,\\qquad N_{families}=|\\{\\operatorname{family}(x):x\\in D\\}|","explanation":"The executable corpus has 12 training, 8 development and 4 final examples per domain. Across copy and swap this is 24/16/8 example records, but only 12/8/4 distinct prompt families because each domain assigns its own answer to the same family. Treating all 48 records as independent prompt families would double-count the evidence.","assumptions":"Two intentionally conflicting domains share prompt families within the same split, never across splits. No external dataset license or privacy review is implied."}],"history":[{"year":2021,"title":"Deduplication is measured as a training intervention","significance":"Lee and colleagues showed that removing repeated training material could reduce memorized output and train–test overlap while preserving or improving measured accuracy. Their preprint preceded ACL 2022 publication.","source":{"title":"Lee et al. — Deduplicating Training Data Makes Language Models Better","url":"https://arxiv.org/abs/2107.06499"}}],"prerequisites":["tokenizer"],"learningObjective":"Calculate a mixture-weighted loss and explain why changing the data mix changes the learning objective.","engineeringExample":"Synthetic source losses are 2 and 4 under the same loss definition. Sampling weights 0.75 and 0.25 give mixture loss 2.5; changing to 0.5 and 0.5 gives 3. The calculation changes source influence even when total example count is unchanged.\n\nIf a training-only normalizer has mean 60 and standard deviation 10, input 70 maps to 1. Recomputing a mean of 70 using a later population maps the same input to 0. Comparing model predictions before and after that change as if only the weights mattered hides a preprocessing change. The new artifact lesson gives a logistic example where the resulting threshold decision reverses.\n\nThe executable corpus has 12 training, 8 development and 4 final examples per domain. Across copy and swap this is 24/16/8 example records, but only 12/8/4 distinct prompt families because each domain assigns its own answer to the same family. Treating all 48 records as independent prompt families would double-count the evidence.\n\nAt split seed17 in the original maintenance fixture, splitting individual documents places fan7-original in training and its exact mirror in evaluation. Whole-family assignment removes that flagged cross-split overlap. At default seed7, exact-copy exclusion removes three training records while preserving their original texts and retained-document pointers.","evidenceNotes":"The fixed-mixture expectation is the relation stated in the lesson. Values are synthetic; deduplication, representativeness, contamination and provenance require independent checks and cannot be inferred from this weighted average.\n\nThe sensor datasets and numeric normalization values are authored. They establish a leakage/artifact mechanism, not field data quality or generalization on real equipment.\n\nMode: executable corpus lineage. Two intentionally conflicting domains share prompt families within the same split, never across splits. No external dataset license or privacy review is implied. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nThese are computed relationships in twelve authored documents across seven known families. No language-model quality, memorization rate or representative real-corpus coverage is measured.","checks":[{"question":"Should held-out evaluation labels or future observations fit a preprocessing statistic used to claim an earlier forecast?","options":["Yes, more data always improves the experiment","No, that violates the stated availability boundary","Only if the resulting accuracy increases"],"answer":1,"explanation":"The comparison must use information that was allowed at training or prediction time."},{"question":"Why are there only 24 distinct prompt families across the 48 records?","options":["Each prompt family is represented once in copy and once in swap.","Half the records are secretly final-test examples.","Every source record is necessarily an independent task."],"answer":0,"explanation":"Each prompt family is represented once in copy and once in swap. Two intentionally conflicting domains share prompt families within the same split, never across splits. No external dataset license or privacy review is implied."},{"question":"A mirrored document appears in training and evaluation after individual-document splitting. What does this establish?","options":["The model must have memorized it","The evaluation has observable overlap that must be disclosed; model impact needs separate measurement","The mirror becomes independent because its ID differs"],"answer":1,"explanation":"An ID difference does not erase shared content or lineage. The data relationship is measurable before training; a causal model-quality claim requires another experiment."}],"labId":"library","scope":"Use the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism.","investigationId":"execution","labScope":"Use the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism."}