{"id":"data-quality-dedup-audit","name":"Filtering, duplication and contamination","shortName":"Filtering, duplication and contamination","role":"Distinguish exact duplicates, approximate matches and evaluation contamination using explicit document families.","description":"Duplicate detection defines an equivalence or similarity rule over content; it does not directly measure whether that content is true or useful. Raw-byte identity, normalized-text identity and overlap between token shingles can produce different groups. A group must be assigned to a split before its members are sampled, otherwise paraphrases or mirrors can leak across a nominal document split. Removing repeated data also changes exposure and source balance. A false merge discards distinct information, while a missed duplicate can overrepresent one source. The appropriate audit records both kinds of error on reviewed examples and measures downstream effects at a fixed token budget.","mechanism":["Preserve original bytes and source identity before normalization.","Compute exact equality and a declared shingle similarity separately.","Form document-family groups, inspect borderline pairs, and assign whole groups to train, development or final.","Count unique retained families, mistaken merges and cross-split matches with separate denominators."],"learningObjective":"Distinguish exact duplicates, approximate matches and evaluation contamination using explicit document families.","engineeringExample":"Authored documents A and B have shingle sets {red, square, left} and {red, square, right}. Their intersection has 2 items and union 4, so Jaccard similarity is 0.5. A threshold of 0.5 merges them even though left and right encode a different answer. Two other files with identical normalized text but different whitespace are exact normalized duplicates. Of 12 input files, retaining 9 representatives means a 25% file reduction; it says nothing by itself about token reduction or correctness.","evidenceNotes":"Mode: worked example. Nonempty shingle union; exact set arithmetic. Threshold and authored semantic labels are teaching choices, not a universal deduplication rule. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","specs":[{"label":"Learning mode","value":"worked example","context":"The executable decoder has 423 parameters; worked extensions do not imply an additional simulator."},{"label":"Worked contract","value":"Distinguish exact duplicates, approximate matches and evaluation contamination using explicit document families.","context":"Nonempty shingle union; exact set arithmetic. Threshold and authored semantic labels are teaching choices, not a universal deduplication rule."}],"interfaces":[{"name":"Inputs and state","direction":"in","signal":"Preserve original bytes and source identity before normalization."},{"name":"Result and check","direction":"out","signal":"Distinguish exact duplicates, approximate matches and evaluation contamination using explicit document families."}],"tradeoff":"Looser matching catches more repeated material but can erase rare distinctions. Stricter matching retains more variety and more missed duplicates.","whyAI":"Make the effective training distribution inspectable rather than treating tokens as a homogeneous commodity. Distinguish exact duplicates, approximate matches and evaluation contamination using explicit document families.","misconception":"It is a mistake to conclude: Jaccard overlap proves the two statements have the same meaning.","check":{"question":"Why can a 0.5 similarity threshold remove useful information in this example?","options":["The shared shingles hide the answer-changing left/right distinction.","Jaccard overlap proves the two statements have the same meaning.","Removing a quarter of files proves a quarter of tokens were removed."],"answer":0,"explanation":"The shared shingles hide the answer-changing left/right distinction. Nonempty shingle union; exact set arithmetic. Threshold and authored semantic labels are teaching choices, not a universal deduplication rule."},"prerequisites":["training-data"],"related":["training-data","data-mixture-sampling","sequence-packing-loss-denominators","synthetic-data-selection-feedback","data-coverage-language-slices"],"sources":[{"title":"The FineWeb Datasets","url":"https://arxiv.org/html/2406.17557v1"},{"title":"Deduplicating Training Data Makes Language Models Better","url":"https://arxiv.org/html/2107.06499v1"}],"level":"model","scale":"Mathematical, data and behavior space; logical dimensions are not physical distance","kind":"model","branch":"model-lifecycle-data","subbranch":"Filtering, duplication and contamination","workstream":"W11","visualFamily":"mathematical","labId":"library","labScope":"Model lifecycle studio: worked example. Nonempty shingle union; exact set arithmetic. Threshold and authored semantic labels are teaching choices, not a universal deduplication rule.","science":[{"title":"Filtering, duplication and contamination · worked relation","equation":"J(A,B)=\\frac{|A\\cap B|}{|A\\cup B|},\\qquad r_{\\rm files}=1-\\frac{N_{\\rm kept}}{N_{\\rm input}}","explanation":"Authored documents A and B have shingle sets {red, square, left} and {red, square, right}. Their intersection has 2 items and union 4, so Jaccard similarity is 0.5. A threshold of 0.5 merges them even though left and right encode a different answer. Two other files with identical normalized text but different whitespace are exact normalized duplicates. Of 12 input files, retaining 9 representatives means a 25% file reduction; it says nothing by itself about token reduction or correctness.","assumptions":"Nonempty shingle union; exact set arithmetic. Threshold and authored semantic labels are teaching choices, not a universal deduplication rule."}],"history":[]}