# Filtering, duplication and contamination Canonical: https://semiaifoundry.com/learn/atlas/data-quality-dedup-audit/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html model-lifecycle-data · Atlas v1.3.0 # Filtering, duplication and contamination Distinguish exact duplicates, approximate matches and evaluation contamination using explicit document families. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#data-quality-dedup-audit)Mathematical, data and behavior space; logical dimensions are not physical distance Learning objectiveDistinguish exact duplicates, approximate matches and evaluation contamination using explicit document families. ## The mechanism Duplicate detection defines an equivalence or similarity rule over content; it does not directly measure whether that content is true or useful. Raw-byte identity, normalized-text identity and overlap between token shingles can produce different groups. A group must be assigned to a split before its members are sampled, otherwise paraphrases or mirrors can leak across a nominal document split. Removing repeated data also changes exposure and source balance. A false merge discards distinct information, while a missed duplicate can overrepresent one source. The appropriate audit records both kinds of error on reviewed examples and measures downstream effects at a fixed token budget. 1. Preserve original bytes and source identity before normalization. 2. Compute exact equality and a declared shingle similarity separately. 3. Form document-family groups, inspect borderline pairs, and assign whole groups to train, development or final. 4. Count unique retained families, mistaken merges and cross-split matches with separate denominators. ## Science and assumptions ### Filtering, duplication and contamination · worked relation $$ J(A,B)=\frac{|A\cap B|}{|A\cup B|},\qquad r_{\rm files}=1-\frac{N_{\rm kept}}{N_{\rm input}} $$ Distinguish exact duplicates, approximate matches and evaluation contamination using explicit document families. Symbols and units A,B Sets of declared shingles set J Jaccard overlap for a nonempty union dimensionless N Document counts at the specified boundary documents r\_files Removed-file fraction dimensionless Authored documents A and B have shingle sets {red, square, left} and {red, square, right}. Their intersection has 2 items and union 4, so Jaccard similarity is 0.5. A threshold of 0.5 merges them even though left and right encode a different answer. Two other files with identical normalized text but different whitespace are exact normalized duplicates. Of 12 input files, retaining 9 representatives means a 25% file reduction; it says nothing by itself about token reduction or correctness. Assumptions Nonempty shingle union; exact set arithmetic. Threshold and authored semantic labels are teaching choices, not a universal deduplication rule. ## Engineering example Authored documents A and B have shingle sets {red, square, left} and {red, square, right}. Their intersection has 2 items and union 4, so Jaccard similarity is 0.5. A threshold of 0.5 merges them even though left and right encode a different answer. Two other files with identical normalized text but different whitespace are exact normalized duplicates. Of 12 input files, retaining 9 representatives means a 25% file reduction; it says nothing by itself about token reduction or correctness. ## Distinctions and tradeoffs It is a mistake to conclude: Jaccard overlap proves the two statements have the same meaning. Looser matching catches more repeated material but can erase rare distinctions. Stricter matching retains more variety and more missed duplicates. Make the effective training distribution inspectable rather than treating tokens as a homogeneous commodity. Distinguish exact duplicates, approximate matches and evaluation contamination using explicit document families. ## Specifications and interfaces Learning mode worked example The executable decoder has 423 parameters; worked extensions do not imply an additional simulator. Worked contract Distinguish exact duplicates, approximate matches and evaluation contamination using explicit document families. Nonempty shingle union; exact set arithmetic. Threshold and authored semantic labels are teaching choices, not a universal deduplication rule. ### Interfaces Inputs and state Input Preserve original bytes and source identity before normalization. Result and check Output Distinguish exact duplicates, approximate matches and evaluation contamination using explicit document families. ## Check your understanding ### Why can a 0.5 similarity threshold remove useful information in this example? 1. The shared shingles hide the answer-changing left/right distinction. 2. Jaccard overlap proves the two statements have the same meaning. 3. Removing a quarter of files proves a quarter of tokens were removed. Answer and reasoning The shared shingles hide the answer-changing left/right distinction. The shared shingles hide the answer-changing left/right distinction. Nonempty shingle union; exact set arithmetic. Threshold and authored semantic labels are teaching choices, not a universal deduplication rule. ## Evidence and scope Mode: worked example. Nonempty shingle union; exact set arithmetic. Threshold and authored semantic labels are teaching choices, not a universal deduplication rule. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied. Model lifecycle studio: worked example. Nonempty shingle union; exact set arithmetic. Threshold and authored semantic labels are teaching choices, not a universal deduplication rule. ## Sources - [The FineWeb Datasets](https://arxiv.org/html/2406.17557v1) - [Deduplicating Training Data Makes Language Models Better](https://arxiv.org/html/2107.06499v1) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/data-quality-dedup-audit-ba4b0dcd4d72bfb6.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Foundations - [Training data pipeline](https://semiaifoundry.com/learn/atlas/training-data/) ### Related lessons - [Mixtures, epochs and exposure](https://semiaifoundry.com/learn/atlas/data-mixture-sampling/) - [Packing documents and masking valid targets](https://semiaifoundry.com/learn/atlas/sequence-packing-loss-denominators/) - [Synthetic supervision and selection loops](https://semiaifoundry.com/learn/atlas/synthetic-data-selection-feedback/) - [Coverage across language and source slices](https://semiaifoundry.com/learn/atlas/data-coverage-language-slices/)