{
  "schemaVersion": 1,
  "id": "atlas-lesson-data-quality-dedup-audit",
  "title": "Filtering, duplication and contamination",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Distinguish exact duplicates, approximate matches and evaluation contamination using explicit document families.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/data-quality-dedup-audit/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "model-lifecycle-data",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Mode: worked example. Nonempty shingle union; exact set arithmetic. Threshold and authored semantic labels are teaching choices, not a universal deduplication rule. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/data-quality-dedup-audit-ba4b0dcd4d72bfb6.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#data-quality-dedup-audit",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "89dce7ee13256bceafb26ff8a54b8d871d06d709cf69c4f46834041f0378fee3",
  "contentLength": 6638,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-data-quality-dedup-audit.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-data-quality-dedup-audit.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Filtering, duplication and contamination",
      "start": 37,
      "end": 492
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 492,
      "end": 1564
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1564,
      "end": 1592
    },
    {
      "id": "section-4",
      "title": "Filtering, duplication and contamination · worked relation",
      "start": 1592,
      "end": 2738
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2738,
      "end": 3257
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 3257,
      "end": 3754
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3754,
      "end": 4194
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 4194,
      "end": 4440
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 4440,
      "end": 4469
    },
    {
      "id": "section-10",
      "title": "Why can a 0.5 similarity threshold remove useful information in this example?",
      "start": 4469,
      "end": 5073
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 5073,
      "end": 5650
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5650,
      "end": 6030
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 6030,
      "end": 6053
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 6053,
      "end": 6154
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 6154,
      "end": 6638
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#data-quality-dedup-audit"
    },
    {
      "title": "The FineWeb Datasets",
      "url": "https://arxiv.org/html/2406.17557v1"
    },
    {
      "title": "Deduplicating Training Data Makes Language Models Better",
      "url": "https://arxiv.org/html/2107.06499v1"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/data-quality-dedup-audit-ba4b0dcd4d72bfb6.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Training data pipeline",
      "url": "https://semiaifoundry.com/learn/atlas/training-data/"
    },
    {
      "title": "Mixtures, epochs and exposure",
      "url": "https://semiaifoundry.com/learn/atlas/data-mixture-sampling/"
    },
    {
      "title": "Packing documents and masking valid targets",
      "url": "https://semiaifoundry.com/learn/atlas/sequence-packing-loss-denominators/"
    },
    {
      "title": "Synthetic supervision and selection loops",
      "url": "https://semiaifoundry.com/learn/atlas/synthetic-data-selection-feedback/"
    },
    {
      "title": "Coverage across language and source slices",
      "url": "https://semiaifoundry.com/learn/atlas/data-coverage-language-slices/"
    }
  ],
  "recordHash": "750939e4bad336dfe4fc46e58ac0484fd939f91f9caffd659235578dc87c7751",
  "contentFormat": "text/markdown",
  "content": "model-lifecycle-data · Atlas v1.3.0\n\n# Filtering, duplication and contamination\n\nDistinguish exact duplicates, approximate matches and evaluation contamination using explicit document families.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#data-quality-dedup-audit)Mathematical, data and behavior space; logical dimensions are not physical distance\n\nLearning objectiveDistinguish exact duplicates, approximate matches and evaluation contamination using explicit document families.\n\n## The mechanism\n\nDuplicate detection defines an equivalence or similarity rule over content; it does not directly measure whether that content is true or useful. Raw-byte identity, normalized-text identity and overlap between token shingles can produce different groups. A group must be assigned to a split before its members are sampled, otherwise paraphrases or mirrors can leak across a nominal document split. Removing repeated data also changes exposure and source balance. A false merge discards distinct information, while a missed duplicate can overrepresent one source. The appropriate audit records both kinds of error on reviewed examples and measures downstream effects at a fixed token budget.\n\n1.  Preserve original bytes and source identity before normalization.\n2.  Compute exact equality and a declared shingle similarity separately.\n3.  Form document-family groups, inspect borderline pairs, and assign whole groups to train, development or final.\n4.  Count unique retained families, mistaken merges and cross-split matches with separate denominators.\n\n## Science and assumptions\n\n### Filtering, duplication and contamination · worked relation\n\n$$\nJ(A,B)=\\frac{|A\\cap B|}{|A\\cup B|},\\qquad r_{\\rm files}=1-\\frac{N_{\\rm kept}}{N_{\\rm input}}\n$$\n\nDistinguish exact duplicates, approximate matches and evaluation contamination using explicit document families.\n\nSymbols and units\n\nA,B\n\nSets of declared shingles\n\nset\n\nJ\n\nJaccard overlap for a nonempty union\n\ndimensionless\n\nN\n\nDocument counts at the specified boundary\n\ndocuments\n\nr\\_files\n\nRemoved-file fraction\n\ndimensionless\n\nAuthored documents A and B have shingle sets {red, square, left} and {red, square, right}. Their intersection has 2 items and union 4, so Jaccard similarity is 0.5. A threshold of 0.5 merges them even though left and right encode a different answer. Two other files with identical normalized text but different whitespace are exact normalized duplicates. Of 12 input files, retaining 9 representatives means a 25% file reduction; it says nothing by itself about token reduction or correctness.\n\nAssumptions Nonempty shingle union; exact set arithmetic. Threshold and authored semantic labels are teaching choices, not a universal deduplication rule.\n\n## Engineering example\n\nAuthored documents A and B have shingle sets {red, square, left} and {red, square, right}. Their intersection has 2 items and union 4, so Jaccard similarity is 0.5. A threshold of 0.5 merges them even though left and right encode a different answer. Two other files with identical normalized text but different whitespace are exact normalized duplicates. Of 12 input files, retaining 9 representatives means a 25% file reduction; it says nothing by itself about token reduction or correctness.\n\n## Distinctions and tradeoffs\n\nIt is a mistake to conclude: Jaccard overlap proves the two statements have the same meaning.\n\nLooser matching catches more repeated material but can erase rare distinctions. Stricter matching retains more variety and more missed duplicates.\n\nMake the effective training distribution inspectable rather than treating tokens as a homogeneous commodity. Distinguish exact duplicates, approximate matches and evaluation contamination using explicit document families.\n\n## Specifications and interfaces\n\nLearning mode\n\nworked example\n\nThe executable decoder has 423 parameters; worked extensions do not imply an additional simulator.\n\nWorked contract\n\nDistinguish exact duplicates, approximate matches and evaluation contamination using explicit document families.\n\nNonempty shingle union; exact set arithmetic. Threshold and authored semantic labels are teaching choices, not a universal deduplication rule.\n\n### Interfaces\n\nInputs and state Input\n\nPreserve original bytes and source identity before normalization.\n\nResult and check Output\n\nDistinguish exact duplicates, approximate matches and evaluation contamination using explicit document families.\n\n## Check your understanding\n\n### Why can a 0.5 similarity threshold remove useful information in this example?\n\n1.  The shared shingles hide the answer-changing left/right distinction.\n2.  Jaccard overlap proves the two statements have the same meaning.\n3.  Removing a quarter of files proves a quarter of tokens were removed.\n\nAnswer and reasoning\n\nThe shared shingles hide the answer-changing left/right distinction.\n\nThe shared shingles hide the answer-changing left/right distinction. Nonempty shingle union; exact set arithmetic. Threshold and authored semantic labels are teaching choices, not a universal deduplication rule.\n\n## Evidence and scope\n\nMode: worked example. Nonempty shingle union; exact set arithmetic. Threshold and authored semantic labels are teaching choices, not a universal deduplication rule. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nModel lifecycle studio: worked example. Nonempty shingle union; exact set arithmetic. Threshold and authored semantic labels are teaching choices, not a universal deduplication rule.\n\n## Sources\n\n-   [The FineWeb Datasets](https://arxiv.org/html/2406.17557v1)\n-   [Deduplicating Training Data Makes Language Models Better](https://arxiv.org/html/2107.06499v1)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/data-quality-dedup-audit-ba4b0dcd4d72bfb6.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Training data pipeline](https://semiaifoundry.com/learn/atlas/training-data/)\n\n### Related lessons\n\n-   [Mixtures, epochs and exposure](https://semiaifoundry.com/learn/atlas/data-mixture-sampling/)\n-   [Packing documents and masking valid targets](https://semiaifoundry.com/learn/atlas/sequence-packing-loss-denominators/)\n-   [Synthetic supervision and selection loops](https://semiaifoundry.com/learn/atlas/synthetic-data-selection-feedback/)\n-   [Coverage across language and source slices](https://semiaifoundry.com/learn/atlas/data-coverage-language-slices/)"
}