Filtering, duplication and contamination

The mechanism

Duplicate detection defines an equivalence or similarity rule over content; it does not directly measure whether that content is true or useful. Raw-byte identity, normalized-text identity and overlap between token shingles can produce different groups. A group must be assigned to a split before its members are sampled, otherwise paraphrases or mirrors can leak across a nominal document split. Removing repeated data also changes exposure and source balance. A false merge discards distinct information, while a missed duplicate can overrepresent one source. The appropriate audit records both kinds of error on reviewed examples and measures downstream effects at a fixed token budget.

1. Preserve original bytes and source identity before normalization.
2. Compute exact equality and a declared shingle similarity separately.
3. Form document-family groups, inspect borderline pairs, and assign whole groups to train, development or final.
4. Count unique retained families, mistaken merges and cross-split matches with separate denominators.

Science and assumptions

Engineering example

Authored documents A and B have shingle sets {red, square, left} and {red, square, right}. Their intersection has 2 items and union 4, so Jaccard similarity is 0.5. A threshold of 0.5 merges them even though left and right encode a different answer. Two other files with identical normalized text but different whitespace are exact normalized duplicates. Of 12 input files, retaining 9 representatives means a 25% file reduction; it says nothing by itself about token reduction or correctness.

Distinctions and tradeoffs

It is a mistake to conclude: Jaccard overlap proves the two statements have the same meaning.

Looser matching catches more repeated material but can erase rare distinctions. Stricter matching retains more variety and more missed duplicates.

Make the effective training distribution inspectable rather than treating tokens as a homogeneous commodity. Distinguish exact duplicates, approximate matches and evaluation contamination using explicit document families.

Specifications and interfaces

Learning mode

worked example

The executable decoder has 423 parameters; worked extensions do not imply an additional simulator.

Worked contract

Distinguish exact duplicates, approximate matches and evaluation contamination using explicit document families.

Nonempty shingle union; exact set arithmetic. Threshold and authored semantic labels are teaching choices, not a universal deduplication rule.

Interfaces

Inputs and state Input

Preserve original bytes and source identity before normalization.

Result and check Output

Distinguish exact duplicates, approximate matches and evaluation contamination using explicit document families.

Check your understanding

Evidence and scope

Mode: worked example. Nonempty shingle union; exact set arithmetic. Threshold and authored semantic labels are teaching choices, not a universal deduplication rule. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.

Model lifecycle studio: worked example. Nonempty shingle union; exact set arithmetic. Threshold and authored semantic labels are teaching choices, not a universal deduplication rule.

Sources

• The FineWeb Datasets
• Deduplicating Training Data Makes Language Models Better

Published lesson data · Atlas releases and licensing

Continue exploring

Foundations

• Training data pipeline

Related lessons

• Mixtures, epochs and exposure
• Packing documents and masking valid targets
• Synthetic supervision and selection loops
• Coverage across language and source slices