{
  "schemaVersion": 1,
  "id": "atlas-lesson-data-coverage-language-slices",
  "title": "Coverage across language and source slices",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Compare source slices under explicit byte, token and task denominators.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/data-coverage-language-slices/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "model-lifecycle-data",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Mode: worked example. Authored strings and slice counts; UTF-8 and a fixed character tokenizer. Cross-tokenizer loss values require further normalization and do not directly rank language ability. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/data-coverage-language-slices-d2af2cc16e30915c.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#data-coverage-language-slices",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "714bc48bc019b3652f84f665302ffd37d3740b2f235a65bf7f402b091f8b4700",
  "contentLength": 6105,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-data-coverage-language-slices.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-data-coverage-language-slices.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Coverage across language and source slices",
      "start": 37,
      "end": 417
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 417,
      "end": 1327
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1327,
      "end": 1355
    },
    {
      "id": "section-4",
      "title": "Coverage across language and source slices · worked relation",
      "start": 1355,
      "end": 2430
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2430,
      "end": 2848
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2848,
      "end": 3272
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3272,
      "end": 3703
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3703,
      "end": 3915
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 3915,
      "end": 3944
    },
    {
      "id": "section-10",
      "title": "Which observation is concealed by the overall 76.36% retention?",
      "start": 3944,
      "end": 4430
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4430,
      "end": 5071
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5071,
      "end": 5438
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 5438,
      "end": 5461
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 5461,
      "end": 5628
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 5628,
      "end": 6105
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#data-coverage-language-slices"
    },
    {
      "title": "The FineWeb Datasets",
      "url": "https://arxiv.org/html/2406.17557v1"
    },
    {
      "title": "Holistic Evaluation of Language Models",
      "url": "https://arxiv.org/html/2211.09110v1"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/data-coverage-language-slices-d2af2cc16e30915c.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Training data pipeline",
      "url": "https://semiaifoundry.com/learn/atlas/training-data/"
    },
    {
      "title": "Tokenizer",
      "url": "https://semiaifoundry.com/learn/atlas/tokenizer/"
    },
    {
      "title": "Filtering, duplication and contamination",
      "url": "https://semiaifoundry.com/learn/atlas/data-quality-dedup-audit/"
    },
    {
      "title": "Mixtures, epochs and exposure",
      "url": "https://semiaifoundry.com/learn/atlas/data-mixture-sampling/"
    },
    {
      "title": "Packing documents and masking valid targets",
      "url": "https://semiaifoundry.com/learn/atlas/sequence-packing-loss-denominators/"
    },
    {
      "title": "Synthetic supervision and selection loops",
      "url": "https://semiaifoundry.com/learn/atlas/synthetic-data-selection-feedback/"
    }
  ],
  "recordHash": "70c060f69b6a17125296ddeb97a9e8c09b778b7d6834ab140172ca3a1eaf87fc",
  "contentFormat": "text/markdown",
  "content": "model-lifecycle-data · Atlas v1.3.0\n\n# Coverage across language and source slices\n\nCompare source slices under explicit byte, token and task denominators.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#data-coverage-language-slices)Mathematical, data and behavior space; logical dimensions are not physical distance\n\nLearning objectiveCompare source slices under explicit byte, token and task denominators.\n\n## The mechanism\n\nToken count depends on both text and tokenizer. Different scripts and source formats can consume different token budgets for comparable tasks; byte length introduces another encoding-dependent denominator. Neither a low token count nor a low token-normalized loss alone establishes better understanding. A coverage audit preserves source and language labels, inspects retention after filtering, and measures quality on declared task slices. FineWeb’s documented English filtering is a scoped recipe, not evidence that its thresholds fit every language or community.\n\n1.  Record the exact UTF-8 text and tokenizer version for each source slice.\n2.  Count retained documents, bytes and tokens separately.\n3.  Compute loss and downstream success within slices before aggregating.\n4.  Change a mixture only after stating whether the goal is token exposure, task coverage or a quality threshold.\n\n## Science and assumptions\n\n### Coverage across language and source slices · worked relation\n\n$$\nr_s=\\frac{N_{{\\rm kept},s}}{N_{{\\rm input},s}},\\qquad \\bar\\ell_s=\\frac{\\sum_{t\\in s}\\ell_t}{T_s}\n$$\n\nCompare source slices under explicit byte, token and task denominators.\n\nSymbols and units\n\nr\\_s\n\nRetention within source slice s\n\ndimensionless\n\nN\n\nDocument count\n\ndocuments\n\nT\\_s\n\nIncluded targets under one tokenizer\n\ntokens\n\nell\\_t\n\nOne target negative log likelihood\n\nnats\n\nbar ell\\_s\n\nMean slice loss\n\nnats per token\n\nThe authored strings \"cat\" and \"猫\" each occupy 3 UTF-8 bytes, but a character tokenizer emits 3 and 1 tokens respectively. This is a coding example, not a comparison of linguistic meaning. In a separate audit, slice A retains 80/100 documents and B retains 4/10. Overall retention is 84/110=76.36%, which conceals B’s 40% retention. Reporting only the overall value would miss that imbalance.\n\nAssumptions Authored strings and slice counts; UTF-8 and a fixed character tokenizer. Cross-tokenizer loss values require further normalization and do not directly rank language ability.\n\n## Engineering example\n\nThe authored strings \"cat\" and \"猫\" each occupy 3 UTF-8 bytes, but a character tokenizer emits 3 and 1 tokens respectively. This is a coding example, not a comparison of linguistic meaning. In a separate audit, slice A retains 80/100 documents and B retains 4/10. Overall retention is 84/110=76.36%, which conceals B’s 40% retention. Reporting only the overall value would miss that imbalance.\n\n## Distinctions and tradeoffs\n\nIt is a mistake to conclude: Both slices retained 76.36%.\n\nAggregate scores simplify comparison but can conceal minority-source failures. Slice metrics expose them at the cost of smaller, noisier denominators.\n\nMake the effective training distribution inspectable rather than treating tokens as a homogeneous commodity. Compare source slices under explicit byte, token and task denominators.\n\n## Specifications and interfaces\n\nLearning mode\n\nworked example\n\nThe executable decoder has 423 parameters; worked extensions do not imply an additional simulator.\n\nWorked contract\n\nCompare source slices under explicit byte, token and task denominators.\n\nAuthored strings and slice counts; UTF-8 and a fixed character tokenizer. Cross-tokenizer loss values require further normalization and do not directly rank language ability.\n\n### Interfaces\n\nInputs and state Input\n\nRecord the exact UTF-8 text and tokenizer version for each source slice.\n\nResult and check Output\n\nCompare source slices under explicit byte, token and task denominators.\n\n## Check your understanding\n\n### Which observation is concealed by the overall 76.36% retention?\n\n1.  Three UTF-8 bytes always mean three tokens.\n2.  Slice B retained only 40% of its documents.\n3.  Both slices retained 76.36%.\n\nAnswer and reasoning\n\nSlice B retained only 40% of its documents.\n\nSlice B retained only 40% of its documents. Authored strings and slice counts; UTF-8 and a fixed character tokenizer. Cross-tokenizer loss values require further normalization and do not directly rank language ability.\n\n## Evidence and scope\n\nMode: worked example. Authored strings and slice counts; UTF-8 and a fixed character tokenizer. Cross-tokenizer loss values require further normalization and do not directly rank language ability. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nModel lifecycle studio: worked example. Authored strings and slice counts; UTF-8 and a fixed character tokenizer. Cross-tokenizer loss values require further normalization and do not directly rank language ability.\n\n## Sources\n\n-   [The FineWeb Datasets](https://arxiv.org/html/2406.17557v1)\n-   [Holistic Evaluation of Language Models](https://arxiv.org/html/2211.09110v1)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/data-coverage-language-slices-d2af2cc16e30915c.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Training data pipeline](https://semiaifoundry.com/learn/atlas/training-data/)\n-   [Tokenizer](https://semiaifoundry.com/learn/atlas/tokenizer/)\n\n### Related lessons\n\n-   [Filtering, duplication and contamination](https://semiaifoundry.com/learn/atlas/data-quality-dedup-audit/)\n-   [Mixtures, epochs and exposure](https://semiaifoundry.com/learn/atlas/data-mixture-sampling/)\n-   [Packing documents and masking valid targets](https://semiaifoundry.com/learn/atlas/sequence-packing-loss-denominators/)\n-   [Synthetic supervision and selection loops](https://semiaifoundry.com/learn/atlas/synthetic-data-selection-feedback/)"
}