{
  "schemaVersion": 1,
  "id": "atlas-lesson-data-mixture-sampling",
  "title": "Mixtures, epochs and exposure",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Calculate expected exposure per source and distinguish it from one finite seeded draw.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/data-mixture-sampling/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "model-lifecycle-data",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Mode: worked example. Independent source draws with replacement, positive pool sizes and probabilities summing to one. These are exposure calculations, not predicted learning curves. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/data-mixture-sampling-89d1b24806fbf75b.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#data-mixture-sampling",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "06f1277dc91bf54e7bbfc152dcac6a6a3b597f9dadbd6d6e7f93a387be804bef",
  "contentLength": 6008,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-data-mixture-sampling.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-data-mixture-sampling.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Mixtures, epochs and exposure",
      "start": 37,
      "end": 426
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 426,
      "end": 1353
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1353,
      "end": 1381
    },
    {
      "id": "section-4",
      "title": "Mixtures, epochs and exposure · worked relation",
      "start": 1381,
      "end": 2403
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2403,
      "end": 2801
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2801,
      "end": 3256
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3256,
      "end": 3688
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3688,
      "end": 3897
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 3897,
      "end": 3926
    },
    {
      "id": "section-10",
      "title": "Which pool receives more expected passes?",
      "start": 3926,
      "end": 4436
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4436,
      "end": 5049
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5049,
      "end": 5386
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 5386,
      "end": 5409
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 5409,
      "end": 5510
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 5510,
      "end": 6008
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#data-mixture-sampling"
    },
    {
      "title": "The FineWeb Datasets",
      "url": "https://arxiv.org/html/2406.17557v1"
    },
    {
      "title": "2 OLMo 2 Furious",
      "url": "https://arxiv.org/html/2501.00656v1"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/data-mixture-sampling-89d1b24806fbf75b.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Training data pipeline",
      "url": "https://semiaifoundry.com/learn/atlas/training-data/"
    },
    {
      "title": "Filtering, duplication and contamination",
      "url": "https://semiaifoundry.com/learn/atlas/data-quality-dedup-audit/"
    },
    {
      "title": "Packing documents and masking valid targets",
      "url": "https://semiaifoundry.com/learn/atlas/sequence-packing-loss-denominators/"
    },
    {
      "title": "Synthetic supervision and selection loops",
      "url": "https://semiaifoundry.com/learn/atlas/synthetic-data-selection-feedback/"
    },
    {
      "title": "Coverage across language and source slices",
      "url": "https://semiaifoundry.com/learn/atlas/data-coverage-language-slices/"
    }
  ],
  "recordHash": "6143fd5b2abb444119dcda9b9205655242f94e5a0a03c3e9398081a2cc40538f",
  "contentFormat": "text/markdown",
  "content": "model-lifecycle-data · Atlas v1.3.0\n\n# Mixtures, epochs and exposure\n\nCalculate expected exposure per source and distinguish it from one finite seeded draw.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#data-mixture-sampling)Mathematical, data and behavior space; logical dimensions are not physical distance\n\nLearning objectiveCalculate expected exposure per source and distinguish it from one finite seeded draw.\n\n## The mechanism\n\nA training mixture is a probability distribution over available data, not a statement that every unique token is seen once. A small source can receive more expected epochs than a large source even when it receives fewer total draws. Sampling with replacement permits repeats and omissions. Sampling without replacement within an epoch has a different cursor contract. Changing mixture weights changes the learning objective in expectation; it also changes which domains consume a fixed budget. Report unique pool size, total draws, probabilities, replacement policy and tokenizer identity together.\n\n1.  Count the unique available tokens under one tokenizer.\n2.  Allocate a fixed draw budget by source probability.\n3.  Divide expected draws by pool size to obtain expected epochs.\n4.  Inspect realized source counts and repeated examples for a specified RNG seed; do not replace them with their expectation.\n\n## Science and assumptions\n\n### Mixtures, epochs and exposure · worked relation\n\n$$\n\\mathbb E[D_i]=B p_i,\\qquad E_i=\\frac{B p_i}{U_i},\\qquad \\operatorname{Var}(D_i)=B p_i(1-p_i)\n$$\n\nCalculate expected exposure per source and distinguish it from one finite seeded draw.\n\nSymbols and units\n\nB\n\nTotal independent token draws\n\ntokens\n\np\\_i\n\nSource selection probability\n\ndimensionless\n\nU\\_i\n\nUnique tokens in source i\n\ntokens\n\nD\\_i\n\nRealized source draws\n\ntokens\n\nE\\_i\n\nExpected equivalent passes\n\nepochs\n\nPool A contains 100 unique tokens and B contains 20. For 240 token draws with weights 0.75 and 0.25, expected draws are 180 and 60, hence expected exposures are 1.8 and 3 epochs. Under independent source selection the standard deviation of A draws is sqrt(240×0.75×0.25)=6.708. An observed 174/66 allocation is possible and does not alter the configured 3:1 probabilities.\n\nAssumptions Independent source draws with replacement, positive pool sizes and probabilities summing to one. These are exposure calculations, not predicted learning curves.\n\n## Engineering example\n\nPool A contains 100 unique tokens and B contains 20. For 240 token draws with weights 0.75 and 0.25, expected draws are 180 and 60, hence expected exposures are 1.8 and 3 epochs. Under independent source selection the standard deviation of A draws is sqrt(240×0.75×0.25)=6.708. An observed 174/66 allocation is possible and does not alter the configured 3:1 probabilities.\n\n## Distinctions and tradeoffs\n\nIt is a mistake to conclude: A receives more passes because its pool is larger.\n\nUpweighting a small high-value source can strengthen its signal while increasing repetition and reducing the budget available for other domains.\n\nMake the effective training distribution inspectable rather than treating tokens as a homogeneous commodity. Calculate expected exposure per source and distinguish it from one finite seeded draw.\n\n## Specifications and interfaces\n\nLearning mode\n\nworked example\n\nThe executable decoder has 423 parameters; worked extensions do not imply an additional simulator.\n\nWorked contract\n\nCalculate expected exposure per source and distinguish it from one finite seeded draw.\n\nIndependent source draws with replacement, positive pool sizes and probabilities summing to one. These are exposure calculations, not predicted learning curves.\n\n### Interfaces\n\nInputs and state Input\n\nCount the unique available tokens under one tokenizer.\n\nResult and check Output\n\nCalculate expected exposure per source and distinguish it from one finite seeded draw.\n\n## Check your understanding\n\n### Which pool receives more expected passes?\n\n1.  Both receive exactly one pass.\n2.  B receives 3 expected passes despite only 60 expected draws.\n3.  A receives more passes because its pool is larger.\n\nAnswer and reasoning\n\nB receives 3 expected passes despite only 60 expected draws.\n\nB receives 3 expected passes despite only 60 expected draws. Independent source draws with replacement, positive pool sizes and probabilities summing to one. These are exposure calculations, not predicted learning curves.\n\n## Evidence and scope\n\nMode: worked example. Independent source draws with replacement, positive pool sizes and probabilities summing to one. These are exposure calculations, not predicted learning curves. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nModel lifecycle studio: worked example. Independent source draws with replacement, positive pool sizes and probabilities summing to one. These are exposure calculations, not predicted learning curves.\n\n## Sources\n\n-   [The FineWeb Datasets](https://arxiv.org/html/2406.17557v1)\n-   [2 OLMo 2 Furious](https://arxiv.org/html/2501.00656v1)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/data-mixture-sampling-89d1b24806fbf75b.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Training data pipeline](https://semiaifoundry.com/learn/atlas/training-data/)\n\n### Related lessons\n\n-   [Filtering, duplication and contamination](https://semiaifoundry.com/learn/atlas/data-quality-dedup-audit/)\n-   [Packing documents and masking valid targets](https://semiaifoundry.com/learn/atlas/sequence-packing-loss-denominators/)\n-   [Synthetic supervision and selection loops](https://semiaifoundry.com/learn/atlas/synthetic-data-selection-feedback/)\n-   [Coverage across language and source slices](https://semiaifoundry.com/learn/atlas/data-coverage-language-slices/)"
}