Mixtures, epochs and exposure

The mechanism

A training mixture is a probability distribution over available data, not a statement that every unique token is seen once. A small source can receive more expected epochs than a large source even when it receives fewer total draws. Sampling with replacement permits repeats and omissions. Sampling without replacement within an epoch has a different cursor contract. Changing mixture weights changes the learning objective in expectation; it also changes which domains consume a fixed budget. Report unique pool size, total draws, probabilities, replacement policy and tokenizer identity together.

1. Count the unique available tokens under one tokenizer.
2. Allocate a fixed draw budget by source probability.
3. Divide expected draws by pool size to obtain expected epochs.
4. Inspect realized source counts and repeated examples for a specified RNG seed; do not replace them with their expectation.

Science and assumptions

Engineering example

Pool A contains 100 unique tokens and B contains 20. For 240 token draws with weights 0.75 and 0.25, expected draws are 180 and 60, hence expected exposures are 1.8 and 3 epochs. Under independent source selection the standard deviation of A draws is sqrt(240×0.75×0.25)=6.708. An observed 174/66 allocation is possible and does not alter the configured 3:1 probabilities.

Distinctions and tradeoffs

It is a mistake to conclude: A receives more passes because its pool is larger.

Upweighting a small high-value source can strengthen its signal while increasing repetition and reducing the budget available for other domains.

Make the effective training distribution inspectable rather than treating tokens as a homogeneous commodity. Calculate expected exposure per source and distinguish it from one finite seeded draw.

Specifications and interfaces

Learning mode

worked example

The executable decoder has 423 parameters; worked extensions do not imply an additional simulator.

Worked contract

Calculate expected exposure per source and distinguish it from one finite seeded draw.

Independent source draws with replacement, positive pool sizes and probabilities summing to one. These are exposure calculations, not predicted learning curves.

Interfaces

Inputs and state Input

Count the unique available tokens under one tokenizer.

Result and check Output

Calculate expected exposure per source and distinguish it from one finite seeded draw.

Check your understanding

Evidence and scope

Mode: worked example. Independent source draws with replacement, positive pool sizes and probabilities summing to one. These are exposure calculations, not predicted learning curves. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.

Model lifecycle studio: worked example. Independent source draws with replacement, positive pool sizes and probabilities summing to one. These are exposure calculations, not predicted learning curves.

Sources

• The FineWeb Datasets
• 2 OLMo 2 Furious

Published lesson data · Atlas releases and licensing

Continue exploring

Foundations

• Training data pipeline

Related lessons

• Filtering, duplication and contamination
• Packing documents and masking valid targets
• Synthetic supervision and selection loops
• Coverage across language and source slices