Coverage across language and source slices

The mechanism

Token count depends on both text and tokenizer. Different scripts and source formats can consume different token budgets for comparable tasks; byte length introduces another encoding-dependent denominator. Neither a low token count nor a low token-normalized loss alone establishes better understanding. A coverage audit preserves source and language labels, inspects retention after filtering, and measures quality on declared task slices. FineWeb’s documented English filtering is a scoped recipe, not evidence that its thresholds fit every language or community.

Record the exact UTF-8 text and tokenizer version for each source slice.
Count retained documents, bytes and tokens separately.
Compute loss and downstream success within slices before aggregating.
Change a mixture only after stating whether the goal is token exposure, task coverage or a quality threshold.

Science and assumptions

Coverage across language and source slices · worked relation

r_s=\frac{N_{{\rm kept},s}}{N_{{\rm input},s}},\qquad \bar\ell_s=\frac{\sum_{t\in s}\ell_t}{T_s}

Compare source slices under explicit byte, token and task denominators.

Symbols and units

r_s: Retention within source slice s
dimensionless
N: Document count
documents
T_s: Included targets under one tokenizer
tokens
ell_t: One target negative log likelihood
nats
bar ell_s: Mean slice loss
nats per token

The authored strings "cat" and "猫" each occupy 3 UTF-8 bytes, but a character tokenizer emits 3 and 1 tokens respectively. This is a coding example, not a comparison of linguistic meaning. In a separate audit, slice A retains 80/100 documents and B retains 4/10. Overall retention is 84/110=76.36%, which conceals B’s 40% retention. Reporting only the overall value would miss that imbalance.

Assumptions Authored strings and slice counts; UTF-8 and a fixed character tokenizer. Cross-tokenizer loss values require further normalization and do not directly rank language ability.

Engineering example

Distinctions and tradeoffs

It is a mistake to conclude: Both slices retained 76.36%.

Aggregate scores simplify comparison but can conceal minority-source failures. Slice metrics expose them at the cost of smaller, noisier denominators.

Make the effective training distribution inspectable rather than treating tokens as a homogeneous commodity. Compare source slices under explicit byte, token and task denominators.

Specifications and interfaces

Learning mode: worked example
The executable decoder has 423 parameters; worked extensions do not imply an additional simulator.
Worked contract: Compare source slices under explicit byte, token and task denominators.
Authored strings and slice counts; UTF-8 and a fixed character tokenizer. Cross-tokenizer loss values require further normalization and do not directly rank language ability.

Interfaces

Inputs and state Input: Record the exact UTF-8 text and tokenizer version for each source slice.
Result and check Output: Compare source slices under explicit byte, token and task denominators.

Check your understanding

Which observation is concealed by the overall 76.36% retention?

Three UTF-8 bytes always mean three tokens.
Slice B retained only 40% of its documents.
Both slices retained 76.36%.

Answer and reasoning

Slice B retained only 40% of its documents.

Slice B retained only 40% of its documents. Authored strings and slice counts; UTF-8 and a fixed character tokenizer. Cross-tokenizer loss values require further normalization and do not directly rank language ability.

Evidence and scope

Mode: worked example. Authored strings and slice counts; UTF-8 and a fixed character tokenizer. Cross-tokenizer loss values require further normalization and do not directly rank language ability. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.

Model lifecycle studio: worked example. Authored strings and slice counts; UTF-8 and a fixed character tokenizer. Cross-tokenizer loss values require further normalization and do not directly rank language ability.