# Coverage across language and source slices

Canonical: https://semiaifoundry.com/learn/atlas/data-coverage-language-slices/
Revision: 2026-10-07
Rights: https://semiaifoundry.com/ai-atlas/licenses.html

model-lifecycle-data · Atlas v1.3.0

# Coverage across language and source slices

Compare source slices under explicit byte, token and task denominators.

[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#data-coverage-language-slices)Mathematical, data and behavior space; logical dimensions are not physical distance

Learning objectiveCompare source slices under explicit byte, token and task denominators.

## The mechanism

Token count depends on both text and tokenizer. Different scripts and source formats can consume different token budgets for comparable tasks; byte length introduces another encoding-dependent denominator. Neither a low token count nor a low token-normalized loss alone establishes better understanding. A coverage audit preserves source and language labels, inspects retention after filtering, and measures quality on declared task slices. FineWeb’s documented English filtering is a scoped recipe, not evidence that its thresholds fit every language or community.

1.  Record the exact UTF-8 text and tokenizer version for each source slice.
2.  Count retained documents, bytes and tokens separately.
3.  Compute loss and downstream success within slices before aggregating.
4.  Change a mixture only after stating whether the goal is token exposure, task coverage or a quality threshold.

## Science and assumptions

### Coverage across language and source slices · worked relation

$$
r_s=\frac{N_{{\rm kept},s}}{N_{{\rm input},s}},\qquad \bar\ell_s=\frac{\sum_{t\in s}\ell_t}{T_s}
$$

Compare source slices under explicit byte, token and task denominators.

Symbols and units

r\_s

Retention within source slice s

dimensionless

N

Document count

documents

T\_s

Included targets under one tokenizer

tokens

ell\_t

One target negative log likelihood

nats

bar ell\_s

Mean slice loss

nats per token

The authored strings "cat" and "猫" each occupy 3 UTF-8 bytes, but a character tokenizer emits 3 and 1 tokens respectively. This is a coding example, not a comparison of linguistic meaning. In a separate audit, slice A retains 80/100 documents and B retains 4/10. Overall retention is 84/110=76.36%, which conceals B’s 40% retention. Reporting only the overall value would miss that imbalance.

Assumptions Authored strings and slice counts; UTF-8 and a fixed character tokenizer. Cross-tokenizer loss values require further normalization and do not directly rank language ability.

## Engineering example

The authored strings "cat" and "猫" each occupy 3 UTF-8 bytes, but a character tokenizer emits 3 and 1 tokens respectively. This is a coding example, not a comparison of linguistic meaning. In a separate audit, slice A retains 80/100 documents and B retains 4/10. Overall retention is 84/110=76.36%, which conceals B’s 40% retention. Reporting only the overall value would miss that imbalance.

## Distinctions and tradeoffs

It is a mistake to conclude: Both slices retained 76.36%.

Aggregate scores simplify comparison but can conceal minority-source failures. Slice metrics expose them at the cost of smaller, noisier denominators.

Make the effective training distribution inspectable rather than treating tokens as a homogeneous commodity. Compare source slices under explicit byte, token and task denominators.

## Specifications and interfaces

Learning mode

worked example

The executable decoder has 423 parameters; worked extensions do not imply an additional simulator.

Worked contract

Compare source slices under explicit byte, token and task denominators.

Authored strings and slice counts; UTF-8 and a fixed character tokenizer. Cross-tokenizer loss values require further normalization and do not directly rank language ability.

### Interfaces

Inputs and state Input

Record the exact UTF-8 text and tokenizer version for each source slice.

Result and check Output

Compare source slices under explicit byte, token and task denominators.

## Check your understanding

### Which observation is concealed by the overall 76.36% retention?

1.  Three UTF-8 bytes always mean three tokens.
2.  Slice B retained only 40% of its documents.
3.  Both slices retained 76.36%.

Answer and reasoning

Slice B retained only 40% of its documents.

Slice B retained only 40% of its documents. Authored strings and slice counts; UTF-8 and a fixed character tokenizer. Cross-tokenizer loss values require further normalization and do not directly rank language ability.

## Evidence and scope

Mode: worked example. Authored strings and slice counts; UTF-8 and a fixed character tokenizer. Cross-tokenizer loss values require further normalization and do not directly rank language ability. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.

Model lifecycle studio: worked example. Authored strings and slice counts; UTF-8 and a fixed character tokenizer. Cross-tokenizer loss values require further normalization and do not directly rank language ability.

## Sources

-   [The FineWeb Datasets](https://arxiv.org/html/2406.17557v1)
-   [Holistic Evaluation of Language Models](https://arxiv.org/html/2211.09110v1)

[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/data-coverage-language-slices-d2af2cc16e30915c.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)

## Continue exploring

### Foundations

-   [Training data pipeline](https://semiaifoundry.com/learn/atlas/training-data/)
-   [Tokenizer](https://semiaifoundry.com/learn/atlas/tokenizer/)

### Related lessons

-   [Filtering, duplication and contamination](https://semiaifoundry.com/learn/atlas/data-quality-dedup-audit/)
-   [Mixtures, epochs and exposure](https://semiaifoundry.com/learn/atlas/data-mixture-sampling/)
-   [Packing documents and masking valid targets](https://semiaifoundry.com/learn/atlas/sequence-packing-loss-denominators/)
-   [Synthetic supervision and selection loops](https://semiaifoundry.com/learn/atlas/synthetic-data-selection-feedback/)