{"id":"data-coverage-language-slices","name":"Coverage across language and source slices","shortName":"Coverage across language and source slices","role":"Compare source slices under explicit byte, token and task denominators.","description":"Token count depends on both text and tokenizer. Different scripts and source formats can consume different token budgets for comparable tasks; byte length introduces another encoding-dependent denominator. Neither a low token count nor a low token-normalized loss alone establishes better understanding. A coverage audit preserves source and language labels, inspects retention after filtering, and measures quality on declared task slices. FineWeb’s documented English filtering is a scoped recipe, not evidence that its thresholds fit every language or community.","mechanism":["Record the exact UTF-8 text and tokenizer version for each source slice.","Count retained documents, bytes and tokens separately.","Compute loss and downstream success within slices before aggregating.","Change a mixture only after stating whether the goal is token exposure, task coverage or a quality threshold."],"learningObjective":"Compare source slices under explicit byte, token and task denominators.","engineeringExample":"The authored strings \"cat\" and \"猫\" each occupy 3 UTF-8 bytes, but a character tokenizer emits 3 and 1 tokens respectively. This is a coding example, not a comparison of linguistic meaning. In a separate audit, slice A retains 80/100 documents and B retains 4/10. Overall retention is 84/110=76.36%, which conceals B’s 40% retention. Reporting only the overall value would miss that imbalance.","evidenceNotes":"Mode: worked example. Authored strings and slice counts; UTF-8 and a fixed character tokenizer. Cross-tokenizer loss values require further normalization and do not directly rank language ability. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","specs":[{"label":"Learning mode","value":"worked example","context":"The executable decoder has 423 parameters; worked extensions do not imply an additional simulator."},{"label":"Worked contract","value":"Compare source slices under explicit byte, token and task denominators.","context":"Authored strings and slice counts; UTF-8 and a fixed character tokenizer. Cross-tokenizer loss values require further normalization and do not directly rank language ability."}],"interfaces":[{"name":"Inputs and state","direction":"in","signal":"Record the exact UTF-8 text and tokenizer version for each source slice."},{"name":"Result and check","direction":"out","signal":"Compare source slices under explicit byte, token and task denominators."}],"tradeoff":"Aggregate scores simplify comparison but can conceal minority-source failures. Slice metrics expose them at the cost of smaller, noisier denominators.","whyAI":"Make the effective training distribution inspectable rather than treating tokens as a homogeneous commodity. Compare source slices under explicit byte, token and task denominators.","misconception":"It is a mistake to conclude: Both slices retained 76.36%.","check":{"question":"Which observation is concealed by the overall 76.36% retention?","options":["Three UTF-8 bytes always mean three tokens.","Slice B retained only 40% of its documents.","Both slices retained 76.36%."],"answer":1,"explanation":"Slice B retained only 40% of its documents. Authored strings and slice counts; UTF-8 and a fixed character tokenizer. Cross-tokenizer loss values require further normalization and do not directly rank language ability."},"prerequisites":["training-data","tokenizer"],"related":["training-data","tokenizer","data-quality-dedup-audit","data-mixture-sampling","sequence-packing-loss-denominators","synthetic-data-selection-feedback"],"sources":[{"title":"The FineWeb Datasets","url":"https://arxiv.org/html/2406.17557v1"},{"title":"Holistic Evaluation of Language Models","url":"https://arxiv.org/html/2211.09110v1"}],"level":"model","scale":"Mathematical, data and behavior space; logical dimensions are not physical distance","kind":"model","branch":"model-lifecycle-data","subbranch":"Coverage across language and source slices","workstream":"W11","visualFamily":"mathematical","labId":"library","labScope":"Model lifecycle studio: worked example. Authored strings and slice counts; UTF-8 and a fixed character tokenizer. Cross-tokenizer loss values require further normalization and do not directly rank language ability.","science":[{"title":"Coverage across language and source slices · worked relation","equation":"r_s=\\frac{N_{{\\rm kept},s}}{N_{{\\rm input},s}},\\qquad \\bar\\ell_s=\\frac{\\sum_{t\\in s}\\ell_t}{T_s}","explanation":"The authored strings \"cat\" and \"猫\" each occupy 3 UTF-8 bytes, but a character tokenizer emits 3 and 1 tokens respectively. This is a coding example, not a comparison of linguistic meaning. In a separate audit, slice A retains 80/100 documents and B retains 4/10. Overall retention is 84/110=76.36%, which conceals B’s 40% retention. Reporting only the overall value would miss that imbalance.","assumptions":"Authored strings and slice counts; UTF-8 and a fixed character tokenizer. Cross-tokenizer loss values require further normalization and do not directly rank language ability."}],"history":[]}