# Tokenizer

Canonical: https://semiaifoundry.com/learn/atlas/tokenizer/
Revision: 2026-10-07
Rights: https://semiaifoundry.com/ai-atlas/licenses.html

Representation · Atlas v1.3.0

# Tokenizer

Turn text into a sequence of discrete IDs.

[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#tokenizer)Logical structure · mapped onto physical hardware

Learning objectiveCompare token counts for identical text and explain how they affect a sequence budget.

## The mechanism

Before a text model can compute, its input needs a numerical representation. A tokenizer breaks a byte or character stream into pieces and assigns each piece an integer. Common subword methods represent frequent fragments compactly while retaining a way to encode unfamiliar text. The IDs are addresses, not meaning scores. The same visible word may occupy different numbers of tokens depending on the vocabulary and surrounding text.

Advanced tokenization work treats merge learning as a data-dependent training step. Merge counts must come from training text, with a deterministic tie rule and an explicit fallback for unseen pieces. Encoding a held-out string with already learned merges is inference; updating merges from that string changes the tokenizer and leaks held-out information. The lifecycle core intentionally uses a fixed nine-symbol whitespace vocabulary, so it does not simulate BPE fitting.

The corpus experience actually trains a bounded byte-pair tokenizer. Its initial 256-byte inventory is fixed; frequent adjacent pairs are merged using only retained training prompt and response fields. This is separate from the lifecycle model, whose nine-token whitespace vocabulary stays fixed.

1.  Encode the input using a fixed vocabulary and segmentation rule.
2.  Produce integer IDs and any required control tokens.
3.  Use each ID to select a row of the embedding table.
4.  Count adjacent symbol pairs only in the training corpus under the boundary convention.
5.  Select the most frequent pair with a declared tie rule and apply merges consistently.
6.  Freeze the learned tokenizer before encoding development and final text.
7.  Count candidate pairs inside each training field, break equal-frequency ties by numeric IDs, and replace non-overlapping occurrences left to right.
8.  Apply the frozen merge sequence to every split; keep structural boundaries outside content-byte encoding and verify full UTF-8 reconstruction.

## Science and assumptions

### Sequence length is an engineering variable

$$
T=|\operatorname{encode}(\mathrm{text})|,\qquad b_{\mathrm{token}}=\frac{\operatorname{UTF8\_bytes}(\mathrm{text})}{T}
$$

Token count is the length of the encoded token-ID sequence. Bytes per token divides the original string’s UTF-8 byte count by that token count.

The ratio requires T > 0 and is not a semantic-information measure.

Symbols and units

T

Number of produced token IDs

tokens

encode

Specified tokenizer and control-token convention

text → token-ID sequence

text

Identical input string being compared

text

UTF8\_bytes(text)

UTF-8 byte count for that string

bytes

b\_token

Encoding compactness for nonzero T

bytes/token

T is the number of token IDs produced by the chosen tokenizer. For the same text, different vocabularies can change T and therefore the attention workload, cache size, and amount of text that fits a context budget. This ratio describes encoding compactness; it does not measure intelligence or semantic information.

Assumptions Compare identical text, a specified tokenizer version, and the same treatment of control tokens. There is no universal bytes-per-token constant, especially across languages and code.

### Train a tokenizer, then encode unseen text · worked relation

$$
(u^*,v^*)=\arg\max_{(u,v)}\sum_{w\in D_{train}}c_w(u,v)
$$

Distinguish fitting a vocabulary from using a fixed token mapping on held-out input.

Symbols and units

u,v

Adjacent current symbols or subwords

symbols

c\_w

Adjacent-pair occurrences in training item w

count

D\_train

Tokenizer fitting corpus

dataset

Use training words “aa” twice and “ab” once, represented as character sequences with no cross-word pairs. Pair (a,a) has count 2 and (a,b) count 1, so the first merge forms aa. Encoding held-out “aab” yields \[aa,b\] under this toy merge list. Letting held-out “ab” occurrences change the winning pair would change both input IDs and evaluation token denominators.

Assumptions Toy word boundaries and deterministic lexical tie rule; no byte fallback or production BPE trainer is executed by the core.

## Engineering example

Synthetic text has 24 UTF-8 bytes. Tokenizer A returns 6 IDs and tokenizer B returns 8, giving 4 and 3 bytes/token respectively. At fixed per-token cached state, B needs 8/6 ≈ 1.33× as much cache for this text; the ratio is about encoding, not understanding.

Use training words “aa” twice and “ab” once, represented as character sequences with no cross-word pairs. Pair (a,a) has count 2 and (a,b) count 1, so the first merge forms aa. Encoding held-out “aab” yields \[aa,b\] under this toy merge list. Letting held-out “ab” occurrences change the winning pair would change both input IDs and evaluation token denominators.

In the field abab, the pair ab occurs twice. One merge maps its two bytes to a new symbol, and encoding becomes two copies of that symbol. In aaa, the pair aa has two overlapping counts but only one non-overlapping replacement. These counts describe different stages of the algorithm.

## Distinctions and tradeoffs

An integer token ID is not a learned semantic vector; the embedding lookup produces that vector.

A larger vocabulary can shorten sequences while increasing embedding storage. Different languages and encodings have different token efficiency.

Token count affects context capacity, attention work, and serving memory.

## Specifications and interfaces

Unit

Token ID

Vocabulary-specific integer, not a voltage.

Granularity

Often subword

A token is not necessarily one word.

### Interfaces

Text Input

Bytes or characters

Embedding Output

Sequence of vocabulary IDs

## Check your understanding

### Does one token always equal one word?

1.  Yes, for all languages
2.  No; it can be a fragment, punctuation, or other encoded piece
3.  Only during training

Answer and reasoning

No; it can be a fragment, punctuation, or other encoded piece

Tokenization depends on the tokenizer and text, not a universal word boundary.

### Which operation would leak held-out information into the tokenizer?

1.  Recording the frozen vocabulary identity.
2.  Counting held-out pairs when choosing the next merge.
3.  Applying already frozen merges to held-out text.

Answer and reasoning

Counting held-out pairs when choosing the next merge.

Counting held-out pairs when choosing the next merge. Toy word boundaries and deterministic lexical tie rule; no byte fallback or production BPE trainer is executed by the core.

### Why do two adjacent aa counts in aaa produce only one replacement during one left-to-right merge pass?

1.  Overlapping occurrences share a byte and cannot both be replaced simultaneously
2.  The tokenizer ignores the final word
3.  One occurrence belongs to evaluation

Answer and reasoning

Overlapping occurrences share a byte and cannot both be replaced simultaneously

Candidate counting scans adjacent positions. A replacement consumes both members of the pair, so the next replacement cannot reuse the consumed middle symbol.

## Historical context

### 2015 · Subword BPE enters neural translation

Sennrich, Haddow, and Birch adapted byte-pair segmentation to represent rare words as subword sequences. The preprint appeared in 2015; the conference paper followed at ACL 2016.

[Sennrich et al. — Neural Machine Translation of Rare Words with Subword Units](https://arxiv.org/abs/1508.07909)

## Evidence and scope

Counts are invented to illustrate the lesson’s identity, not outputs of a named tokenizer. Real comparisons require the identical text, tokenizer version and control-token rules; languages and code do not share a universal bytes/token ratio.

Mode: worked example. Toy word boundaries and deterministic lexical tie rule; no byte fallback or production BPE trainer is executed by the core. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.

The implemented whole-field UTF-8 byte variant has no word pre-tokenizer and can merge across spaces within a field. Smaller token count is not by itself evidence of better learning. BPE IDs never enter the fixed nine-token lifecycle decoder.

No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.

## Sources

-   [OpenAI — tiktoken](https://github.com/openai/tiktoken)
-   [Sennrich et al. — Subword Units](https://arxiv.org/abs/1508.07909)
-   [Neural Machine Translation of Rare Words with Subword Units](https://aclanthology.org/P16-1162.pdf)
-   [Sennrich et al. · BPE construction §3.2](https://arxiv.org/html/1508.07909v5)

[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/tokenizer-76457284d6a4e6f9.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)

## Continue exploring

### Related lessons

-   [Embedding table](https://semiaifoundry.com/learn/atlas/embedding/)
-   [Key–value cache](https://semiaifoundry.com/learn/atlas/kv-cache/)
-   [Filtering, duplication and contamination](https://semiaifoundry.com/learn/atlas/data-quality-dedup-audit/)
-   [Mixtures, epochs and exposure](https://semiaifoundry.com/learn/atlas/data-mixture-sampling/)
-   [Packing documents and masking valid targets](https://semiaifoundry.com/learn/atlas/sequence-packing-loss-denominators/)
-   [Synthetic supervision and selection loops](https://semiaifoundry.com/learn/atlas/synthetic-data-selection-feedback/)
-   [Coverage across language and source slices](https://semiaifoundry.com/learn/atlas/data-coverage-language-slices/)
-   [Training data pipeline](https://semiaifoundry.com/learn/atlas/training-data/)
-   [Loss, gradients & optimizer](https://semiaifoundry.com/learn/atlas/loss-optimizer/)