{
  "schemaVersion": 1,
  "id": "atlas-lesson-tokenizer",
  "title": "Tokenizer",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Turn text into a sequence of discrete IDs.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/tokenizer/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Representation",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Counts are invented to illustrate the lesson’s identity, not outputs of a named tokenizer. Real comparisons require the identical text, tokenizer version and control-token rules; languages and code do not share a universal bytes/token ratio.\n\nMode: worked example. Toy word boundaries and deterministic lexical tie rule; no byte fallback or production BPE trainer is executed by the core. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nThe implemented whole-field UTF-8 byte variant has no word pre-tokenizer and can merge across spaces within a field. Smaller token count is not by itself evidence of better learning. BPE IDs never enter the fixed nine-token lifecycle decoder.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/tokenizer-76457284d6a4e6f9.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#tokenizer",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "3f3cca881c5cb7e82c145c2b7796a923c05df7bdaab6a2037034085b0e47bf71",
  "contentLength": 10165,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-tokenizer.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-tokenizer.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Tokenizer",
      "start": 31,
      "end": 310
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 310,
      "end": 2278
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 2278,
      "end": 2306
    },
    {
      "id": "section-4",
      "title": "Sequence length is an engineering variable",
      "start": 2306,
      "end": 3521
    },
    {
      "id": "section-5",
      "title": "Train a tokenizer, then encode unseen text · worked relation",
      "start": 3521,
      "end": 4414
    },
    {
      "id": "section-6",
      "title": "Engineering example",
      "start": 4414,
      "end": 5350
    },
    {
      "id": "section-7",
      "title": "Distinctions and tradeoffs",
      "start": 5350,
      "end": 5700
    },
    {
      "id": "section-8",
      "title": "Specifications and interfaces",
      "start": 5700,
      "end": 5861
    },
    {
      "id": "section-9",
      "title": "Interfaces",
      "start": 5861,
      "end": 5956
    },
    {
      "id": "section-10",
      "title": "Check your understanding",
      "start": 5956,
      "end": 5985
    },
    {
      "id": "section-11",
      "title": "Does one token always equal one word?",
      "start": 5985,
      "end": 6312
    },
    {
      "id": "section-12",
      "title": "Which operation would leak held-out information into the tokenizer?",
      "start": 6312,
      "end": 6799
    },
    {
      "id": "section-13",
      "title": "Why do two adjacent aa counts in aaa produce only one replacement during one left-to-right merge pass?",
      "start": 6799,
      "end": 7337
    },
    {
      "id": "section-14",
      "title": "Historical context",
      "start": 7337,
      "end": 7360
    },
    {
      "id": "section-15",
      "title": "2015 · Subword BPE enters neural translation",
      "start": 7360,
      "end": 7705
    },
    {
      "id": "section-16",
      "title": "Evidence and scope",
      "start": 7705,
      "end": 8710
    },
    {
      "id": "section-17",
      "title": "Sources",
      "start": 8710,
      "end": 9231
    },
    {
      "id": "section-18",
      "title": "Continue exploring",
      "start": 9231,
      "end": 9254
    },
    {
      "id": "section-19",
      "title": "Related lessons",
      "start": 9254,
      "end": 10165
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#tokenizer"
    },
    {
      "title": "Sennrich et al. — Neural Machine Translation of Rare Words with Subword Units",
      "url": "https://arxiv.org/abs/1508.07909"
    },
    {
      "title": "OpenAI — tiktoken",
      "url": "https://github.com/openai/tiktoken"
    },
    {
      "title": "Neural Machine Translation of Rare Words with Subword Units",
      "url": "https://aclanthology.org/P16-1162.pdf"
    },
    {
      "title": "Sennrich et al. · BPE construction §3.2",
      "url": "https://arxiv.org/html/1508.07909v5"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/tokenizer-76457284d6a4e6f9.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Embedding table",
      "url": "https://semiaifoundry.com/learn/atlas/embedding/"
    },
    {
      "title": "Key–value cache",
      "url": "https://semiaifoundry.com/learn/atlas/kv-cache/"
    },
    {
      "title": "Filtering, duplication and contamination",
      "url": "https://semiaifoundry.com/learn/atlas/data-quality-dedup-audit/"
    },
    {
      "title": "Mixtures, epochs and exposure",
      "url": "https://semiaifoundry.com/learn/atlas/data-mixture-sampling/"
    },
    {
      "title": "Packing documents and masking valid targets",
      "url": "https://semiaifoundry.com/learn/atlas/sequence-packing-loss-denominators/"
    },
    {
      "title": "Synthetic supervision and selection loops",
      "url": "https://semiaifoundry.com/learn/atlas/synthetic-data-selection-feedback/"
    },
    {
      "title": "Coverage across language and source slices",
      "url": "https://semiaifoundry.com/learn/atlas/data-coverage-language-slices/"
    },
    {
      "title": "Training data pipeline",
      "url": "https://semiaifoundry.com/learn/atlas/training-data/"
    },
    {
      "title": "Loss, gradients & optimizer",
      "url": "https://semiaifoundry.com/learn/atlas/loss-optimizer/"
    }
  ],
  "recordHash": "721bb7086ad9e97bd55823c8c1e06aff03696cf800bbdae8f3907b582de89e24",
  "contentFormat": "text/markdown",
  "content": "Representation · Atlas v1.3.0\n\n# Tokenizer\n\nTurn text into a sequence of discrete IDs.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#tokenizer)Logical structure · mapped onto physical hardware\n\nLearning objectiveCompare token counts for identical text and explain how they affect a sequence budget.\n\n## The mechanism\n\nBefore a text model can compute, its input needs a numerical representation. A tokenizer breaks a byte or character stream into pieces and assigns each piece an integer. Common subword methods represent frequent fragments compactly while retaining a way to encode unfamiliar text. The IDs are addresses, not meaning scores. The same visible word may occupy different numbers of tokens depending on the vocabulary and surrounding text.\n\nAdvanced tokenization work treats merge learning as a data-dependent training step. Merge counts must come from training text, with a deterministic tie rule and an explicit fallback for unseen pieces. Encoding a held-out string with already learned merges is inference; updating merges from that string changes the tokenizer and leaks held-out information. The lifecycle core intentionally uses a fixed nine-symbol whitespace vocabulary, so it does not simulate BPE fitting.\n\nThe corpus experience actually trains a bounded byte-pair tokenizer. Its initial 256-byte inventory is fixed; frequent adjacent pairs are merged using only retained training prompt and response fields. This is separate from the lifecycle model, whose nine-token whitespace vocabulary stays fixed.\n\n1.  Encode the input using a fixed vocabulary and segmentation rule.\n2.  Produce integer IDs and any required control tokens.\n3.  Use each ID to select a row of the embedding table.\n4.  Count adjacent symbol pairs only in the training corpus under the boundary convention.\n5.  Select the most frequent pair with a declared tie rule and apply merges consistently.\n6.  Freeze the learned tokenizer before encoding development and final text.\n7.  Count candidate pairs inside each training field, break equal-frequency ties by numeric IDs, and replace non-overlapping occurrences left to right.\n8.  Apply the frozen merge sequence to every split; keep structural boundaries outside content-byte encoding and verify full UTF-8 reconstruction.\n\n## Science and assumptions\n\n### Sequence length is an engineering variable\n\n$$\nT=|\\operatorname{encode}(\\mathrm{text})|,\\qquad b_{\\mathrm{token}}=\\frac{\\operatorname{UTF8\\_bytes}(\\mathrm{text})}{T}\n$$\n\nToken count is the length of the encoded token-ID sequence. Bytes per token divides the original string’s UTF-8 byte count by that token count.\n\nThe ratio requires T > 0 and is not a semantic-information measure.\n\nSymbols and units\n\nT\n\nNumber of produced token IDs\n\ntokens\n\nencode\n\nSpecified tokenizer and control-token convention\n\ntext → token-ID sequence\n\ntext\n\nIdentical input string being compared\n\ntext\n\nUTF8\\_bytes(text)\n\nUTF-8 byte count for that string\n\nbytes\n\nb\\_token\n\nEncoding compactness for nonzero T\n\nbytes/token\n\nT is the number of token IDs produced by the chosen tokenizer. For the same text, different vocabularies can change T and therefore the attention workload, cache size, and amount of text that fits a context budget. This ratio describes encoding compactness; it does not measure intelligence or semantic information.\n\nAssumptions Compare identical text, a specified tokenizer version, and the same treatment of control tokens. There is no universal bytes-per-token constant, especially across languages and code.\n\n### Train a tokenizer, then encode unseen text · worked relation\n\n$$\n(u^*,v^*)=\\arg\\max_{(u,v)}\\sum_{w\\in D_{train}}c_w(u,v)\n$$\n\nDistinguish fitting a vocabulary from using a fixed token mapping on held-out input.\n\nSymbols and units\n\nu,v\n\nAdjacent current symbols or subwords\n\nsymbols\n\nc\\_w\n\nAdjacent-pair occurrences in training item w\n\ncount\n\nD\\_train\n\nTokenizer fitting corpus\n\ndataset\n\nUse training words “aa” twice and “ab” once, represented as character sequences with no cross-word pairs. Pair (a,a) has count 2 and (a,b) count 1, so the first merge forms aa. Encoding held-out “aab” yields \\[aa,b\\] under this toy merge list. Letting held-out “ab” occurrences change the winning pair would change both input IDs and evaluation token denominators.\n\nAssumptions Toy word boundaries and deterministic lexical tie rule; no byte fallback or production BPE trainer is executed by the core.\n\n## Engineering example\n\nSynthetic text has 24 UTF-8 bytes. Tokenizer A returns 6 IDs and tokenizer B returns 8, giving 4 and 3 bytes/token respectively. At fixed per-token cached state, B needs 8/6 ≈ 1.33× as much cache for this text; the ratio is about encoding, not understanding.\n\nUse training words “aa” twice and “ab” once, represented as character sequences with no cross-word pairs. Pair (a,a) has count 2 and (a,b) count 1, so the first merge forms aa. Encoding held-out “aab” yields \\[aa,b\\] under this toy merge list. Letting held-out “ab” occurrences change the winning pair would change both input IDs and evaluation token denominators.\n\nIn the field abab, the pair ab occurs twice. One merge maps its two bytes to a new symbol, and encoding becomes two copies of that symbol. In aaa, the pair aa has two overlapping counts but only one non-overlapping replacement. These counts describe different stages of the algorithm.\n\n## Distinctions and tradeoffs\n\nAn integer token ID is not a learned semantic vector; the embedding lookup produces that vector.\n\nA larger vocabulary can shorten sequences while increasing embedding storage. Different languages and encodings have different token efficiency.\n\nToken count affects context capacity, attention work, and serving memory.\n\n## Specifications and interfaces\n\nUnit\n\nToken ID\n\nVocabulary-specific integer, not a voltage.\n\nGranularity\n\nOften subword\n\nA token is not necessarily one word.\n\n### Interfaces\n\nText Input\n\nBytes or characters\n\nEmbedding Output\n\nSequence of vocabulary IDs\n\n## Check your understanding\n\n### Does one token always equal one word?\n\n1.  Yes, for all languages\n2.  No; it can be a fragment, punctuation, or other encoded piece\n3.  Only during training\n\nAnswer and reasoning\n\nNo; it can be a fragment, punctuation, or other encoded piece\n\nTokenization depends on the tokenizer and text, not a universal word boundary.\n\n### Which operation would leak held-out information into the tokenizer?\n\n1.  Recording the frozen vocabulary identity.\n2.  Counting held-out pairs when choosing the next merge.\n3.  Applying already frozen merges to held-out text.\n\nAnswer and reasoning\n\nCounting held-out pairs when choosing the next merge.\n\nCounting held-out pairs when choosing the next merge. Toy word boundaries and deterministic lexical tie rule; no byte fallback or production BPE trainer is executed by the core.\n\n### Why do two adjacent aa counts in aaa produce only one replacement during one left-to-right merge pass?\n\n1.  Overlapping occurrences share a byte and cannot both be replaced simultaneously\n2.  The tokenizer ignores the final word\n3.  One occurrence belongs to evaluation\n\nAnswer and reasoning\n\nOverlapping occurrences share a byte and cannot both be replaced simultaneously\n\nCandidate counting scans adjacent positions. A replacement consumes both members of the pair, so the next replacement cannot reuse the consumed middle symbol.\n\n## Historical context\n\n### 2015 · Subword BPE enters neural translation\n\nSennrich, Haddow, and Birch adapted byte-pair segmentation to represent rare words as subword sequences. The preprint appeared in 2015; the conference paper followed at ACL 2016.\n\n[Sennrich et al. — Neural Machine Translation of Rare Words with Subword Units](https://arxiv.org/abs/1508.07909)\n\n## Evidence and scope\n\nCounts are invented to illustrate the lesson’s identity, not outputs of a named tokenizer. Real comparisons require the identical text, tokenizer version and control-token rules; languages and code do not share a universal bytes/token ratio.\n\nMode: worked example. Toy word boundaries and deterministic lexical tie rule; no byte fallback or production BPE trainer is executed by the core. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nThe implemented whole-field UTF-8 byte variant has no word pre-tokenizer and can merge across spaces within a field. Smaller token count is not by itself evidence of better learning. BPE IDs never enter the fixed nine-token lifecycle decoder.\n\nNo current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.\n\n## Sources\n\n-   [OpenAI — tiktoken](https://github.com/openai/tiktoken)\n-   [Sennrich et al. — Subword Units](https://arxiv.org/abs/1508.07909)\n-   [Neural Machine Translation of Rare Words with Subword Units](https://aclanthology.org/P16-1162.pdf)\n-   [Sennrich et al. · BPE construction §3.2](https://arxiv.org/html/1508.07909v5)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/tokenizer-76457284d6a4e6f9.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Related lessons\n\n-   [Embedding table](https://semiaifoundry.com/learn/atlas/embedding/)\n-   [Key–value cache](https://semiaifoundry.com/learn/atlas/kv-cache/)\n-   [Filtering, duplication and contamination](https://semiaifoundry.com/learn/atlas/data-quality-dedup-audit/)\n-   [Mixtures, epochs and exposure](https://semiaifoundry.com/learn/atlas/data-mixture-sampling/)\n-   [Packing documents and masking valid targets](https://semiaifoundry.com/learn/atlas/sequence-packing-loss-denominators/)\n-   [Synthetic supervision and selection loops](https://semiaifoundry.com/learn/atlas/synthetic-data-selection-feedback/)\n-   [Coverage across language and source slices](https://semiaifoundry.com/learn/atlas/data-coverage-language-slices/)\n-   [Training data pipeline](https://semiaifoundry.com/learn/atlas/training-data/)\n-   [Loss, gradients & optimizer](https://semiaifoundry.com/learn/atlas/loss-optimizer/)"
}