{
  "schemaVersion": 1,
  "id": "atlas-lesson-tiny-causal-decoder",
  "title": "Assemble an initialized causal language model",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Join tensor shapes, causal probabilities, gradients and optimizer state into one reproducible trained model.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/tiny-causal-decoder/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "model-lifecycle-base-model",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Mode: executable core. One head, separate input/output weights, two feed-forward biases, output bias, fixed sinusoidal positions, gain-free RMS normalization, tanh activation and binary64 arithmetic. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/tiny-causal-decoder-bb9fed5742cbbfda.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#tiny-causal-decoder",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "7f5242af593e46ac05b0047f46f74ef9cc51641af83c8f4e37ecbbd1cbb29818",
  "contentLength": 6700,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-tiny-causal-decoder.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-tiny-causal-decoder.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Assemble an initialized causal language model",
      "start": 43,
      "end": 490
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 490,
      "end": 1592
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1592,
      "end": 1620
    },
    {
      "id": "section-4",
      "title": "Assemble an initialized causal language model · worked relation",
      "start": 1620,
      "end": 2668
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2668,
      "end": 3129
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 3129,
      "end": 3593
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3593,
      "end": 4064
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 4064,
      "end": 4352
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 4352,
      "end": 4381
    },
    {
      "id": "section-10",
      "title": "Why are the fixed positional vectors absent from the 423 count?",
      "start": 4381,
      "end": 4989
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4989,
      "end": 5636
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5636,
      "end": 6153
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 6153,
      "end": 6176
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 6176,
      "end": 6700
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#tiny-causal-decoder"
    },
    {
      "title": "Attention Is All You Need",
      "url": "https://arxiv.org/html/1706.03762v7"
    },
    {
      "title": "2 OLMo 2 Furious",
      "url": "https://arxiv.org/html/2501.00656v1"
    },
    {
      "title": "Root Mean Square Layer Normalization",
      "url": "https://arxiv.org/html/1910.07467v1"
    },
    {
      "title": "PyTorch Adam algorithm",
      "url": "https://docs.pytorch.org/docs/2.14/generated/torch.optim.Adam.html"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/tiny-causal-decoder-bb9fed5742cbbfda.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Embedding table",
      "url": "https://semiaifoundry.com/learn/atlas/embedding/"
    },
    {
      "title": "Self-attention",
      "url": "https://semiaifoundry.com/learn/atlas/attention/"
    },
    {
      "title": "Feed-forward network",
      "url": "https://semiaifoundry.com/learn/atlas/feed-forward/"
    },
    {
      "title": "Residual stream & normalization",
      "url": "https://semiaifoundry.com/learn/atlas/residual-normalization/"
    },
    {
      "title": "Output head & decoding",
      "url": "https://semiaifoundry.com/learn/atlas/output-head/"
    },
    {
      "title": "A differentiable computation graph",
      "url": "https://semiaifoundry.com/learn/atlas/reverse-mode-autodiff/"
    }
  ],
  "recordHash": "0f44f5b5e0377ccd43cb260b682bd10f154b492123e4c968379b2398f6a1ee55",
  "contentFormat": "text/markdown",
  "content": "model-lifecycle-base-model · Atlas v1.3.0\n\n# Assemble an initialized causal language model\n\nJoin tensor shapes, causal probabilities, gradients and optimizer state into one reproducible trained model.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#tiny-causal-decoder)Mathematical, data and behavior space; logical dimensions are not physical distance\n\nLearning objectiveJoin tensor shapes, causal probabilities, gradients and optimizer state into one reproducible trained model.\n\n## The mechanism\n\nThe bounded lifecycle model is a complete causal decoder rather than a collection of supplied-gradient formulas. A nine-symbol lookup feeds one width-six attention head and a width-twelve nonlinear block. Residual paths carry activations around each sublayer; normalization controls their local scale. A separate output head predicts nine next-token probabilities. Every learned coefficient participates in the same reverse-mode graph. Fixed positions, masks and tokenizer are part of the computation but are not trainable parameters. The exact small architecture is a teaching choice, so its behavior must be measured on its own documented symbol task.\n\n1.  Initialize all learned weights from the saved seed; keep fixed positions and normalization convention separate.\n2.  Trace embeddings through causal Q/K/V mixing, residuals, tanh features and output logits.\n3.  Form target losses, backpropagate through all eleven tensor groups, and apply the selected optimizer.\n4.  Compare reserved copy and swap cases with a train-fitted baseline and preserve the entire experiment state.\n\n## Science and assumptions\n\n### Assemble an initialized causal language model · worked relation\n\n$$\nP=Vd+4d^2+(dh+h+hd+d)+(dV+V)=423\n$$\n\nJoin tensor shapes, causal probabilities, gradients and optimizer state into one reproducible trained model.\n\nSymbols and units\n\nP\n\nLearned scalar coefficients in this declared decoder\n\nparameters\n\nV\n\nVocabulary size, 9\n\ntokens\n\nd\n\nHidden and single-head width, 6\n\nfeatures\n\nh\n\nFeed-forward width, 12\n\nfeatures\n\nThe exact parameter ledger is 9×6 embedding + 4×6×6 attention projections + (6×12+12+12×6+6) feed-forward + (6×9+9) output =423. Fixed positions and gain-free RMS normalization add no parameters. At binary64 payload size the weights occupy 423×8=3384 bytes, excluding runtime objects and activations. In the default tested run, 320 actual updates produce 8/8 copy development answers; this finite toy result is not language competence.\n\nAssumptions One head, separate input/output weights, two feed-forward biases, output bias, fixed sinusoidal positions, gain-free RMS normalization, tanh activation and binary64 arithmetic.\n\n## Engineering example\n\nThe exact parameter ledger is 9×6 embedding + 4×6×6 attention projections + (6×12+12+12×6+6) feed-forward + (6×9+9) output =423. Fixed positions and gain-free RMS normalization add no parameters. At binary64 payload size the weights occupy 423×8=3384 bytes, excluding runtime objects and activations. In the default tested run, 320 actual updates produce 8/8 copy development answers; this finite toy result is not language competence.\n\n## Distinctions and tradeoffs\n\nIt is a mistake to conclude: They are ignored by causal attention.\n\nA small model permits complete inspection and exact tests. Its narrow vocabulary and task leave most real language phenomena outside the evidence.\n\nJoin existing forward mechanisms into an initialized model with real gradients and measurable predictions. Join tensor shapes, causal probabilities, gradients and optimizer state into one reproducible trained model.\n\n## Specifications and interfaces\n\nLearning mode\n\nexecutable core\n\nThe executable decoder has 423 parameters; worked extensions do not imply an additional simulator.\n\nWorked contract\n\nJoin tensor shapes, causal probabilities, gradients and optimizer state into one reproducible trained model.\n\nOne head, separate input/output weights, two feed-forward biases, output bias, fixed sinusoidal positions, gain-free RMS normalization, tanh activation and binary64 arithmetic.\n\n### Interfaces\n\nInputs and state Input\n\nInitialize all learned weights from the saved seed; keep fixed positions and normalization convention separate.\n\nResult and check Output\n\nJoin tensor shapes, causal probabilities, gradients and optimizer state into one reproducible trained model.\n\n## Check your understanding\n\n### Why are the fixed positional vectors absent from the 423 count?\n\n1.  They affect the forward computation but are not updated as learned parameters.\n2.  They are ignored by causal attention.\n3.  All vectors in a model are necessarily trainable.\n\nAnswer and reasoning\n\nThey affect the forward computation but are not updated as learned parameters.\n\nThey affect the forward computation but are not updated as learned parameters. One head, separate input/output weights, two feed-forward biases, output bias, fixed sinusoidal positions, gain-free RMS normalization, tanh activation and binary64 arithmetic.\n\n## Evidence and scope\n\nMode: executable core. One head, separate input/output weights, two feed-forward biases, output bias, fixed sinusoidal positions, gain-free RMS normalization, tanh activation and binary64 arithmetic. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nModel lifecycle studio: executable core. One head, separate input/output weights, two feed-forward biases, output bias, fixed sinusoidal positions, gain-free RMS normalization, tanh activation and binary64 arithmetic.\n\n## Sources\n\n-   [Attention Is All You Need](https://arxiv.org/html/1706.03762v7)\n-   [2 OLMo 2 Furious](https://arxiv.org/html/2501.00656v1)\n-   [Root Mean Square Layer Normalization](https://arxiv.org/html/1910.07467v1)\n-   [PyTorch Adam algorithm](https://docs.pytorch.org/docs/2.14/generated/torch.optim.Adam.html)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/tiny-causal-decoder-bb9fed5742cbbfda.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Embedding table](https://semiaifoundry.com/learn/atlas/embedding/)\n-   [Self-attention](https://semiaifoundry.com/learn/atlas/attention/)\n-   [Feed-forward network](https://semiaifoundry.com/learn/atlas/feed-forward/)\n-   [Residual stream & normalization](https://semiaifoundry.com/learn/atlas/residual-normalization/)\n-   [Output head & decoding](https://semiaifoundry.com/learn/atlas/output-head/)\n-   [A differentiable computation graph](https://semiaifoundry.com/learn/atlas/reverse-mode-autodiff/)"
}