{
  "schemaVersion": 1,
  "id": "atlas-lesson-low-rank-adaptation",
  "title": "Frozen base weights and trainable adapters",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Multiply a low-rank update and count trainable factors without confusing them with the training objective.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/low-rank-adaptation/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "model-lifecycle-posttraining",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Mode: worked example. Rank one, d=4, k=6 and alpha/r=1 in the example. Biases and any additional adapted matrices are excluded. Worked-only parameterization. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/low-rank-adaptation-3726097baa621ba3.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#low-rank-adaptation",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "d423b4b35c014e771cc64b17500150e20a2353c95b14a81382f564559c15c5c8",
  "contentLength": 6329,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-low-rank-adaptation.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-low-rank-adaptation.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Frozen base weights and trainable adapters",
      "start": 45,
      "end": 485
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 485,
      "end": 1452
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1452,
      "end": 1480
    },
    {
      "id": "section-4",
      "title": "Frozen base weights and trainable adapters · worked relation",
      "start": 1480,
      "end": 2540
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2540,
      "end": 2915
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2915,
      "end": 3377
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3377,
      "end": 3804
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3804,
      "end": 4040
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 4040,
      "end": 4069
    },
    {
      "id": "section-10",
      "title": "What does the number 10 count?",
      "start": 4069,
      "end": 4516
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4516,
      "end": 5079
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5079,
      "end": 5384
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 5384,
      "end": 5407
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 5407,
      "end": 5749
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 5749,
      "end": 6329
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#low-rank-adaptation"
    },
    {
      "title": "LoRA: Low-Rank Adaptation of Large Language Models",
      "url": "https://arxiv.org/html/2106.09685v2"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/low-rank-adaptation-3726097baa621ba3.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Linear maps, rank and retained directions",
      "url": "https://semiaifoundry.com/learn/atlas/linear-map-rank-subspace/"
    },
    {
      "title": "Loss, gradients & optimizer",
      "url": "https://semiaifoundry.com/learn/atlas/loss-optimizer/"
    },
    {
      "title": "Sharded matrix multiplication and result ownership",
      "url": "https://semiaifoundry.com/learn/atlas/sharded-matmul-ownership/"
    },
    {
      "title": "Instruction tuning and assistant-only loss",
      "url": "https://semiaifoundry.com/learn/atlas/sft-chat-loss-masks/"
    },
    {
      "title": "Learn a reward from pairwise choices",
      "url": "https://semiaifoundry.com/learn/atlas/preference-reward-model/"
    },
    {
      "title": "DPO policy/reference likelihood ratios",
      "url": "https://semiaifoundry.com/learn/atlas/direct-preference-objective/"
    },
    {
      "title": "Rollouts, advantages and PPO constraints",
      "url": "https://semiaifoundry.com/learn/atlas/policy-gradient-kl-control/"
    },
    {
      "title": "Verifiers, grouped advantages and RLVR",
      "url": "https://semiaifoundry.com/learn/atlas/verifiable-reward-group-updates/"
    }
  ],
  "recordHash": "063ea7c2734e3f12346bd5278e965e0de585cb9ceb8adde78ccdd6958d40b614",
  "contentFormat": "text/markdown",
  "content": "model-lifecycle-posttraining · Atlas v1.3.0\n\n# Frozen base weights and trainable adapters\n\nMultiply a low-rank update and count trainable factors without confusing them with the training objective.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#low-rank-adaptation)Mathematical, data and behavior space; logical dimensions are not physical distance\n\nLearning objectiveMultiply a low-rank update and count trainable factors without confusing them with the training objective.\n\n## The mechanism\n\nLoRA freezes a base matrix and expresses its update as a product of smaller trainable factors. Rank limits the directions available to that update, rather than declaring which behavior is trained. The same parameterization can be optimized under supervised, preference or other differentiable objectives. The base weights remain part of the forward computation and still need storage unless another representation changes that requirement. Factor count, optimizer state, activation memory and task quality are separate quantities. The 423-parameter lifecycle core currently trains all weights; it does not implement adapters.\n\n1.  Declare the base matrix shape, factor shapes, rank and scale.\n2.  Keep the base fixed while computing the factor product and adding the scaled update.\n3.  Differentiate through the factors only, then verify base entries are unchanged.\n4.  Compare task behavior and trainable state under a fixed objective and budget.\n\n## Science and assumptions\n\n### Frozen base weights and trainable adapters · worked relation\n\n$$\nW=W_0+\\frac{\\alpha}{r}BA,\\quad B\\in\\mathbb R^{d\\times r},\\quad A\\in\\mathbb R^{r\\times k},\\quad P_{adapter}=r(d+k)\n$$\n\nMultiply a low-rank update and count trainable factors without confusing them with the training objective.\n\nSymbols and units\n\nW\\_0\n\nFrozen base matrix\n\ndimensionless\n\nB,A\n\nTrainable low-rank factors\n\ndimensionless\n\nd,k\n\nOutput and input widths\n\nfeatures\n\nr\n\nAdapter rank\n\nrank\n\nalpha\n\nDeclared adapter scale\n\ndimensionless\n\nP\\_adapter\n\nTrainable factor count\n\nparameters\n\nFor W of shape 4×6 and rank 1, B=\\[1,2,0,−1\\]^T and A=\\[1,0,2,0,0,−1\\] have 4+6=10 trainable entries versus 24 base entries. At scale 1, the first row of BA is \\[1,0,2,0,0,−1\\] and the second is twice that. Input e3 (third coordinate one) receives update \\[2,4,0,−2\\]^T. This exact multiplication says nothing by itself about quality after training.\n\nAssumptions Rank one, d=4, k=6 and alpha/r=1 in the example. Biases and any additional adapted matrices are excluded. Worked-only parameterization.\n\n## Engineering example\n\nFor W of shape 4×6 and rank 1, B=\\[1,2,0,−1\\]^T and A=\\[1,0,2,0,0,−1\\] have 4+6=10 trainable entries versus 24 base entries. At scale 1, the first row of BA is \\[1,0,2,0,0,−1\\] and the second is twice that. Input e3 (third coordinate one) receives update \\[2,4,0,−2\\]^T. This exact multiplication says nothing by itself about quality after training.\n\n## Distinctions and tradeoffs\n\nIt is a mistake to conclude: All weights needed to run the base model.\n\nLow-rank factors reduce trainable optimizer state but restrict the update subspace. Lower trainable count is not a guarantee of equivalent task quality.\n\nSeparate training signals and algorithms, then inspect actual policy changes and proxy failures. Multiply a low-rank update and count trainable factors without confusing them with the training objective.\n\n## Specifications and interfaces\n\nLearning mode\n\nworked example\n\nThe executable decoder has 423 parameters; worked extensions do not imply an additional simulator.\n\nWorked contract\n\nMultiply a low-rank update and count trainable factors without confusing them with the training objective.\n\nRank one, d=4, k=6 and alpha/r=1 in the example. Biases and any additional adapted matrices are excluded. Worked-only parameterization.\n\n### Interfaces\n\nInputs and state Input\n\nDeclare the base matrix shape, factor shapes, rank and scale.\n\nResult and check Output\n\nMultiply a low-rank update and count trainable factors without confusing them with the training objective.\n\n## Check your understanding\n\n### What does the number 10 count?\n\n1.  Trainable entries in the two rank-one factors.\n2.  All weights needed to run the base model.\n3.  The number of different objectives that LoRA supports.\n\nAnswer and reasoning\n\nTrainable entries in the two rank-one factors.\n\nTrainable entries in the two rank-one factors. Rank one, d=4, k=6 and alpha/r=1 in the example. Biases and any additional adapted matrices are excluded. Worked-only parameterization.\n\n## Evidence and scope\n\nMode: worked example. Rank one, d=4, k=6 and alpha/r=1 in the example. Biases and any additional adapted matrices are excluded. Worked-only parameterization. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nModel lifecycle studio: worked example. Rank one, d=4, k=6 and alpha/r=1 in the example. Biases and any additional adapted matrices are excluded. Worked-only parameterization.\n\n## Sources\n\n-   [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/html/2106.09685v2)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/low-rank-adaptation-3726097baa621ba3.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Linear maps, rank and retained directions](https://semiaifoundry.com/learn/atlas/linear-map-rank-subspace/)\n-   [Loss, gradients & optimizer](https://semiaifoundry.com/learn/atlas/loss-optimizer/)\n-   [Sharded matrix multiplication and result ownership](https://semiaifoundry.com/learn/atlas/sharded-matmul-ownership/)\n\n### Related lessons\n\n-   [Instruction tuning and assistant-only loss](https://semiaifoundry.com/learn/atlas/sft-chat-loss-masks/)\n-   [Learn a reward from pairwise choices](https://semiaifoundry.com/learn/atlas/preference-reward-model/)\n-   [DPO policy/reference likelihood ratios](https://semiaifoundry.com/learn/atlas/direct-preference-objective/)\n-   [Rollouts, advantages and PPO constraints](https://semiaifoundry.com/learn/atlas/policy-gradient-kl-control/)\n-   [Verifiers, grouped advantages and RLVR](https://semiaifoundry.com/learn/atlas/verifiable-reward-group-updates/)"
}