# Frozen base weights and trainable adapters Canonical: https://semiaifoundry.com/learn/atlas/low-rank-adaptation/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html model-lifecycle-posttraining · Atlas v1.3.0 # Frozen base weights and trainable adapters Multiply a low-rank update and count trainable factors without confusing them with the training objective. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#low-rank-adaptation)Mathematical, data and behavior space; logical dimensions are not physical distance Learning objectiveMultiply a low-rank update and count trainable factors without confusing them with the training objective. ## The mechanism LoRA freezes a base matrix and expresses its update as a product of smaller trainable factors. Rank limits the directions available to that update, rather than declaring which behavior is trained. The same parameterization can be optimized under supervised, preference or other differentiable objectives. The base weights remain part of the forward computation and still need storage unless another representation changes that requirement. Factor count, optimizer state, activation memory and task quality are separate quantities. The 423-parameter lifecycle core currently trains all weights; it does not implement adapters. 1. Declare the base matrix shape, factor shapes, rank and scale. 2. Keep the base fixed while computing the factor product and adding the scaled update. 3. Differentiate through the factors only, then verify base entries are unchanged. 4. Compare task behavior and trainable state under a fixed objective and budget. ## Science and assumptions ### Frozen base weights and trainable adapters · worked relation $$ W=W_0+\frac{\alpha}{r}BA,\quad B\in\mathbb R^{d\times r},\quad A\in\mathbb R^{r\times k},\quad P_{adapter}=r(d+k) $$ Multiply a low-rank update and count trainable factors without confusing them with the training objective. Symbols and units W\_0 Frozen base matrix dimensionless B,A Trainable low-rank factors dimensionless d,k Output and input widths features r Adapter rank rank alpha Declared adapter scale dimensionless P\_adapter Trainable factor count parameters For W of shape 4×6 and rank 1, B=\[1,2,0,−1\]^T and A=\[1,0,2,0,0,−1\] have 4+6=10 trainable entries versus 24 base entries. At scale 1, the first row of BA is \[1,0,2,0,0,−1\] and the second is twice that. Input e3 (third coordinate one) receives update \[2,4,0,−2\]^T. This exact multiplication says nothing by itself about quality after training. Assumptions Rank one, d=4, k=6 and alpha/r=1 in the example. Biases and any additional adapted matrices are excluded. Worked-only parameterization. ## Engineering example For W of shape 4×6 and rank 1, B=\[1,2,0,−1\]^T and A=\[1,0,2,0,0,−1\] have 4+6=10 trainable entries versus 24 base entries. At scale 1, the first row of BA is \[1,0,2,0,0,−1\] and the second is twice that. Input e3 (third coordinate one) receives update \[2,4,0,−2\]^T. This exact multiplication says nothing by itself about quality after training. ## Distinctions and tradeoffs It is a mistake to conclude: All weights needed to run the base model. Low-rank factors reduce trainable optimizer state but restrict the update subspace. Lower trainable count is not a guarantee of equivalent task quality. Separate training signals and algorithms, then inspect actual policy changes and proxy failures. Multiply a low-rank update and count trainable factors without confusing them with the training objective. ## Specifications and interfaces Learning mode worked example The executable decoder has 423 parameters; worked extensions do not imply an additional simulator. Worked contract Multiply a low-rank update and count trainable factors without confusing them with the training objective. Rank one, d=4, k=6 and alpha/r=1 in the example. Biases and any additional adapted matrices are excluded. Worked-only parameterization. ### Interfaces Inputs and state Input Declare the base matrix shape, factor shapes, rank and scale. Result and check Output Multiply a low-rank update and count trainable factors without confusing them with the training objective. ## Check your understanding ### What does the number 10 count? 1. Trainable entries in the two rank-one factors. 2. All weights needed to run the base model. 3. The number of different objectives that LoRA supports. Answer and reasoning Trainable entries in the two rank-one factors. Trainable entries in the two rank-one factors. Rank one, d=4, k=6 and alpha/r=1 in the example. Biases and any additional adapted matrices are excluded. Worked-only parameterization. ## Evidence and scope Mode: worked example. Rank one, d=4, k=6 and alpha/r=1 in the example. Biases and any additional adapted matrices are excluded. Worked-only parameterization. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied. Model lifecycle studio: worked example. Rank one, d=4, k=6 and alpha/r=1 in the example. Biases and any additional adapted matrices are excluded. Worked-only parameterization. ## Sources - [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/html/2106.09685v2) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/low-rank-adaptation-3726097baa621ba3.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Foundations - [Linear maps, rank and retained directions](https://semiaifoundry.com/learn/atlas/linear-map-rank-subspace/) - [Loss, gradients & optimizer](https://semiaifoundry.com/learn/atlas/loss-optimizer/) - [Sharded matrix multiplication and result ownership](https://semiaifoundry.com/learn/atlas/sharded-matmul-ownership/) ### Related lessons - [Instruction tuning and assistant-only loss](https://semiaifoundry.com/learn/atlas/sft-chat-loss-masks/) - [Learn a reward from pairwise choices](https://semiaifoundry.com/learn/atlas/preference-reward-model/) - [DPO policy/reference likelihood ratios](https://semiaifoundry.com/learn/atlas/direct-preference-objective/) - [Rollouts, advantages and PPO constraints](https://semiaifoundry.com/learn/atlas/policy-gradient-kl-control/) - [Verifiers, grouped advantages and RLVR](https://semiaifoundry.com/learn/atlas/verifiable-reward-group-updates/)