Frozen base weights and trainable adapters

The mechanism

LoRA freezes a base matrix and expresses its update as a product of smaller trainable factors. Rank limits the directions available to that update, rather than declaring which behavior is trained. The same parameterization can be optimized under supervised, preference or other differentiable objectives. The base weights remain part of the forward computation and still need storage unless another representation changes that requirement. Factor count, optimizer state, activation memory and task quality are separate quantities. The 423-parameter lifecycle core currently trains all weights; it does not implement adapters.

1. Declare the base matrix shape, factor shapes, rank and scale.
2. Keep the base fixed while computing the factor product and adding the scaled update.
3. Differentiate through the factors only, then verify base entries are unchanged.
4. Compare task behavior and trainable state under a fixed objective and budget.

Science and assumptions

Engineering example

For W of shape 4×6 and rank 1, B=[1,2,0,−1]^T and A=[1,0,2,0,0,−1] have 4+6=10 trainable entries versus 24 base entries. At scale 1, the first row of BA is [1,0,2,0,0,−1] and the second is twice that. Input e3 (third coordinate one) receives update [2,4,0,−2]^T. This exact multiplication says nothing by itself about quality after training.

Distinctions and tradeoffs

It is a mistake to conclude: All weights needed to run the base model.

Low-rank factors reduce trainable optimizer state but restrict the update subspace. Lower trainable count is not a guarantee of equivalent task quality.

Separate training signals and algorithms, then inspect actual policy changes and proxy failures. Multiply a low-rank update and count trainable factors without confusing them with the training objective.

Specifications and interfaces

Learning mode

worked example

The executable decoder has 423 parameters; worked extensions do not imply an additional simulator.

Worked contract

Multiply a low-rank update and count trainable factors without confusing them with the training objective.

Rank one, d=4, k=6 and alpha/r=1 in the example. Biases and any additional adapted matrices are excluded. Worked-only parameterization.

Interfaces

Inputs and state Input

Declare the base matrix shape, factor shapes, rank and scale.

Result and check Output

Multiply a low-rank update and count trainable factors without confusing them with the training objective.

Check your understanding

Evidence and scope

Mode: worked example. Rank one, d=4, k=6 and alpha/r=1 in the example. Biases and any additional adapted matrices are excluded. Worked-only parameterization. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.

Model lifecycle studio: worked example. Rank one, d=4, k=6 and alpha/r=1 in the example. Biases and any additional adapted matrices are excluded. Worked-only parameterization.

Sources

• LoRA: Low-Rank Adaptation of Large Language Models

Published lesson data · Atlas releases and licensing

Continue exploring

Foundations

• Linear maps, rank and retained directions
• Loss, gradients & optimizer
• Sharded matrix multiplication and result ownership

Related lessons

• Instruction tuning and assistant-only loss
• Learn a reward from pairwise choices
• DPO policy/reference likelihood ratios
• Rollouts, advantages and PPO constraints
• Verifiers, grouped advantages and RLVR