The mechanism
A model accepting a longer input is only the first part of context extension. Positional representations must remain meaningful over new distances, masks must still enforce causality, and training/evaluation must test whether earlier evidence is actually used. YaRN studies a particular rotary-position extension method; its mechanism is not the fixed sinusoidal position rule of the 423-parameter core. Retrieval at long distances, distractor resistance and instruction following can fail separately. Increasing the storage limit alone establishes none of those abilities.
- Declare original and extended positional configurations and the data used for adaptation.
- Test future-token invariance independently of task correctness.
- Move a required symbol to different distances and compare an evidence-removed control.
- Inventory KV elements at each length with explicit head, layer and dtype conventions.
Science and assumptions
Context extension is a learned-use test · worked relation
Separate accepted context length, causal correctness, memory growth and demonstrated long-distance use.
Symbols and units
- B_KV
- Key plus value payload
bytes
- L
- Layer count
layers
- T
- Stored token positions
tokens
- H_KV
- KV head count
heads
- d_h
- Features per KV head
features
- b
- Stored bytes per element
bytes per element
With two layers, two KV heads, four features per head and two bytes per stored element, a full KV cache uses 2×layers×tokens×heads×features×bytes. At 128 tokens it is 8192 bytes; at 512 it is 32768 bytes. This fourfold storage increase follows from the declared shape. If a 512-token interface still answers only 5/10 far-evidence cases, accepted length 512 is not evidence of successful long-context use.
Assumptions Uncompressed full KV payload with no sharing or overhead. This lesson is a worked extension; the lifecycle core has at most eight input positions and does not implement YaRN.
Engineering example
With two layers, two KV heads, four features per head and two bytes per stored element, a full KV cache uses 2×layers×tokens×heads×features×bytes. At 128 tokens it is 8192 bytes; at 512 it is 32768 bytes. This fourfold storage increase follows from the declared shape. If a 512-token interface still answers only 5/10 far-evidence cases, accepted length 512 is not evidence of successful long-context use.
Distinctions and tradeoffs
It is a mistake to conclude: Answers change correctly when the required far-away evidence changes.
Longer contexts consume storage and attention work; positional adaptation and useful information retrieval require their own evidence.
Adapt an existing checkpoint while measuring retained and acquired behavior. Separate accepted context length, causal correctness, memory growth and demonstrated long-distance use.
Specifications and interfaces
- Learning mode
- worked example
The executable decoder has 423 parameters; worked extensions do not imply an additional simulator.
- Worked contract
- Separate accepted context length, causal correctness, memory growth and demonstrated long-distance use.
Uncompressed full KV payload with no sharing or overhead. This lesson is a worked extension; the lifecycle core has at most eight input positions and does not implement YaRN.
Interfaces
- Inputs and state Input
- Declare original and extended positional configurations and the data used for adaptation.
- Result and check Output
- Separate accepted context length, causal correctness, memory growth and demonstrated long-distance use.
Check your understanding
Which observation establishes storage capacity but not learned long-distance use?
- A reserved long-distance task is solved under a fixed scoring rule.
- The system accepts 512 tokens and allocates the corresponding KV payload.
- Answers change correctly when the required far-away evidence changes.
Answer and reasoning
The system accepts 512 tokens and allocates the corresponding KV payload.
The system accepts 512 tokens and allocates the corresponding KV payload. Uncompressed full KV payload with no sharing or overhead. This lesson is a worked extension; the lifecycle core has at most eight input positions and does not implement YaRN.
Evidence and scope
Mode: worked example. Uncompressed full KV payload with no sharing or overhead. This lesson is a worked extension; the lifecycle core has at most eight input positions and does not implement YaRN. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.
Model lifecycle studio: worked example. Uncompressed full KV payload with no sharing or overhead. This lesson is a worked extension; the lifecycle core has at most eight input positions and does not implement YaRN.