# Context extension is a learned-use test Canonical: https://semiaifoundry.com/learn/atlas/long-context-adaptation/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html model-lifecycle-continuation · Atlas v1.3.0 # Context extension is a learned-use test Separate accepted context length, causal correctness, memory growth and demonstrated long-distance use. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#long-context-adaptation)Mathematical, data and behavior space; logical dimensions are not physical distance Learning objectiveSeparate accepted context length, causal correctness, memory growth and demonstrated long-distance use. ## The mechanism A model accepting a longer input is only the first part of context extension. Positional representations must remain meaningful over new distances, masks must still enforce causality, and training/evaluation must test whether earlier evidence is actually used. YaRN studies a particular rotary-position extension method; its mechanism is not the fixed sinusoidal position rule of the 423-parameter core. Retrieval at long distances, distractor resistance and instruction following can fail separately. Increasing the storage limit alone establishes none of those abilities. 1. Declare original and extended positional configurations and the data used for adaptation. 2. Test future-token invariance independently of task correctness. 3. Move a required symbol to different distances and compare an evidence-removed control. 4. Inventory KV elements at each length with explicit head, layer and dtype conventions. ## Science and assumptions ### Context extension is a learned-use test · worked relation $$ B_{KV}=2LTH_{KV}d_hb $$ Separate accepted context length, causal correctness, memory growth and demonstrated long-distance use. Symbols and units B\_KV Key plus value payload bytes L Layer count layers T Stored token positions tokens H\_KV KV head count heads d\_h Features per KV head features b Stored bytes per element bytes per element With two layers, two KV heads, four features per head and two bytes per stored element, a full KV cache uses 2×layers×tokens×heads×features×bytes. At 128 tokens it is 8192 bytes; at 512 it is 32768 bytes. This fourfold storage increase follows from the declared shape. If a 512-token interface still answers only 5/10 far-evidence cases, accepted length 512 is not evidence of successful long-context use. Assumptions Uncompressed full KV payload with no sharing or overhead. This lesson is a worked extension; the lifecycle core has at most eight input positions and does not implement YaRN. ## Engineering example With two layers, two KV heads, four features per head and two bytes per stored element, a full KV cache uses 2×layers×tokens×heads×features×bytes. At 128 tokens it is 8192 bytes; at 512 it is 32768 bytes. This fourfold storage increase follows from the declared shape. If a 512-token interface still answers only 5/10 far-evidence cases, accepted length 512 is not evidence of successful long-context use. ## Distinctions and tradeoffs It is a mistake to conclude: Answers change correctly when the required far-away evidence changes. Longer contexts consume storage and attention work; positional adaptation and useful information retrieval require their own evidence. Adapt an existing checkpoint while measuring retained and acquired behavior. Separate accepted context length, causal correctness, memory growth and demonstrated long-distance use. ## Specifications and interfaces Learning mode worked example The executable decoder has 423 parameters; worked extensions do not imply an additional simulator. Worked contract Separate accepted context length, causal correctness, memory growth and demonstrated long-distance use. Uncompressed full KV payload with no sharing or overhead. This lesson is a worked extension; the lifecycle core has at most eight input positions and does not implement YaRN. ### Interfaces Inputs and state Input Declare original and extended positional configurations and the data used for adaptation. Result and check Output Separate accepted context length, causal correctness, memory growth and demonstrated long-distance use. ## Check your understanding ### Which observation establishes storage capacity but not learned long-distance use? 1. A reserved long-distance task is solved under a fixed scoring rule. 2. The system accepts 512 tokens and allocates the corresponding KV payload. 3. Answers change correctly when the required far-away evidence changes. Answer and reasoning The system accepts 512 tokens and allocates the corresponding KV payload. The system accepts 512 tokens and allocates the corresponding KV payload. Uncompressed full KV payload with no sharing or overhead. This lesson is a worked extension; the lifecycle core has at most eight input positions and does not implement YaRN. ## Evidence and scope Mode: worked example. Uncompressed full KV payload with no sharing or overhead. This lesson is a worked extension; the lifecycle core has at most eight input positions and does not implement YaRN. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied. Model lifecycle studio: worked example. Uncompressed full KV payload with no sharing or overhead. This lesson is a worked extension; the lifecycle core has at most eight input positions and does not implement YaRN. ## Sources - [YaRN: Efficient Context Window Extension of Large Language Models](https://arxiv.org/html/2309.00071v2) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/long-context-adaptation-2a11cf9f6ad5a9fe.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Foundations - [Position representation](https://semiaifoundry.com/learn/atlas/positional-encoding/) - [Self-attention](https://semiaifoundry.com/learn/atlas/attention/) - [Key–value cache](https://semiaifoundry.com/learn/atlas/kv-cache/) - [Continue language modeling on a new domain](https://semiaifoundry.com/learn/atlas/domain-adaptive-continuation/) ### Related lessons - [Describe a stage by its changed contract](https://semiaifoundry.com/learn/atlas/model-lifecycle-stage-contract/) - [Retention, replay and transfer](https://semiaifoundry.com/learn/atlas/replay-retention-forgetting/) - [Compatible weight averaging and regressions](https://semiaifoundry.com/learn/atlas/checkpoint-averaging-boundaries/)