{"id":"long-context-adaptation","name":"Context extension is a learned-use test","shortName":"Context extension is a learned-use test","role":"Separate accepted context length, causal correctness, memory growth and demonstrated long-distance use.","description":"A model accepting a longer input is only the first part of context extension. Positional representations must remain meaningful over new distances, masks must still enforce causality, and training/evaluation must test whether earlier evidence is actually used. YaRN studies a particular rotary-position extension method; its mechanism is not the fixed sinusoidal position rule of the 423-parameter core. Retrieval at long distances, distractor resistance and instruction following can fail separately. Increasing the storage limit alone establishes none of those abilities.","mechanism":["Declare original and extended positional configurations and the data used for adaptation.","Test future-token invariance independently of task correctness.","Move a required symbol to different distances and compare an evidence-removed control.","Inventory KV elements at each length with explicit head, layer and dtype conventions."],"learningObjective":"Separate accepted context length, causal correctness, memory growth and demonstrated long-distance use.","engineeringExample":"With two layers, two KV heads, four features per head and two bytes per stored element, a full KV cache uses 2×layers×tokens×heads×features×bytes. At 128 tokens it is 8192 bytes; at 512 it is 32768 bytes. This fourfold storage increase follows from the declared shape. If a 512-token interface still answers only 5/10 far-evidence cases, accepted length 512 is not evidence of successful long-context use.","evidenceNotes":"Mode: worked example. Uncompressed full KV payload with no sharing or overhead. This lesson is a worked extension; the lifecycle core has at most eight input positions and does not implement YaRN. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","specs":[{"label":"Learning mode","value":"worked example","context":"The executable decoder has 423 parameters; worked extensions do not imply an additional simulator."},{"label":"Worked contract","value":"Separate accepted context length, causal correctness, memory growth and demonstrated long-distance use.","context":"Uncompressed full KV payload with no sharing or overhead. This lesson is a worked extension; the lifecycle core has at most eight input positions and does not implement YaRN."}],"interfaces":[{"name":"Inputs and state","direction":"in","signal":"Declare original and extended positional configurations and the data used for adaptation."},{"name":"Result and check","direction":"out","signal":"Separate accepted context length, causal correctness, memory growth and demonstrated long-distance use."}],"tradeoff":"Longer contexts consume storage and attention work; positional adaptation and useful information retrieval require their own evidence.","whyAI":"Adapt an existing checkpoint while measuring retained and acquired behavior. Separate accepted context length, causal correctness, memory growth and demonstrated long-distance use.","misconception":"It is a mistake to conclude: Answers change correctly when the required far-away evidence changes.","check":{"question":"Which observation establishes storage capacity but not learned long-distance use?","options":["A reserved long-distance task is solved under a fixed scoring rule.","The system accepts 512 tokens and allocates the corresponding KV payload.","Answers change correctly when the required far-away evidence changes."],"answer":1,"explanation":"The system accepts 512 tokens and allocates the corresponding KV payload. Uncompressed full KV payload with no sharing or overhead. This lesson is a worked extension; the lifecycle core has at most eight input positions and does not implement YaRN."},"prerequisites":["positional-encoding","attention","kv-cache","domain-adaptive-continuation"],"related":["positional-encoding","attention","kv-cache","domain-adaptive-continuation","model-lifecycle-stage-contract","replay-retention-forgetting","checkpoint-averaging-boundaries"],"sources":[{"title":"YaRN: Efficient Context Window Extension of Large Language Models","url":"https://arxiv.org/html/2309.00071v2"}],"level":"model","scale":"Mathematical, data and behavior space; logical dimensions are not physical distance","kind":"model","branch":"model-lifecycle-continuation","subbranch":"Context extension is a learned-use test","workstream":"W11","visualFamily":"mathematical","labId":"library","labScope":"Model lifecycle studio: worked example. Uncompressed full KV payload with no sharing or overhead. This lesson is a worked extension; the lifecycle core has at most eight input positions and does not implement YaRN.","science":[{"title":"Context extension is a learned-use test · worked relation","equation":"B_{KV}=2LTH_{KV}d_hb","explanation":"With two layers, two KV heads, four features per head and two bytes per stored element, a full KV cache uses 2×layers×tokens×heads×features×bytes. At 128 tokens it is 8192 bytes; at 512 it is 32768 bytes. This fourfold storage increase follows from the declared shape. If a 512-token interface still answers only 5/10 far-evidence cases, accepted length 512 is not evidence of successful long-context use.","assumptions":"Uncompressed full KV payload with no sharing or overhead. This lesson is a worked extension; the lifecycle core has at most eight input positions and does not implement YaRN."}],"history":[]}