{"id":"scaling-laws","level":"model","name":"Scaling laws & resource allocation","shortName":"Scaling laws & resource allocation","role":"Understand how data, parameters, and compute interact.","description":"Scaling studies fit empirical relationships between training resources and predictive loss over measured regimes. They help estimate how to allocate a compute budget between model size and training data. For a decreasing power-law excess loss, a fixed resource multiplier produces a smaller absolute loss reduction at larger scales; exponential growth in available compute is a separate claim about resources over time. Neither relationship is a proof that general intelligence arrives at a specific date. Algorithms, data quality, hardware efficiency, and inference-time work also matter.\n\nA scaling relation is an empirical fit with a metric, training recipe and observed range. It does not turn parameter count or compute into a universal capability guarantee. Compute-optimal allocation asks how a fixed budget is divided between model size and training data under the study’s cost model. The lifecycle’s one fixed architecture and short runs do not estimate frontier scaling exponents. Comparing several actual runs would require controlled token budgets, held-out fitting checks and residuals rather than a smooth curve drawn from assumptions.","mechanism":["Measure comparable models across controlled resource budgets.","Fit and test a relationship within the observed regime.","Use the estimate to plan a new run, then validate the result.","Name the fitted loss metric, tokenizer and data mixture.","Separate parameter/data budgets from measured or modeled computation.","Hold evaluation out of fitting and report residuals and extrapolation boundaries."],"specs":[{"label":"Power law","value":"L − L∞ ≈ aC⁻ᵅ","context":"Illustrative form; fitted constants are regime dependent."},{"label":"Compute allocation","value":"Parameters × training tokens","context":"Simple dense-model estimates omit many real costs."}],"interfaces":[{"name":"Experiments","direction":"in","signal":"Resource use and measured loss"},{"name":"Training plan","direction":"out","signal":"Budget allocation and predictions"}],"tradeoff":"Extrapolation gets less trustworthy beyond measured regimes. Loss reduction and broad capability are related but not interchangeable.","whyAI":"Scaling explains part of the arc from early GPTs to larger systems, alongside algorithmic progress.","related":["training-data","loss-optimizer","tensor-core","evaluation","causal-pretraining-objective","optimizer-schedule-stability","microbatch-equivalent-updates"],"sources":[{"title":"Kaplan et al. — Scaling Laws","url":"https://arxiv.org/abs/2001.08361"},{"title":"Hoffmann et al. — Compute-Optimal Training","url":"https://arxiv.org/abs/2203.15556"},{"title":"Training Compute-Optimal Large Language Models","url":"https://arxiv.org/html/2203.15556v1"},{"title":"The FineWeb Datasets","url":"https://arxiv.org/html/2406.17557v1"}],"scale":"Logical structure · mapped onto physical hardware","kind":"scaling-laws","misconception":"Exponential resource growth is not the same claim as an exponential improvement in every capability.","check":{"question":"Do scaling laws establish a guaranteed AGI date?","options":["Yes","No; they are empirical relationships over measured regimes","Only from transistor count"],"answer":1,"explanation":"They guide resource allocation; they do not prove an AGI timeline."},"branch":"Learning","subbranch":"Resource allocation","science":[{"title":"Compute-optimal balance is an empirical fit","equation":"ℒ(N,D) ≈ ℒ∞ + A/Nᵅ + B/Dᵝ; C ≈ 6ND; N_opt ∝ C^[β/(α+β)]; D_opt ∝ C^[α/(α+β)]","explanation":"N denotes model parameters and D training tokens. Under the fitted loss model and fixed compute C, spending everything on either dimension leaves the other as a bottleneck. Minimizing the two error terms yields the allocation exponents shown. These are planning relationships for measured regimes, not a law mapping FLOPs to general intelligence.","assumptions":"Positive fitted constants and exponents; dense Transformer parameter-dominated training. The 6ND approximation counts forward and backward matrix work and omits important long-context, embedding, communication, and system costs. Data quality, architecture, and objective must remain sufficiently comparable."},{"title":"Fit a bounded compute/data experiment · worked relation","equation":"L(N,D)=E+AN^{-\\alpha}+BD^{-\\beta},\\qquad C\\approx6ND","explanation":"For an authored fit L(N,D)=1+4/sqrt(N)+2/sqrt(D), N=16,D=16 gives 2.5. Doubling both to 32 gives 2.060660. This is arithmetic from chosen coefficients, not measured learning. Under the illustrative dense-training cost C=6ND, the two settings cost 1536 and 6144 modeled operations, a fourfold ratio. A real fit requires observed runs and a justified operation convention.","assumptions":"Illustrative coefficients only. The 6ND heuristic omits architecture-specific terms and is not a measured cost of the scalar-tape core."}],"history":[{"year":2022,"title":"Chinchilla revisits compute-optimal model and data scaling","significance":"Hoffmann and colleagues found that their compute-optimal runs called for growing training tokens alongside parameters, challenging an emphasis on ever-larger models with comparatively fixed data budgets.","source":{"title":"Hoffmann et al. — Training Compute-Optimal Large Language Models","url":"https://arxiv.org/abs/2203.15556"}}],"learningObjective":"Distinguish empirical fits from guarantees about capability.","engineeringExample":"A fitted loss curve can improve with N and D while hardware, data quality, optimization and evaluation constraints bind elsewhere.\n\nFor an authored fit L(N,D)=1+4/sqrt(N)+2/sqrt(D), N=16,D=16 gives 2.5. Doubling both to 32 gives 2.060660. This is arithmetic from chosen coefficients, not measured learning. Under the illustrative dense-training cost C=6ND, the two settings cost 1536 and 6144 modeled operations, a fourfold ratio. A real fit requires observed runs and a justified operation convention.","evidenceNotes":"The coefficients and exponents are fit parameters, not a universal law.\n\nMode: worked example. Illustrative coefficients only. The 6ND heuristic omits architecture-specific terms and is not a measured cost of the scalar-tape core. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","visualFamily":"model-workload","prerequisites":["loss-optimizer","evaluation"],"checks":[{"question":"What would justify calling the displayed relation an empirical fit?","options":["Choosing coefficients that make a smooth curve.","One successful 423-parameter run alone.","Observed controlled runs, fitting procedure and held-out residual checks."],"answer":2,"explanation":"Observed controlled runs, fitting procedure and held-out residual checks. Illustrative coefficients only. The 6ND heuristic omits architecture-specific terms and is not a measured cost of the scalar-tape core."}],"labId":"scaling","scope":"Explore a synthetic loss-versus-compute power law; this is not a fit to a named model or a capability forecast.","labScope":"Explore a synthetic loss-versus-compute power law; this is not a fit to a named model or a capability forecast."}