# Scaling laws & resource allocation Canonical: https://semiaifoundry.com/learn/atlas/scaling-laws/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html Learning · Atlas v1.3.0 # Scaling laws & resource allocation Understand how data, parameters, and compute interact. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#scaling-laws)Logical structure · mapped onto physical hardware Learning objectiveDistinguish empirical fits from guarantees about capability. ## The mechanism Scaling studies fit empirical relationships between training resources and predictive loss over measured regimes. They help estimate how to allocate a compute budget between model size and training data. For a decreasing power-law excess loss, a fixed resource multiplier produces a smaller absolute loss reduction at larger scales; exponential growth in available compute is a separate claim about resources over time. Neither relationship is a proof that general intelligence arrives at a specific date. Algorithms, data quality, hardware efficiency, and inference-time work also matter. A scaling relation is an empirical fit with a metric, training recipe and observed range. It does not turn parameter count or compute into a universal capability guarantee. Compute-optimal allocation asks how a fixed budget is divided between model size and training data under the study’s cost model. The lifecycle’s one fixed architecture and short runs do not estimate frontier scaling exponents. Comparing several actual runs would require controlled token budgets, held-out fitting checks and residuals rather than a smooth curve drawn from assumptions. 1. Measure comparable models across controlled resource budgets. 2. Fit and test a relationship within the observed regime. 3. Use the estimate to plan a new run, then validate the result. 4. Name the fitted loss metric, tokenizer and data mixture. 5. Separate parameter/data budgets from measured or modeled computation. 6. Hold evaluation out of fitting and report residuals and extrapolation boundaries. ## Science and assumptions ### Compute-optimal balance is an empirical fit $$ \begin{aligned}\mathcal L(N,D)&\approx\mathcal L_\infty+\frac{A}{N^\alpha}+\frac{B}{D^\beta},\qquad C\approx6ND\\N_{\mathrm{opt}}&\propto C^{\beta/(\alpha+\beta)},\qquad D_{\mathrm{opt}}\propto C^{\alpha/(\alpha+\beta)}\end{aligned} $$ The fitted loss is a floor plus inverse-power model-size and data-size terms. With compute approximately six times parameters times training tokens, the optimal parameter and data allocations follow the stated complementary compute exponents. Powers use fixed numerical count conventions. Their fitted coefficients carry the corresponding normalization. This is an empirical planning model, not an intelligence law. Symbols and units ℒ(N,D) Fitted final training loss declared loss units ℒ∞ Fitted irreducible loss floor same loss units N Dense model parameter count parameters D Training-token count tokens A Fitted model-size loss coefficient for the chosen count convention loss units × parameter-count^α B Fitted data-size loss coefficient for the chosen count convention loss units × token-count^β α,β Positive fitted loss exponents dimensionless C Approximate parameter-dominated training compute FLOP N\_opt Loss-minimizing parameter count at fixed C in this model parameters D\_opt Loss-minimizing training-token count at fixed C in this model tokens 6 Approximate forward/backward work coefficient FLOP/(parameter·token) N denotes model parameters and D training tokens. Under the fitted loss model and fixed compute C, spending everything on either dimension leaves the other as a bottleneck. Minimizing the two error terms yields the allocation exponents shown. These are planning relationships for measured regimes, not a law mapping FLOPs to general intelligence. Assumptions Positive fitted constants and exponents; dense Transformer parameter-dominated training. The 6ND approximation counts forward and backward matrix work and omits important long-context, embedding, communication, and system costs. Data quality, architecture, and objective must remain sufficiently comparable. ### Fit a bounded compute/data experiment · worked relation $$ L(N,D)=E+AN^{-\alpha}+BD^{-\beta},\qquad C\approx6ND $$ Interpret a fitted loss surface under a bounded compute and evaluation regime. Symbols and units L Fitted held-out target loss nats per token N Trainable parameter count under the fitted convention parameters D Training token presentations tokens E,A,B Fitted coefficients with units consistent with N and D fit coefficients alpha,beta Fitted positive exponents dimensionless C Approximate dense training work under the chosen convention FLOPs For an authored fit L(N,D)=1+4/sqrt(N)+2/sqrt(D), N=16,D=16 gives 2.5. Doubling both to 32 gives 2.060660. This is arithmetic from chosen coefficients, not measured learning. Under the illustrative dense-training cost C=6ND, the two settings cost 1536 and 6144 modeled operations, a fourfold ratio. A real fit requires observed runs and a justified operation convention. Assumptions Illustrative coefficients only. The 6ND heuristic omits architecture-specific terms and is not a measured cost of the scalar-tape core. ## Engineering example A fitted loss curve can improve with N and D while hardware, data quality, optimization and evaluation constraints bind elsewhere. For an authored fit L(N,D)=1+4/sqrt(N)+2/sqrt(D), N=16,D=16 gives 2.5. Doubling both to 32 gives 2.060660. This is arithmetic from chosen coefficients, not measured learning. Under the illustrative dense-training cost C=6ND, the two settings cost 1536 and 6144 modeled operations, a fourfold ratio. A real fit requires observed runs and a justified operation convention. ## Distinctions and tradeoffs Exponential resource growth is not the same claim as an exponential improvement in every capability. Extrapolation gets less trustworthy beyond measured regimes. Loss reduction and broad capability are related but not interchangeable. Scaling explains part of the arc from early GPTs to larger systems, alongside algorithmic progress. ## Specifications and interfaces Power law L − L∞ ≈ aC⁻ᵅ Illustrative form; fitted constants are regime dependent. Compute allocation Parameters × training tokens Simple dense-model estimates omit many real costs. ### Interfaces Experiments Input Resource use and measured loss Training plan Output Budget allocation and predictions ## Check your understanding ### Do scaling laws establish a guaranteed AGI date? 1. Yes 2. No; they are empirical relationships over measured regimes 3. Only from transistor count Answer and reasoning No; they are empirical relationships over measured regimes They guide resource allocation; they do not prove an AGI timeline. ### What would justify calling the displayed relation an empirical fit? 1. Choosing coefficients that make a smooth curve. 2. One successful 423-parameter run alone. 3. Observed controlled runs, fitting procedure and held-out residual checks. Answer and reasoning Observed controlled runs, fitting procedure and held-out residual checks. Observed controlled runs, fitting procedure and held-out residual checks. Illustrative coefficients only. The 6ND heuristic omits architecture-specific terms and is not a measured cost of the scalar-tape core. ## Historical context ### 2022 · Chinchilla revisits compute-optimal model and data scaling Hoffmann and colleagues found that their compute-optimal runs called for growing training tokens alongside parameters, challenging an emphasis on ever-larger models with comparatively fixed data budgets. [Hoffmann et al. — Training Compute-Optimal Large Language Models](https://arxiv.org/abs/2203.15556) ## Evidence and scope The coefficients and exponents are fit parameters, not a universal law. Mode: worked example. Illustrative coefficients only. The 6ND heuristic omits architecture-specific terms and is not a measured cost of the scalar-tape core. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied. Explore a synthetic loss-versus-compute power law; this is not a fit to a named model or a capability forecast. ## Sources - [Kaplan et al. — Scaling Laws](https://arxiv.org/abs/2001.08361) - [Hoffmann et al. — Compute-Optimal Training](https://arxiv.org/abs/2203.15556) - [Training Compute-Optimal Large Language Models](https://arxiv.org/html/2203.15556v1) - [The FineWeb Datasets](https://arxiv.org/html/2406.17557v1) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/scaling-laws-93d14a5ba25c61e5.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Foundations - [Loss, gradients & optimizer](https://semiaifoundry.com/learn/atlas/loss-optimizer/) - [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/) ### Related lessons - [Training data pipeline](https://semiaifoundry.com/learn/atlas/training-data/) - [Tensor Core matrix unit](https://semiaifoundry.com/learn/atlas/tensor-core/) - [Next-token supervision from raw sequences](https://semiaifoundry.com/learn/atlas/causal-pretraining-objective/) - [Initialization, learning rate and stable updates](https://semiaifoundry.com/learn/atlas/optimizer-schedule-stability/) - [Gradient accumulation under a fixed global objective](https://semiaifoundry.com/learn/atlas/microbatch-equivalent-updates/)