The mechanism
Scaling studies fit empirical relationships between training resources and predictive loss over measured regimes. They help estimate how to allocate a compute budget between model size and training data. For a decreasing power-law excess loss, a fixed resource multiplier produces a smaller absolute loss reduction at larger scales; exponential growth in available compute is a separate claim about resources over time. Neither relationship is a proof that general intelligence arrives at a specific date. Algorithms, data quality, hardware efficiency, and inference-time work also matter.
A scaling relation is an empirical fit with a metric, training recipe and observed range. It does not turn parameter count or compute into a universal capability guarantee. Compute-optimal allocation asks how a fixed budget is divided between model size and training data under the study’s cost model. The lifecycle’s one fixed architecture and short runs do not estimate frontier scaling exponents. Comparing several actual runs would require controlled token budgets, held-out fitting checks and residuals rather than a smooth curve drawn from assumptions.
- Measure comparable models across controlled resource budgets.
- Fit and test a relationship within the observed regime.
- Use the estimate to plan a new run, then validate the result.
- Name the fitted loss metric, tokenizer and data mixture.
- Separate parameter/data budgets from measured or modeled computation.
- Hold evaluation out of fitting and report residuals and extrapolation boundaries.
Science and assumptions
Compute-optimal balance is an empirical fit
The fitted loss is a floor plus inverse-power model-size and data-size terms. With compute approximately six times parameters times training tokens, the optimal parameter and data allocations follow the stated complementary compute exponents.
Powers use fixed numerical count conventions. Their fitted coefficients carry the corresponding normalization. This is an empirical planning model, not an intelligence law.
Symbols and units
- ℒ(N,D)
- Fitted final training loss
declared loss units
- ℒ∞
- Fitted irreducible loss floor
same loss units
- N
- Dense model parameter count
parameters
- D
- Training-token count
tokens
- A
- Fitted model-size loss coefficient for the chosen count convention
loss units × parameter-count^α
- B
- Fitted data-size loss coefficient for the chosen count convention
loss units × token-count^β
- α,β
- Positive fitted loss exponents
dimensionless
- C
- Approximate parameter-dominated training compute
FLOP
- N_opt
- Loss-minimizing parameter count at fixed C in this model
parameters
- D_opt
- Loss-minimizing training-token count at fixed C in this model
tokens
- 6
- Approximate forward/backward work coefficient
FLOP/(parameter·token)
N denotes model parameters and D training tokens. Under the fitted loss model and fixed compute C, spending everything on either dimension leaves the other as a bottleneck. Minimizing the two error terms yields the allocation exponents shown. These are planning relationships for measured regimes, not a law mapping FLOPs to general intelligence.
Assumptions Positive fitted constants and exponents; dense Transformer parameter-dominated training. The 6ND approximation counts forward and backward matrix work and omits important long-context, embedding, communication, and system costs. Data quality, architecture, and objective must remain sufficiently comparable.
Fit a bounded compute/data experiment · worked relation
Interpret a fitted loss surface under a bounded compute and evaluation regime.
Symbols and units
- L
- Fitted held-out target loss
nats per token
- N
- Trainable parameter count under the fitted convention
parameters
- D
- Training token presentations
tokens
- E,A,B
- Fitted coefficients with units consistent with N and D
fit coefficients
- alpha,beta
- Fitted positive exponents
dimensionless
- C
- Approximate dense training work under the chosen convention
FLOPs
For an authored fit L(N,D)=1+4/sqrt(N)+2/sqrt(D), N=16,D=16 gives 2.5. Doubling both to 32 gives 2.060660. This is arithmetic from chosen coefficients, not measured learning. Under the illustrative dense-training cost C=6ND, the two settings cost 1536 and 6144 modeled operations, a fourfold ratio. A real fit requires observed runs and a justified operation convention.
Assumptions Illustrative coefficients only. The 6ND heuristic omits architecture-specific terms and is not a measured cost of the scalar-tape core.
Engineering example
A fitted loss curve can improve with N and D while hardware, data quality, optimization and evaluation constraints bind elsewhere.
For an authored fit L(N,D)=1+4/sqrt(N)+2/sqrt(D), N=16,D=16 gives 2.5. Doubling both to 32 gives 2.060660. This is arithmetic from chosen coefficients, not measured learning. Under the illustrative dense-training cost C=6ND, the two settings cost 1536 and 6144 modeled operations, a fourfold ratio. A real fit requires observed runs and a justified operation convention.
Distinctions and tradeoffs
Exponential resource growth is not the same claim as an exponential improvement in every capability.
Extrapolation gets less trustworthy beyond measured regimes. Loss reduction and broad capability are related but not interchangeable.
Scaling explains part of the arc from early GPTs to larger systems, alongside algorithmic progress.
Specifications and interfaces
- Power law
- L − L∞ ≈ aC⁻ᵅ
Illustrative form; fitted constants are regime dependent.
- Compute allocation
- Parameters × training tokens
Simple dense-model estimates omit many real costs.
Interfaces
- Experiments Input
- Resource use and measured loss
- Training plan Output
- Budget allocation and predictions
Check your understanding
Do scaling laws establish a guaranteed AGI date?
- Yes
- No; they are empirical relationships over measured regimes
- Only from transistor count
Answer and reasoning
No; they are empirical relationships over measured regimes
They guide resource allocation; they do not prove an AGI timeline.
What would justify calling the displayed relation an empirical fit?
- Choosing coefficients that make a smooth curve.
- One successful 423-parameter run alone.
- Observed controlled runs, fitting procedure and held-out residual checks.
Answer and reasoning
Observed controlled runs, fitting procedure and held-out residual checks.
Observed controlled runs, fitting procedure and held-out residual checks. Illustrative coefficients only. The 6ND heuristic omits architecture-specific terms and is not a measured cost of the scalar-tape core.
Historical context
2022 · Chinchilla revisits compute-optimal model and data scaling
Hoffmann and colleagues found that their compute-optimal runs called for growing training tokens alongside parameters, challenging an emphasis on ever-larger models with comparatively fixed data budgets.
Hoffmann et al. — Training Compute-Optimal Large Language Models
Evidence and scope
The coefficients and exponents are fit parameters, not a universal law.
Mode: worked example. Illustrative coefficients only. The 6ND heuristic omits architecture-specific terms and is not a measured cost of the scalar-tape core. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.
Explore a synthetic loss-versus-compute power law; this is not a fit to a named model or a capability forecast.