{
  "schemaVersion": 1,
  "id": "atlas-lesson-scaling-laws",
  "title": "Scaling laws & resource allocation",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Understand how data, parameters, and compute interact.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/scaling-laws/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Learning",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "The coefficients and exponents are fit parameters, not a universal law.\n\nMode: worked example. Illustrative coefficients only. The 6ND heuristic omits architecture-specific terms and is not a measured cost of the scalar-tape core. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/scaling-laws-93d14a5ba25c61e5.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#scaling-laws",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "7680a3366beeb74b3842c8667abdfee52c18406d4f8aa772351c919b11ba2c01",
  "contentLength": 9548,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-scaling-laws.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-scaling-laws.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Scaling laws & resource allocation",
      "start": 25,
      "end": 318
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 318,
      "end": 1901
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1901,
      "end": 1929
    },
    {
      "id": "section-4",
      "title": "Compute-optimal balance is an empirical fit",
      "start": 1929,
      "end": 4053
    },
    {
      "id": "section-5",
      "title": "Fit a bounded compute/data experiment · worked relation",
      "start": 4053,
      "end": 5159
    },
    {
      "id": "section-6",
      "title": "Engineering example",
      "start": 5159,
      "end": 5687
    },
    {
      "id": "section-7",
      "title": "Distinctions and tradeoffs",
      "start": 5687,
      "end": 6056
    },
    {
      "id": "section-8",
      "title": "Specifications and interfaces",
      "start": 6056,
      "end": 6277
    },
    {
      "id": "section-9",
      "title": "Interfaces",
      "start": 6277,
      "end": 6401
    },
    {
      "id": "section-10",
      "title": "Check your understanding",
      "start": 6401,
      "end": 6430
    },
    {
      "id": "section-11",
      "title": "Do scaling laws establish a guaranteed AGI date?",
      "start": 6430,
      "end": 6737
    },
    {
      "id": "section-12",
      "title": "What would justify calling the displayed relation an empirical fit?",
      "start": 6737,
      "end": 7293
    },
    {
      "id": "section-13",
      "title": "Historical context",
      "start": 7293,
      "end": 7316
    },
    {
      "id": "section-14",
      "title": "2022 · Chinchilla revisits compute-optimal model and data scaling",
      "start": 7316,
      "end": 7694
    },
    {
      "id": "section-15",
      "title": "Evidence and scope",
      "start": 7694,
      "end": 8266
    },
    {
      "id": "section-16",
      "title": "Sources",
      "start": 8266,
      "end": 8776
    },
    {
      "id": "section-17",
      "title": "Continue exploring",
      "start": 8776,
      "end": 8799
    },
    {
      "id": "section-18",
      "title": "Foundations",
      "start": 8799,
      "end": 8993
    },
    {
      "id": "section-19",
      "title": "Related lessons",
      "start": 8993,
      "end": 9548
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#scaling-laws"
    },
    {
      "title": "Hoffmann et al. — Training Compute-Optimal Large Language Models",
      "url": "https://arxiv.org/abs/2203.15556"
    },
    {
      "title": "Kaplan et al. — Scaling Laws",
      "url": "https://arxiv.org/abs/2001.08361"
    },
    {
      "title": "Training Compute-Optimal Large Language Models",
      "url": "https://arxiv.org/html/2203.15556v1"
    },
    {
      "title": "The FineWeb Datasets",
      "url": "https://arxiv.org/html/2406.17557v1"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/scaling-laws-93d14a5ba25c61e5.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Loss, gradients & optimizer",
      "url": "https://semiaifoundry.com/learn/atlas/loss-optimizer/"
    },
    {
      "title": "Evaluation & the AGI frontier",
      "url": "https://semiaifoundry.com/learn/atlas/evaluation/"
    },
    {
      "title": "Training data pipeline",
      "url": "https://semiaifoundry.com/learn/atlas/training-data/"
    },
    {
      "title": "Tensor Core matrix unit",
      "url": "https://semiaifoundry.com/learn/atlas/tensor-core/"
    },
    {
      "title": "Next-token supervision from raw sequences",
      "url": "https://semiaifoundry.com/learn/atlas/causal-pretraining-objective/"
    },
    {
      "title": "Initialization, learning rate and stable updates",
      "url": "https://semiaifoundry.com/learn/atlas/optimizer-schedule-stability/"
    },
    {
      "title": "Gradient accumulation under a fixed global objective",
      "url": "https://semiaifoundry.com/learn/atlas/microbatch-equivalent-updates/"
    }
  ],
  "recordHash": "f29b1bf178a0bcabf881488c39ab759cb59e40ca56e8872420aabace1e713a45",
  "contentFormat": "text/markdown",
  "content": "Learning · Atlas v1.3.0\n\n# Scaling laws & resource allocation\n\nUnderstand how data, parameters, and compute interact.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#scaling-laws)Logical structure · mapped onto physical hardware\n\nLearning objectiveDistinguish empirical fits from guarantees about capability.\n\n## The mechanism\n\nScaling studies fit empirical relationships between training resources and predictive loss over measured regimes. They help estimate how to allocate a compute budget between model size and training data. For a decreasing power-law excess loss, a fixed resource multiplier produces a smaller absolute loss reduction at larger scales; exponential growth in available compute is a separate claim about resources over time. Neither relationship is a proof that general intelligence arrives at a specific date. Algorithms, data quality, hardware efficiency, and inference-time work also matter.\n\nA scaling relation is an empirical fit with a metric, training recipe and observed range. It does not turn parameter count or compute into a universal capability guarantee. Compute-optimal allocation asks how a fixed budget is divided between model size and training data under the study’s cost model. The lifecycle’s one fixed architecture and short runs do not estimate frontier scaling exponents. Comparing several actual runs would require controlled token budgets, held-out fitting checks and residuals rather than a smooth curve drawn from assumptions.\n\n1.  Measure comparable models across controlled resource budgets.\n2.  Fit and test a relationship within the observed regime.\n3.  Use the estimate to plan a new run, then validate the result.\n4.  Name the fitted loss metric, tokenizer and data mixture.\n5.  Separate parameter/data budgets from measured or modeled computation.\n6.  Hold evaluation out of fitting and report residuals and extrapolation boundaries.\n\n## Science and assumptions\n\n### Compute-optimal balance is an empirical fit\n\n$$\n\\begin{aligned}\\mathcal L(N,D)&\\approx\\mathcal L_\\infty+\\frac{A}{N^\\alpha}+\\frac{B}{D^\\beta},\\qquad C\\approx6ND\\\\N_{\\mathrm{opt}}&\\propto C^{\\beta/(\\alpha+\\beta)},\\qquad D_{\\mathrm{opt}}\\propto C^{\\alpha/(\\alpha+\\beta)}\\end{aligned}\n$$\n\nThe fitted loss is a floor plus inverse-power model-size and data-size terms. With compute approximately six times parameters times training tokens, the optimal parameter and data allocations follow the stated complementary compute exponents.\n\nPowers use fixed numerical count conventions. Their fitted coefficients carry the corresponding normalization. This is an empirical planning model, not an intelligence law.\n\nSymbols and units\n\nℒ(N,D)\n\nFitted final training loss\n\ndeclared loss units\n\nℒ∞\n\nFitted irreducible loss floor\n\nsame loss units\n\nN\n\nDense model parameter count\n\nparameters\n\nD\n\nTraining-token count\n\ntokens\n\nA\n\nFitted model-size loss coefficient for the chosen count convention\n\nloss units × parameter-count^α\n\nB\n\nFitted data-size loss coefficient for the chosen count convention\n\nloss units × token-count^β\n\nα,β\n\nPositive fitted loss exponents\n\ndimensionless\n\nC\n\nApproximate parameter-dominated training compute\n\nFLOP\n\nN\\_opt\n\nLoss-minimizing parameter count at fixed C in this model\n\nparameters\n\nD\\_opt\n\nLoss-minimizing training-token count at fixed C in this model\n\ntokens\n\n6\n\nApproximate forward/backward work coefficient\n\nFLOP/(parameter·token)\n\nN denotes model parameters and D training tokens. Under the fitted loss model and fixed compute C, spending everything on either dimension leaves the other as a bottleneck. Minimizing the two error terms yields the allocation exponents shown. These are planning relationships for measured regimes, not a law mapping FLOPs to general intelligence.\n\nAssumptions Positive fitted constants and exponents; dense Transformer parameter-dominated training. The 6ND approximation counts forward and backward matrix work and omits important long-context, embedding, communication, and system costs. Data quality, architecture, and objective must remain sufficiently comparable.\n\n### Fit a bounded compute/data experiment · worked relation\n\n$$\nL(N,D)=E+AN^{-\\alpha}+BD^{-\\beta},\\qquad C\\approx6ND\n$$\n\nInterpret a fitted loss surface under a bounded compute and evaluation regime.\n\nSymbols and units\n\nL\n\nFitted held-out target loss\n\nnats per token\n\nN\n\nTrainable parameter count under the fitted convention\n\nparameters\n\nD\n\nTraining token presentations\n\ntokens\n\nE,A,B\n\nFitted coefficients with units consistent with N and D\n\nfit coefficients\n\nalpha,beta\n\nFitted positive exponents\n\ndimensionless\n\nC\n\nApproximate dense training work under the chosen convention\n\nFLOPs\n\nFor an authored fit L(N,D)=1+4/sqrt(N)+2/sqrt(D), N=16,D=16 gives 2.5. Doubling both to 32 gives 2.060660. This is arithmetic from chosen coefficients, not measured learning. Under the illustrative dense-training cost C=6ND, the two settings cost 1536 and 6144 modeled operations, a fourfold ratio. A real fit requires observed runs and a justified operation convention.\n\nAssumptions Illustrative coefficients only. The 6ND heuristic omits architecture-specific terms and is not a measured cost of the scalar-tape core.\n\n## Engineering example\n\nA fitted loss curve can improve with N and D while hardware, data quality, optimization and evaluation constraints bind elsewhere.\n\nFor an authored fit L(N,D)=1+4/sqrt(N)+2/sqrt(D), N=16,D=16 gives 2.5. Doubling both to 32 gives 2.060660. This is arithmetic from chosen coefficients, not measured learning. Under the illustrative dense-training cost C=6ND, the two settings cost 1536 and 6144 modeled operations, a fourfold ratio. A real fit requires observed runs and a justified operation convention.\n\n## Distinctions and tradeoffs\n\nExponential resource growth is not the same claim as an exponential improvement in every capability.\n\nExtrapolation gets less trustworthy beyond measured regimes. Loss reduction and broad capability are related but not interchangeable.\n\nScaling explains part of the arc from early GPTs to larger systems, alongside algorithmic progress.\n\n## Specifications and interfaces\n\nPower law\n\nL − L∞ ≈ aC⁻ᵅ\n\nIllustrative form; fitted constants are regime dependent.\n\nCompute allocation\n\nParameters × training tokens\n\nSimple dense-model estimates omit many real costs.\n\n### Interfaces\n\nExperiments Input\n\nResource use and measured loss\n\nTraining plan Output\n\nBudget allocation and predictions\n\n## Check your understanding\n\n### Do scaling laws establish a guaranteed AGI date?\n\n1.  Yes\n2.  No; they are empirical relationships over measured regimes\n3.  Only from transistor count\n\nAnswer and reasoning\n\nNo; they are empirical relationships over measured regimes\n\nThey guide resource allocation; they do not prove an AGI timeline.\n\n### What would justify calling the displayed relation an empirical fit?\n\n1.  Choosing coefficients that make a smooth curve.\n2.  One successful 423-parameter run alone.\n3.  Observed controlled runs, fitting procedure and held-out residual checks.\n\nAnswer and reasoning\n\nObserved controlled runs, fitting procedure and held-out residual checks.\n\nObserved controlled runs, fitting procedure and held-out residual checks. Illustrative coefficients only. The 6ND heuristic omits architecture-specific terms and is not a measured cost of the scalar-tape core.\n\n## Historical context\n\n### 2022 · Chinchilla revisits compute-optimal model and data scaling\n\nHoffmann and colleagues found that their compute-optimal runs called for growing training tokens alongside parameters, challenging an emphasis on ever-larger models with comparatively fixed data budgets.\n\n[Hoffmann et al. — Training Compute-Optimal Large Language Models](https://arxiv.org/abs/2203.15556)\n\n## Evidence and scope\n\nThe coefficients and exponents are fit parameters, not a universal law.\n\nMode: worked example. Illustrative coefficients only. The 6ND heuristic omits architecture-specific terms and is not a measured cost of the scalar-tape core. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nExplore a synthetic loss-versus-compute power law; this is not a fit to a named model or a capability forecast.\n\n## Sources\n\n-   [Kaplan et al. — Scaling Laws](https://arxiv.org/abs/2001.08361)\n-   [Hoffmann et al. — Compute-Optimal Training](https://arxiv.org/abs/2203.15556)\n-   [Training Compute-Optimal Large Language Models](https://arxiv.org/html/2203.15556v1)\n-   [The FineWeb Datasets](https://arxiv.org/html/2406.17557v1)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/scaling-laws-93d14a5ba25c61e5.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Loss, gradients & optimizer](https://semiaifoundry.com/learn/atlas/loss-optimizer/)\n-   [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/)\n\n### Related lessons\n\n-   [Training data pipeline](https://semiaifoundry.com/learn/atlas/training-data/)\n-   [Tensor Core matrix unit](https://semiaifoundry.com/learn/atlas/tensor-core/)\n-   [Next-token supervision from raw sequences](https://semiaifoundry.com/learn/atlas/causal-pretraining-objective/)\n-   [Initialization, learning rate and stable updates](https://semiaifoundry.com/learn/atlas/optimizer-schedule-stability/)\n-   [Gradient accumulation under a fixed global objective](https://semiaifoundry.com/learn/atlas/microbatch-equivalent-updates/)"
}