{
  "schemaVersion": 1,
  "id": "atlas-lesson-ablation-reproduction-design",
  "title": "Controlled comparisons and confounded improvement",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Makes the design of an experiment part of the evidence for an improvement.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/ablation-reproduction-design/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "data-evidence-foundations",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "The workbench executes the declared small mechanism. Numeric examples are authored and independently checked, not measurements from the cited research. Finite deterministic authored data with no noise and an additive constant effect. This lesson does not establish a general causal-identification method.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/ablation-reproduction-design-e3b9777cc2d5f751.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#ablation-reproduction-design",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "fdf102dfb9d9909d9a6359f7da2dd9d350a727127d03cbfde66ab3e1461fa326",
  "contentLength": 6484,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-ablation-reproduction-design.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-ablation-reproduction-design.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Controlled comparisons and confounded improvement",
      "start": 42,
      "end": 464
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 464,
      "end": 1652
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1652,
      "end": 1680
    },
    {
      "id": "section-4",
      "title": "Controlled comparisons and confounded improvement",
      "start": 1680,
      "end": 2704
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2704,
      "end": 3087
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 3087,
      "end": 3587
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3587,
      "end": 4214
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 4214,
      "end": 4511
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 4511,
      "end": 4540
    },
    {
      "id": "section-10",
      "title": "Why does the confounded contrast exceed the authored effect?",
      "start": 4540,
      "end": 5134
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 5134,
      "end": 5607
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5607,
      "end": 6035
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 6035,
      "end": 6058
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 6058,
      "end": 6200
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 6200,
      "end": 6484
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#ablation-reproduction-design"
    },
    {
      "title": "Goodfellow, Bengio and Courville · Probability and information theory",
      "url": "https://www.deeplearningbook.org/contents/prob.html"
    },
    {
      "title": "Gebru et al. · Datasheets for Datasets",
      "url": "https://arxiv.org/abs/1803.09010"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/ablation-reproduction-design-e3b9777cc2d5f751.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Sampling variability and independent evidence",
      "url": "https://semiaifoundry.com/learn/atlas/sampling-estimation-uncertainty/"
    },
    {
      "title": "Evaluation & the AGI frontier",
      "url": "https://semiaifoundry.com/learn/atlas/evaluation/"
    },
    {
      "title": "Training data pipeline",
      "url": "https://semiaifoundry.com/learn/atlas/training-data/"
    },
    {
      "title": "Scaling laws & resource allocation",
      "url": "https://semiaifoundry.com/learn/atlas/scaling-laws/"
    }
  ],
  "recordHash": "1e7fc15e660b43f9922518b711e50a368959dee02b21e355a2fc0b876bbc920e",
  "contentFormat": "text/markdown",
  "content": "data-evidence-foundations · Atlas v1.3.0\n\n# Controlled comparisons and confounded improvement\n\nMakes the design of an experiment part of the evidence for an improvement.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#ablation-reproduction-design)Mathematical and data space; dimensions are not physical distance\n\nLearning objectiveSeparate a changed treatment from a changed population, and identify a comparison that lacks support in the observed data.\n\n## The mechanism\n\nA performance difference can reflect both an intervention and the conditions under which it was tested. This synthetic specimen assigns outcomes from a known treatment effect plus a low or high baseline. A balanced allocation places treated and untreated examples in both groups. A confounded allocation gives treatment mostly to the high-baseline group. The crude treated-minus-control difference then includes a population-composition change. Within-group contrasts reveal the issue, but a group containing no control observations cannot identify its own treatment contrast from these data. The author knows the generative rule; the learner must distinguish that knowledge from what the observed comparison establishes.\n\n1.  Declare a baseline outcome for each group and an additive treatment effect in outcome units.\n2.  Allocate treatment using either the balanced or confounded assignment while preserving the outcome-generating rule.\n3.  Calculate the crude difference and separate within-group differences where both treatment states occur.\n4.  Keep unsupported contrasts marked unidentified, then explain which additional design or assumptions would be needed.\n\n## Science and assumptions\n\n### Controlled comparisons and confounded improvement\n\n$$\nY_i=b_{g(i)}+\\tau T_i,\\quad \\widehat\\Delta=\\overline Y_{T=1}-\\overline Y_{T=0}\n$$\n\nSynthetic outcome combines a group baseline and treatment contribution; the crude contrast compares observed group means.\n\nSymbols and units\n\nYᵢ\n\nAuthored synthetic outcome\n\noutcome units\n\nb\\_g\n\nBaseline of group g\n\noutcome units\n\nτ\n\nAuthored treatment effect\n\noutcome units\n\nTᵢ\n\nTreatment indicator\n\nzero or one\n\nΔ̂\n\nCrude observed contrast\n\noutcome units\n\nBaseline outcomes are 2 and 12, with an authored treatment effect of 3. Balanced allocation gives a crude contrast of 3. In the confounded eight-row assignment, the treated mean is 13 and the untreated mean is 2, producing a contrast of 11. The high-baseline group has no untreated rows; its within-group contrast remains unidentified by the observed table.\n\nAssumptions Finite deterministic authored data with no noise and an additive constant effect. This lesson does not establish a general causal-identification method.\n\n## Engineering example\n\nBaseline outcomes are 2 and 12, with an authored treatment effect of 3. Balanced allocation gives a crude contrast of 3. In the confounded eight-row assignment, the treated mean is 13 and the untreated mean is 2, producing a contrast of 11. The high-baseline group has no untreated rows; its within-group contrast remains unidentified by the observed table.\n\n## Distinctions and tradeoffs\n\nKnowing an authored synthetic effect is different from estimating a causal effect from incomplete observational data.\n\nTighter controls improve attribution but can narrow relevance to deployment conditions. An ablation should report both its controlled comparison and the circumstances it excludes.\n\nTraining recipes change data, compute, seeds, architecture and evaluation at once. Without explicit controls, a better score does not explain which change produced it.\n\n## Specifications and interfaces\n\nTeaching specimen\n\nWhat changed?\n\nBaseline outcomes are 2 and 12, with an authored treatment effect of 3. Balanced allocation gives a crude contrast of 3. In the confounded eight-row assignment, the treated mean is 13 and the untreated mean is 2, producing a contrast of 11. The high-baseline group has no untreated rows; its within-group contrast remains unidentified by the observed table.\n\nEvidence boundary\n\nExplicit computed example\n\nFinite deterministic authored data with no noise and an additive constant effect. This lesson does not establish a general causal-identification method.\n\n### Interfaces\n\nInputs and convention Input\n\nDeclare a baseline outcome for each group and an additive treatment effect in outcome units.\n\nComputed result and scope Output\n\nSeparate a changed treatment from a changed population, and identify a comparison that lacks support in the observed data.\n\n## Check your understanding\n\n### Why does the confounded contrast exceed the authored effect?\n\n1.  The calculator changed the authored treatment effect\n2.  More digits automatically prove a causal explanation\n3.  Treatment allocation changes the baseline composition of the compared groups\n\nAnswer and reasoning\n\nTreatment allocation changes the baseline composition of the compared groups\n\nTreatment allocation changes the baseline composition of the compared groups. Finite deterministic authored data with no noise and an additive constant effect. This lesson does not establish a general causal-identification method.\n\n## Evidence and scope\n\nThe workbench executes the declared small mechanism. Numeric examples are authored and independently checked, not measurements from the cited research. Finite deterministic authored data with no noise and an additive constant effect. This lesson does not establish a general causal-identification method.\n\nOpen Mathematical foundations for the bounded executable mechanism and its record. General numerical labs are an additional reference library.\n\n## Sources\n\n-   [Goodfellow, Bengio and Courville · Probability and information theory](https://www.deeplearningbook.org/contents/prob.html)\n-   [Gebru et al. · Datasheets for Datasets](https://arxiv.org/abs/1803.09010)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/ablation-reproduction-design-e3b9777cc2d5f751.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Sampling variability and independent evidence](https://semiaifoundry.com/learn/atlas/sampling-estimation-uncertainty/)\n\n### Related lessons\n\n-   [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/)\n-   [Training data pipeline](https://semiaifoundry.com/learn/atlas/training-data/)\n-   [Scaling laws & resource allocation](https://semiaifoundry.com/learn/atlas/scaling-laws/)"
}