# Controlled comparisons and confounded improvement Canonical: https://semiaifoundry.com/learn/atlas/ablation-reproduction-design/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html data-evidence-foundations · Atlas v1.3.0 # Controlled comparisons and confounded improvement Makes the design of an experiment part of the evidence for an improvement. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#ablation-reproduction-design)Mathematical and data space; dimensions are not physical distance Learning objectiveSeparate a changed treatment from a changed population, and identify a comparison that lacks support in the observed data. ## The mechanism A performance difference can reflect both an intervention and the conditions under which it was tested. This synthetic specimen assigns outcomes from a known treatment effect plus a low or high baseline. A balanced allocation places treated and untreated examples in both groups. A confounded allocation gives treatment mostly to the high-baseline group. The crude treated-minus-control difference then includes a population-composition change. Within-group contrasts reveal the issue, but a group containing no control observations cannot identify its own treatment contrast from these data. The author knows the generative rule; the learner must distinguish that knowledge from what the observed comparison establishes. 1. Declare a baseline outcome for each group and an additive treatment effect in outcome units. 2. Allocate treatment using either the balanced or confounded assignment while preserving the outcome-generating rule. 3. Calculate the crude difference and separate within-group differences where both treatment states occur. 4. Keep unsupported contrasts marked unidentified, then explain which additional design or assumptions would be needed. ## Science and assumptions ### Controlled comparisons and confounded improvement $$ Y_i=b_{g(i)}+\tau T_i,\quad \widehat\Delta=\overline Y_{T=1}-\overline Y_{T=0} $$ Synthetic outcome combines a group baseline and treatment contribution; the crude contrast compares observed group means. Symbols and units Yᵢ Authored synthetic outcome outcome units b\_g Baseline of group g outcome units τ Authored treatment effect outcome units Tᵢ Treatment indicator zero or one Δ̂ Crude observed contrast outcome units Baseline outcomes are 2 and 12, with an authored treatment effect of 3. Balanced allocation gives a crude contrast of 3. In the confounded eight-row assignment, the treated mean is 13 and the untreated mean is 2, producing a contrast of 11. The high-baseline group has no untreated rows; its within-group contrast remains unidentified by the observed table. Assumptions Finite deterministic authored data with no noise and an additive constant effect. This lesson does not establish a general causal-identification method. ## Engineering example Baseline outcomes are 2 and 12, with an authored treatment effect of 3. Balanced allocation gives a crude contrast of 3. In the confounded eight-row assignment, the treated mean is 13 and the untreated mean is 2, producing a contrast of 11. The high-baseline group has no untreated rows; its within-group contrast remains unidentified by the observed table. ## Distinctions and tradeoffs Knowing an authored synthetic effect is different from estimating a causal effect from incomplete observational data. Tighter controls improve attribution but can narrow relevance to deployment conditions. An ablation should report both its controlled comparison and the circumstances it excludes. Training recipes change data, compute, seeds, architecture and evaluation at once. Without explicit controls, a better score does not explain which change produced it. ## Specifications and interfaces Teaching specimen What changed? Baseline outcomes are 2 and 12, with an authored treatment effect of 3. Balanced allocation gives a crude contrast of 3. In the confounded eight-row assignment, the treated mean is 13 and the untreated mean is 2, producing a contrast of 11. The high-baseline group has no untreated rows; its within-group contrast remains unidentified by the observed table. Evidence boundary Explicit computed example Finite deterministic authored data with no noise and an additive constant effect. This lesson does not establish a general causal-identification method. ### Interfaces Inputs and convention Input Declare a baseline outcome for each group and an additive treatment effect in outcome units. Computed result and scope Output Separate a changed treatment from a changed population, and identify a comparison that lacks support in the observed data. ## Check your understanding ### Why does the confounded contrast exceed the authored effect? 1. The calculator changed the authored treatment effect 2. More digits automatically prove a causal explanation 3. Treatment allocation changes the baseline composition of the compared groups Answer and reasoning Treatment allocation changes the baseline composition of the compared groups Treatment allocation changes the baseline composition of the compared groups. Finite deterministic authored data with no noise and an additive constant effect. This lesson does not establish a general causal-identification method. ## Evidence and scope The workbench executes the declared small mechanism. Numeric examples are authored and independently checked, not measurements from the cited research. Finite deterministic authored data with no noise and an additive constant effect. This lesson does not establish a general causal-identification method. Open Mathematical foundations for the bounded executable mechanism and its record. General numerical labs are an additional reference library. ## Sources - [Goodfellow, Bengio and Courville · Probability and information theory](https://www.deeplearningbook.org/contents/prob.html) - [Gebru et al. · Datasheets for Datasets](https://arxiv.org/abs/1803.09010) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/ablation-reproduction-design-e3b9777cc2d5f751.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Foundations - [Sampling variability and independent evidence](https://semiaifoundry.com/learn/atlas/sampling-estimation-uncertainty/) ### Related lessons - [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/) - [Training data pipeline](https://semiaifoundry.com/learn/atlas/training-data/) - [Scaling laws & resource allocation](https://semiaifoundry.com/learn/atlas/scaling-laws/)