{"id":"ablation-reproduction-design","name":"Controlled comparisons and confounded improvement","shortName":"What changed?","role":"Makes the design of an experiment part of the evidence for an improvement.","description":"A performance difference can reflect both an intervention and the conditions under which it was tested. This synthetic specimen assigns outcomes from a known treatment effect plus a low or high baseline. A balanced allocation places treated and untreated examples in both groups. A confounded allocation gives treatment mostly to the high-baseline group. The crude treated-minus-control difference then includes a population-composition change. Within-group contrasts reveal the issue, but a group containing no control observations cannot identify its own treatment contrast from these data. The author knows the generative rule; the learner must distinguish that knowledge from what the observed comparison establishes.","mechanism":["Declare a baseline outcome for each group and an additive treatment effect in outcome units.","Allocate treatment using either the balanced or confounded assignment while preserving the outcome-generating rule.","Calculate the crude difference and separate within-group differences where both treatment states occur.","Keep unsupported contrasts marked unidentified, then explain which additional design or assumptions would be needed."],"learningObjective":"Separate a changed treatment from a changed population, and identify a comparison that lacks support in the observed data.","engineeringExample":"Baseline outcomes are 2 and 12, with an authored treatment effect of 3. Balanced allocation gives a crude contrast of 3. In the confounded eight-row assignment, the treated mean is 13 and the untreated mean is 2, producing a contrast of 11. The high-baseline group has no untreated rows; its within-group contrast remains unidentified by the observed table.","evidenceNotes":"The workbench executes the declared small mechanism. Numeric examples are authored and independently checked, not measurements from the cited research. Finite deterministic authored data with no noise and an additive constant effect. This lesson does not establish a general causal-identification method.","specs":[{"label":"Teaching specimen","value":"What changed?","context":"Baseline outcomes are 2 and 12, with an authored treatment effect of 3. Balanced allocation gives a crude contrast of 3. In the confounded eight-row assignment, the treated mean is 13 and the untreated mean is 2, producing a contrast of 11. The high-baseline group has no untreated rows; its within-group contrast remains unidentified by the observed table."},{"label":"Evidence boundary","value":"Explicit computed example","context":"Finite deterministic authored data with no noise and an additive constant effect. This lesson does not establish a general causal-identification method."}],"interfaces":[{"name":"Inputs and convention","direction":"in","signal":"Declare a baseline outcome for each group and an additive treatment effect in outcome units."},{"name":"Computed result and scope","direction":"out","signal":"Separate a changed treatment from a changed population, and identify a comparison that lacks support in the observed data."}],"tradeoff":"Tighter controls improve attribution but can narrow relevance to deployment conditions. An ablation should report both its controlled comparison and the circumstances it excludes.","whyAI":"Training recipes change data, compute, seeds, architecture and evaluation at once. Without explicit controls, a better score does not explain which change produced it.","misconception":"Knowing an authored synthetic effect is different from estimating a causal effect from incomplete observational data.","check":{"question":"Why does the confounded contrast exceed the authored effect?","options":["The calculator changed the authored treatment effect","More digits automatically prove a causal explanation","Treatment allocation changes the baseline composition of the compared groups"],"answer":2,"explanation":"Treatment allocation changes the baseline composition of the compared groups. Finite deterministic authored data with no noise and an additive constant effect. This lesson does not establish a general causal-identification method."},"prerequisites":["sampling-estimation-uncertainty"],"related":["evaluation","training-data","scaling-laws"],"sources":[{"title":"Goodfellow, Bengio and Courville · Probability and information theory","url":"https://www.deeplearningbook.org/contents/prob.html"},{"title":"Gebru et al. · Datasheets for Datasets","url":"https://arxiv.org/abs/1803.09010"}],"level":"model","scale":"Mathematical and data space; dimensions are not physical distance","kind":"model","branch":"data-evidence-foundations","subbranch":"What changed?","workstream":"W11","visualFamily":"mathematical","labId":"library","labScope":"Open Mathematical foundations for the bounded executable mechanism and its record. General numerical labs are an additional reference library.","science":[{"title":"Controlled comparisons and confounded improvement","equation":"Synthetic outcome combines a group baseline and treatment contribution; the crude contrast compares observed group means.","explanation":"Baseline outcomes are 2 and 12, with an authored treatment effect of 3. Balanced allocation gives a crude contrast of 3. In the confounded eight-row assignment, the treated mean is 13 and the untreated mean is 2, producing a contrast of 11. The high-baseline group has no untreated rows; its within-group contrast remains unidentified by the observed table.","assumptions":"Finite deterministic authored data with no noise and an additive constant effect. This lesson does not establish a general causal-identification method."}]}