{
  "schemaVersion": 1,
  "id": "atlas-lesson-tree-ensemble-baseline",
  "title": "A tree ensemble that learns residual corrections",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Provides a contrasting learned prediction mechanism that does not use token attention.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/tree-ensemble-baseline/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "contrasting-ai-methods",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "The workbench executes the declared small mechanism. Numeric examples are authored and independently checked, not measurements from the cited research. One numeric feature, squared loss, finite candidate splits and constant leaves; no regularized second-order XGBoost implementation or performance claim.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/tree-ensemble-baseline-ab1703618a0e7742.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#tree-ensemble-baseline",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "a7f9c098b10352f26e60e5f50c72c99f80265fc7c1b172689e91505d47219f35",
  "contentLength": 6164,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-tree-ensemble-baseline.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-tree-ensemble-baseline.md",
  "sections": [
    {
      "id": "section-1",
      "title": "A tree ensemble that learns residual corrections",
      "start": 39,
      "end": 470
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 470,
      "end": 1673
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1673,
      "end": 1701
    },
    {
      "id": "section-4",
      "title": "A tree ensemble that learns residual corrections",
      "start": 1701,
      "end": 2688
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2688,
      "end": 3042
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 3042,
      "end": 3575
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3575,
      "end": 4181
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 4181,
      "end": 4469
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 4469,
      "end": 4498
    },
    {
      "id": "section-10",
      "title": "What data choose the stump threshold in this experiment?",
      "start": 4498,
      "end": 4983
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4983,
      "end": 5456
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5456,
      "end": 5738
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 5738,
      "end": 5761
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 5761,
      "end": 5904
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 5904,
      "end": 6164
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#tree-ensemble-baseline"
    },
    {
      "title": "Chen and Guestrin · XGBoost",
      "url": "https://arxiv.org/abs/1603.02754"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/tree-ensemble-baseline-ab1703618a0e7742.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Controlled comparisons and confounded improvement",
      "url": "https://semiaifoundry.com/learn/atlas/ablation-reproduction-design/"
    },
    {
      "title": "Loss, gradients & optimizer",
      "url": "https://semiaifoundry.com/learn/atlas/loss-optimizer/"
    },
    {
      "title": "Evaluation & the AGI frontier",
      "url": "https://semiaifoundry.com/learn/atlas/evaluation/"
    },
    {
      "title": "Host CPU",
      "url": "https://semiaifoundry.com/learn/atlas/host-cpu/"
    }
  ],
  "recordHash": "676d17a4d58b48429e74d1d0b03e7179ebcb943473af46e7d71a7de3d5aeca8b",
  "contentFormat": "text/markdown",
  "content": "contrasting-ai-methods · Atlas v1.3.0\n\n# A tree ensemble that learns residual corrections\n\nProvides a contrasting learned prediction mechanism that does not use token attention.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#tree-ensemble-baseline)Mathematical and data space; dimensions are not physical distance\n\nLearning objectiveFit a sequence of regression stumps, inspect their contributions and evaluate both a reserved task and a changed relationship.\n\n## The mechanism\n\nA regression stump divides one feature at a threshold and predicts a value on either side. An additive ensemble can fit successive stumps to the current residuals, gradually correcting its earlier predictions. The workbench actually searches thresholds between training values, computes residual means in both leaves, chooses the lowest squared residual error and applies shrinkage before the next round. The reserved examples never fit the thresholds or leaf values. This simple squared-loss mechanism illustrates boosting; it does not reproduce the many algorithms and engineering choices in production XGBoost. The separate reversed-relationship challenge exposes why fitting the training pattern does not guarantee robustness to a changed task.\n\n1.  Initialize predictions at the mean training target and calculate each residual.\n2.  For every candidate threshold, partition training rows, fit left/right residual means and compute squared error.\n3.  Choose the best stump and add its shrunken leaf contributions to the ensemble. Repeat on the remaining residuals.\n4.  Evaluate reserved examples and a separately defined reversed-label relationship with the same fitted ensemble.\n\n## Science and assumptions\n\n### A tree ensemble that learns residual corrections\n\n$$\nF_{t+1}(x)=F_t(x)+\\eta h_t(x),\\quad h_t\\approx\\operatorname*{arg\\,min}_{h}\\sum_i\\big(y_i-F_t(x_i)-h(x_i)\\big)^2\n$$\n\nFit a stump to current residuals and add a scaled contribution to the existing prediction function.\n\nSymbols and units\n\nF\\_t\n\nCurrent additive prediction\n\ntarget units\n\nh\\_t\n\nFitted residual stump\n\ntarget units\n\nη\n\nShrinkage from zero to one\n\ndimensionless\n\nxᵢ,yᵢ\n\nTraining feature and target\n\nfeature / target units\n\nSix training points map negative values to −2 and nonnegative values to 2. The first fitted split is −0.5. With shrinkage 0.5, its contributions are −1 and 1. Four rounds leave a reserved-set MSE of 0.015625, while reversing the target relationship produces MSE 15.015625. Both results come from the same actually fitted stumps.\n\nAssumptions One numeric feature, squared loss, finite candidate splits and constant leaves; no regularized second-order XGBoost implementation or performance claim.\n\n## Engineering example\n\nSix training points map negative values to −2 and nonnegative values to 2. The first fitted split is −0.5. With shrinkage 0.5, its contributions are −1 and 1. Four rounds leave a reserved-set MSE of 0.015625, while reversing the target relationship produces MSE 15.015625. Both results come from the same actually fitted stumps.\n\n## Distinctions and tradeoffs\n\nA good score on one held-out distribution does not imply resistance to a reversed or otherwise changed input/output relationship.\n\nMore rounds can fit more detail, but complexity and learning rate affect generalization and computation. A task baseline should be evaluated under the same data boundaries as a larger model.\n\nAI applications often use structured/tabular prediction. Understanding this mechanism broadens the atlas beyond the assumption that all useful learning is a decoder transformer.\n\n## Specifications and interfaces\n\nTeaching specimen\n\nLearn a tree ensemble\n\nSix training points map negative values to −2 and nonnegative values to 2. The first fitted split is −0.5. With shrinkage 0.5, its contributions are −1 and 1. Four rounds leave a reserved-set MSE of 0.015625, while reversing the target relationship produces MSE 15.015625. Both results come from the same actually fitted stumps.\n\nEvidence boundary\n\nExplicit computed example\n\nOne numeric feature, squared loss, finite candidate splits and constant leaves; no regularized second-order XGBoost implementation or performance claim.\n\n### Interfaces\n\nInputs and convention Input\n\nInitialize predictions at the mean training target and calculate each residual.\n\nComputed result and scope Output\n\nFit a sequence of regression stumps, inspect their contributions and evaluate both a reserved task and a changed relationship.\n\n## Check your understanding\n\n### What data choose the stump threshold in this experiment?\n\n1.  The reversed-relationship challenge labels\n2.  Only the training rows and their current residuals\n3.  The final reported evaluation score\n\nAnswer and reasoning\n\nOnly the training rows and their current residuals\n\nOnly the training rows and their current residuals. One numeric feature, squared loss, finite candidate splits and constant leaves; no regularized second-order XGBoost implementation or performance claim.\n\n## Evidence and scope\n\nThe workbench executes the declared small mechanism. Numeric examples are authored and independently checked, not measurements from the cited research. One numeric feature, squared loss, finite candidate splits and constant leaves; no regularized second-order XGBoost implementation or performance claim.\n\nOpen Mathematical foundations for the bounded executable mechanism and its record. General numerical labs are an additional reference library.\n\n## Sources\n\n-   [Chen and Guestrin · XGBoost](https://arxiv.org/abs/1603.02754)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/tree-ensemble-baseline-ab1703618a0e7742.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Controlled comparisons and confounded improvement](https://semiaifoundry.com/learn/atlas/ablation-reproduction-design/)\n\n### Related lessons\n\n-   [Loss, gradients & optimizer](https://semiaifoundry.com/learn/atlas/loss-optimizer/)\n-   [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/)\n-   [Host CPU](https://semiaifoundry.com/learn/atlas/host-cpu/)"
}