The mechanism
A regression stump divides one feature at a threshold and predicts a value on either side. An additive ensemble can fit successive stumps to the current residuals, gradually correcting its earlier predictions. The workbench actually searches thresholds between training values, computes residual means in both leaves, chooses the lowest squared residual error and applies shrinkage before the next round. The reserved examples never fit the thresholds or leaf values. This simple squared-loss mechanism illustrates boosting; it does not reproduce the many algorithms and engineering choices in production XGBoost. The separate reversed-relationship challenge exposes why fitting the training pattern does not guarantee robustness to a changed task.
- Initialize predictions at the mean training target and calculate each residual.
- For every candidate threshold, partition training rows, fit left/right residual means and compute squared error.
- Choose the best stump and add its shrunken leaf contributions to the ensemble. Repeat on the remaining residuals.
- Evaluate reserved examples and a separately defined reversed-label relationship with the same fitted ensemble.
Science and assumptions
A tree ensemble that learns residual corrections
Fit a stump to current residuals and add a scaled contribution to the existing prediction function.
Symbols and units
- F_t
- Current additive prediction
target units
- h_t
- Fitted residual stump
target units
- η
- Shrinkage from zero to one
dimensionless
- xᵢ,yᵢ
- Training feature and target
feature / target units
Six training points map negative values to −2 and nonnegative values to 2. The first fitted split is −0.5. With shrinkage 0.5, its contributions are −1 and 1. Four rounds leave a reserved-set MSE of 0.015625, while reversing the target relationship produces MSE 15.015625. Both results come from the same actually fitted stumps.
Assumptions One numeric feature, squared loss, finite candidate splits and constant leaves; no regularized second-order XGBoost implementation or performance claim.
Engineering example
Six training points map negative values to −2 and nonnegative values to 2. The first fitted split is −0.5. With shrinkage 0.5, its contributions are −1 and 1. Four rounds leave a reserved-set MSE of 0.015625, while reversing the target relationship produces MSE 15.015625. Both results come from the same actually fitted stumps.
Distinctions and tradeoffs
A good score on one held-out distribution does not imply resistance to a reversed or otherwise changed input/output relationship.
More rounds can fit more detail, but complexity and learning rate affect generalization and computation. A task baseline should be evaluated under the same data boundaries as a larger model.
AI applications often use structured/tabular prediction. Understanding this mechanism broadens the atlas beyond the assumption that all useful learning is a decoder transformer.
Specifications and interfaces
- Teaching specimen
- Learn a tree ensemble
Six training points map negative values to −2 and nonnegative values to 2. The first fitted split is −0.5. With shrinkage 0.5, its contributions are −1 and 1. Four rounds leave a reserved-set MSE of 0.015625, while reversing the target relationship produces MSE 15.015625. Both results come from the same actually fitted stumps.
- Evidence boundary
- Explicit computed example
One numeric feature, squared loss, finite candidate splits and constant leaves; no regularized second-order XGBoost implementation or performance claim.
Interfaces
- Inputs and convention Input
- Initialize predictions at the mean training target and calculate each residual.
- Computed result and scope Output
- Fit a sequence of regression stumps, inspect their contributions and evaluate both a reserved task and a changed relationship.
Check your understanding
What data choose the stump threshold in this experiment?
- The reversed-relationship challenge labels
- Only the training rows and their current residuals
- The final reported evaluation score
Answer and reasoning
Only the training rows and their current residuals
Only the training rows and their current residuals. One numeric feature, squared loss, finite candidate splits and constant leaves; no regularized second-order XGBoost implementation or performance claim.
Evidence and scope
The workbench executes the declared small mechanism. Numeric examples are authored and independently checked, not measurements from the cited research. One numeric feature, squared loss, finite candidate splits and constant leaves; no regularized second-order XGBoost implementation or performance claim.
Open Mathematical foundations for the bounded executable mechanism and its record. General numerical labs are an additional reference library.