I Grinded Trees Until I Could See the Forest

I started with statistics and classical machine learning, and that was the part of the subject I loved first. I could sit with the maths, understand why a model had made a decision, and then try to code a rough version myself. Deep learning came much later for me. During college, if the data arrived in a CSV, I would almost always try a tree first. I grinded tree-based methods much more than the syllabus ever asked me to.Trying something new.

At first it was mostly assignments. I used random forests all the time, tried boosted trees whenever I wanted to squeeze out a better score, and eventually implemented Isolation Forest from scratch because I wanted to know what the library was doing. I remember that implementation better than most of the coursework. Writing the splits and path-length logic myself was when trees properly clicked for me.

By the time I got serious about neural networks, I already had years of this stuff in my head. Deep learning gave me architectures, activations, optimizers, and loss curves to worry about. Trees still felt more immediate: pick a feature, choose a threshold, look at the two groups, and repeat. On tabular data, they were usually the first baseline I trusted.

That is really where the affection comes from. Trees work, and I can follow what they are doing.

Calling all of these tree-based methods can make them sound more alike than they are. A single decision tree, a random forest, and XGBoost are not just three strengths of the same model. Forests try to average away the instability of individual trees. Boosting keeps returning to what the current model got wrong. XGBoost makes that process more careful with curvature and regularization, while CatBoost pays particular attention to categorical data and leakage. They are related, but they are solving different problems.

My earlier Rotation Forest post was about changing the coordinate system so axis-aligned trees could see oblique structure. This essay starts one level lower. I want to build the whole family from the first split: why the split is greedy, why a tree memorizes, why averaging works, why boosting is not averaging, why modern libraries are so fast, and why the phrase interpretable model deserves more suspicion than it usually gets.

The central idea is almost embarrassingly small: a tree learns by repeatedly asking which single question makes the answers on either side less mixed. Everything else is an argument about how to make that greedy habit useful.

How the forest grew

I used to remember these methods as a list of names from different lectures. The history is much easier to hold onto once the arrows are visible. One useful starting point is Morgan and Sonquist's 1963 work on Automatic Interaction Detection. It recursively split survey data to reduce prediction error, which already contains the basic instinct of a regression tree.[13] At roughly the same time, Hunt, Marin, and Stone were studying computer programs that learned concepts through tree-shaped questions.[14]

Those statistical and machine-learning threads produced CART and ID3 in the 1980s. The next wave did not replace the tree. It kept the tree and changed how many were fitted, what each one saw, and how their answers were combined. That is the lineage I care about in this essay.

A tree begins as one question

Imagine a tiny classification problem. Each row is a student; the columns record attendance, hours studied, sleep, and perhaps whether the student submitted the assignments. The target says whether the student passed. We could begin with any column and any threshold. Attendance above 72 percent? Study time below four hours? All assignments submitted?

A standard CART-style tree searches those candidate questions and chooses the split that most reduces impurityImpurity is a numerical measure of how mixed the labels inside a node are. A pure node contains only one class; a maximally mixed binary node is half and half. It is not moral impurity, despite the vocabulary making every tree sound faintly Victorian.. Then it forgets all the alternatives it did not choose, moves to each child node, and repeats. This is a greedy procedureGreedy means choosing the best move available now without jointly optimizing the entire future tree. Finding the globally smallest optimal decision tree is computationally hard; practical algorithms accept local choices and control the damage later.: it chooses the best immediate improvement, not the best complete tree among every tree that could ever be grown.[1]

For binary classification, the Gini impurity at a node is

where \(p\) is the fraction of class-one examples. If every row has the same label, \(p\) is zero or one and \(G=0\). If the node is split evenly, \(p=1/2\) and \(G=1/2\), its largest value. For more than two classes, the same idea becomes

A candidate split divides the rows \(S\) into left and right subsets. The tree compares the parent's impurity with the size-weighted impurity of the children:

The winning question has the largest \(\Delta G\). Entropy can replace GiniEntropy is \(-\sum_k p_k\log p_k\). It comes from information theory and measures uncertainty in the labels. Gini and entropy curve a little differently, but on ordinary classification problems they often choose very similar splits.:

For regression, labels are not categories, so mixing means spread. CART usually chooses the split that minimizes the sum of squared deviations within the two children. Each leaf then predicts the mean response of its training rows. Change the loss and the leaf changes: absolute-error trees naturally use medians; quantile losses target quantiles; survival trees need a risk-set-aware criterion. The branches look similar because the important choice is hidden in the objective.

Move the split yourself

The formula becomes less abstract when the threshold moves. In the little dataset below, outlined blue circles are class A and black squares are class B. Drag the threshold. The best-looking split is the one that leaves the least mixed pair of children, not necessarily the one that puts the line through the largest empty gap.

Two details from this tiny exercise survive all the way to industrial boosting systems. First, a feature's numeric scale is almost irrelevant: sorting values and trying thresholds gives the same partitions after any monotone rescaling. Second, the procedure cares about local separation, not about a global coefficient. The meaning of “high attendance” can change after an earlier branch has already established that study time is low.

Every path is a region

A root-to-leaf path is a conjunction of rules. If a row travels through \(x_1<4.5\), then \(x_2\geq72\), then \(x_3<6\), its leaf represents the rectangular region where all three statements hold. In \(d\) dimensions the rectangles become hyperrectangles, but the logic does not change.

The fitted regression tree can be written as

where the leaves define disjoint regions \(R_m\), \(c_m\) is the prediction in region \(m\), and the indicator switches that prediction on when \(x\) lands there. This makes a tree a piecewise-constant functionPiecewise constant means the prediction is flat inside each region and jumps at a boundary. A single regression tree does not draw a smooth slope. An ensemble can make the jumps tiny enough to approximate one.. Classification leaves store class counts or probabilities instead of a mean, but they are still constant until a boundary is crossed.

I find this representation useful because it dissolves some mystery. Trees discover interactions automatically because later questions are conditional on earlier ones. A sleep threshold can matter only for low-attendance students without anybody manually adding an interaction term \(\text{sleep}\times\text{attendance}\). The interaction is the path.

The limitation is equally geometric. Ordinary splits are axis-aligned. A clean diagonal boundary like \(x_1+x_2>0\) becomes a staircase of rectangles. A deeper tree can approximate the diagonal, but it spends leaves to do it. This is the reason oblique trees optimize linear combinations at nodes, and the reason rotating the features can turn a complicated staircase back into one easy cut.

There is also no natural notion of distance. Two rows can be numerically close yet fall on opposite sides of an early threshold and receive different predictions. Two distant rows can land in the same leaf and become identical to the model. The learned topology is made of shared paths, not Euclidean neighborhoods.

Why one tree remembers too much

If we keep splitting until every leaf is pure, a tree can often memorize the training set. One mislabeled row gets its own tiny rectangle. A measurement error becomes a branch. A rare coincidence becomes a rule delivered with absolute confidence. Training error falls toward zero while the boundary acquires hundreds of brittle corners.

This is the classic high-variance problemVariance here means sensitivity to the training sample. If replacing a handful of rows produces a very different predictor, the method has high variance. Deep trees are famous for it because early split changes reroute everything below them.. A small perturbation in the data can change the root split. Once the root changes, every later split sees a different subset, so the whole tree can reorganize. The model is not merely wiggly; it is structurally unstable.

We can stop growth in advance with maximum depth, minimum leaf size, minimum split gain, or a maximum number of leaves. Or we can grow a large tree and prune it afterward. CART's cost-complexity pruning chooses a subtree \(T\) by balancing fit against leaf count:

Here \(R(T)\) is the training loss of the leaves and \(|T|\) is their number. At \(\alpha=0\), the biggest tree is attractive. As \(\alpha\) rises, a new leaf must earn its existence by reducing enough error. Cross-validation chooses the trade-off.

Pruning is one answer to instability: make one tree less ambitious. Ensembles take the more interesting answer: let trees remain unstable, then arrange their errors so they cancel.

Bag the unstable thing

Bagging begins with a wonderful act of statistical mischief. From a training set of \(n\) rows, draw \(n\) rows with replacement. Some appear several times; about \(36.8\) percent are absent. Fit a deep tree to that bootstrap sampleBootstrap sample. Drawing \(n\) times with replacement from \(n\) rows leaves a particular row out with probability \((1-1/n)^n\), which approaches \(e^{-1}\approx0.368\). The resample has the same size but a different empirical world.. Repeat hundreds of times. Average regression predictions or vote on classes.[2]

Bagging works because each tree reacts differently to its altered dataset. The wild local errors are not identical, so averaging smooths them. If the trees were independent and each had variance \(\sigma^2\), averaging \(B\) of them would reduce variance to \(\sigma^2/B\). Trees trained on related samples are not independent. If their pairwise error correlation is roughly \(\rho\), the variance of their mean is approximately

The second term shrinks with more trees. The first does not. Once \(B\) is large, adding yet another nearly identical tree buys almost nothing. This equation is the soul of Random Forest: preserve trees that are individually useful while reducing \(\rho\), their shared mistakes.

Breiman's Random Forest does that by hiding most features at each split.[3] A classification tree might be offered only \(\sqrt d\) of the \(d\) columns. If one feature is overwhelmingly strong, ordinary bagged trees all choose it near the root and become variations of the same model. Feature subsampling occasionally removes that obvious answer and forces a tree to discover a second route. Some individual trees weaken. The committee becomes stronger because its members stop making the same errorDecorrelation does not mean the trees become statistically independent. It means their prediction errors become less aligned. Ensemble diversity matters only when the alternative members remain competent enough to be worth averaging..

The rows a tree never saw

Those absent bootstrap rows give bagged forests an almost free validation mechanism. For each training row, collect predictions only from trees whose bootstrap sample excluded it. Aggregate them and compute an out-of-bag scoreOut-of-bag (OOB) predictions are out-of-sample for each individual tree, though not a replacement for a truly untouched final test set. They are especially useful for quick model comparison and permutation importance inside a forest.. It resembles cross-validation without explicitly fitting \(K\) separate forests.

Extremely Randomized Trees push the diversity lever further.[4] Instead of exhaustively finding the best threshold among the offered features, they sample thresholds and select among those random candidates. Less optimization per tree means more bias, less variance, and often faster fitting. Rotation Forest changes the view instead: rotate feature subsets, then grow ordinary trees.[12] These are three answers to the same design question: where should disagreement between competent trees come from?

One operational detail is worth saying plainly. More trees usually do not make a random forest overfit in the way more depth makes one tree overfit. Once the ensemble is large enough, error tends to plateau because we are estimating its average more precisely. Training cost and memory keep rising, though, and a forest of ten thousand trees is not a personality trait.

Boosting changes the conversation

Bagging trains trees in parallel and asks them to disagree. Boosting trains trees in sequence and asks each one to repair the current model. The difference is not cosmetic. A forest averages complete predictors:

A boosted model adds small corrections:

The learning rate \(\eta\) makes each new tree whisper rather than shout. No tree needs to solve the whole task. It needs only to point toward an improvement from where the ensemble currently stands.

AdaBoost: pay more attention to the embarrassment

AdaBoost's story is easiest in classification. Start with equal weight on every training row. Fit a weak classifier, often a stump with one split. Increase the weight of rows it misclassified and decrease the weight of rows it got right. Fit the next learner to the reweighted data. At the end, combine learners using votes weighted by their accuracy.[5]

If labels are \(y_i\in\{-1,+1\}\) and a weak learner is \(h_m(x)\), its weighted error is \(\varepsilon_m\), and its vote is

The row weights update as

Correctly classified rows have positive \(y_i h_m(x_i)\) and shrink in weight; mistakes grow. The final sign of \(\sum_m\alpha_m h_m(x)\) decides the class. More deeply, AdaBoost increases the classification marginMargin is the signed confidence \(yF(x)\). A positive margin means a correct prediction, and a large positive margin means the ensemble is correct with room to spare. Boosting often keeps improving test error after training error reaches zero because it continues widening margins.: not just whether a row is correct, but how far its weighted vote lies from the decision boundary.

The weakness is visible in the update. A mislabeled or pathological row can keep accumulating weight because the ensemble cannot satisfy it. Later learners contort themselves around that one impossible demand. Robust losses and regularization soften this behavior, but the lesson is general: focusing on mistakes is powerful only when mistakes contain signal.

Gradient boosting: fit the direction the loss wants

Friedman's gradient boosting machine made the idea much broader.[6] Treat the current predictor \(F(x)\) as a point in a space of functions. Ask which direction would reduce the loss fastest. Fit a tree to approximate that direction. Add it to the model. This is gradient descent, except the parameter being updated is a functionFunctional gradient means differentiating the objective with respect to the model's output function rather than a fixed vector of coefficients. The fitted tree projects that ideal direction onto the set of functions a small tree can express..

For training pairs \((x_i,y_i)\) and loss \(L(y,F(x))\), compute pseudo-residuals

Fit a regression tree \(h_m\) to \((x_i,r_{im})\), choose the best step size for its leaves, and update

With squared error, \(L=(y-F)^2/2\), the negative gradient is simply \(y-F\), the ordinary residual. The next tree predicts what the ensemble still gets wrong. With logistic loss, pseudo-residuals encode a classification probability error. With a quantile loss, they become asymmetric signs. Same boosting machinery, different definition of “please repair this.”

Tree depth now means something different from tree count. A depth-one stump can express one interaction-free correction. A depth-two or depth-three tree can capture small conditional interactions. Hundreds of shallow corrections often generalize better than a few heroic deep ones. The learning rate and number of trees trade against each other: smaller steps usually need more rounds but can trace a more careful path.

This is shrinkageShrinkage multiplies every new tree by a learning rate below one. It regularizes the stagewise path. A tiny rate is not automatically better: it can require huge ensembles, and with enough rounds even small steps can overfit., and it is one reason early stopping matters. Monitor a held-out fold, stop when its loss has not improved for a patience window, and keep the best iteration. The useful number of boosting rounds is learned from validation, not chosen because 1,000 looks serious.

Why XGBoost feels like Newton's method in a forest

XGBoost did not invent gradient boosting. Its contribution was to turn a beautiful statistical idea into a regularized, sparse-aware, cache-conscious system that could be trusted on large real tables.[7] The mathematical signature is a second-order approximation to the objective.

At boosting round \(t\), let \(g_i\) and \(h_i\) be the first and second derivatives of the loss with respect to the current prediction for row \(i\). For a proposed tree with leaves \(j=1,\ldots,T\) and leaf weights \(w_j\), the approximate regularized objective becomes

where \(G_j=\sum_{i\in I_j}g_i\) and \(H_j=\sum_{i\in I_j}h_i\). The best weight for that leaf has a closed form:

The HessianHessian usually means the matrix of second derivatives. Here each training prediction is a scalar, so \(h_i\) is the corresponding second derivative of the loss. It tells the update how sharply the loss curves around the current prediction. in the denominator makes this Newton-like: when the loss is sharply curved, the step is cautious. The \(\lambda\) term shrinks leaf weights; \(\gamma\) charges for adding a leaf. A candidate split is accepted only if the reduction in approximate loss pays those costs.

Substituting the best leaf weights back into the objective gives the score used to compare a parent leaf with two proposed children:

The engineering matters as much as the derivation. Exact split search sorts every value, which is expensive at scale. Histogram boosters bin continuous values and accumulate gradients per bin. A threshold can then be evaluated over perhaps 255 bins instead of millions of distinct numbers. Missing values can learn a default branch. Sparse columns can skip absent entries. Parallel prefix sums and careful memory layout make an algorithm built from if-statements behave like serious systems software.

LightGBM and CatBoost solve different bottlenecks

LightGBM made histogram training aggressively efficient and grows trees leaf-wise: split the leaf with the largest current gain instead of expanding every level symmetrically.[8] This can reduce loss quickly, but on small data it can create deep, narrow branches that memorize. Parameters such as number of leaves and minimum data per leaf often matter more than a ceremonial maximum depth.

CatBoost begins from the categorical-feature problem.[9] Suppose we replace each category with the mean target for that category. If we compute the mean using all rows, each row's own label leaks into its feature. Rare categories can practically carry the answer. If we compute encodings on one fixed holdout, we waste data and create distribution mismatch.

CatBoost permutes the data and computes an ordered target statisticOrdered target statistic. For a row at position \(i\) in a random permutation, encode its category using only earlier rows with that category, plus a prior. The row's label cannot predict itself because, in this artificial time, it has not happened yet. for each row using only rows that came earlier in a random ordering. Its ordered boosting machinery applies the same instinct to residual construction, reducing a subtle prediction shift between training examples and unseen data. This is not simply “XGBoost but categories.” It is an algorithm built around leakage control.

For a category value \(x_i\), one simplified version of that statistic is

where \(p\) is a global prior and \(a\) controls how strongly we trust it. The restriction \(j<i\) is the whole trick: a row may consult matching categories in its artificial past, never its own target or a future target.

Why tabular data keeps inviting trees back

There is now a small industry devoted to announcing that deep learning has conquered tabular data, followed by a benchmark in which tuned boosted trees remain irritatingly competitive. The fairest reading is not “neural networks cannot learn tables.” They can. The question is which inductive bias spends data most efficiently on the kinds of tables people actually have.

Real tables mix continuous measurements, integer counts, binary flags, missingness patterns, skewed money columns, bounded ratios, dates, and categories. Feature meanings do not usually repeat across columns the way edges repeat across an image or tokens repeat across text. A convolution can share one detector across every location because pixels have a stable local geometry. What should be shared between age, postcode, number of late payments, and browser family?

Trees make fewer demands:

• They are insensitive to monotone scaling. Replacing income with its logarithm changes threshold locations but not the order of rows. Standardization is usually unnecessary.
• They localize interactions. A feature can matter only inside one subgroup, which is common in policy, medicine, credit, churn, and advertising.
• They ignore irrelevant columns reasonably well. A useless feature rarely wins a split repeatedly, although enough noise and depth can still manufacture gains.
• They handle irregular decision surfaces. Threshold effects and saturation are natural rather than forced through smooth activations.
• They are strong in the medium-data regime. When there are thousands or hundreds of thousands of rows rather than billions, their bias can be a gift.

Grinsztajn, Oyallon, and Varoquaux tested this more systematically and found that tree-based models remained strongest on typical medium-sized tabular benchmarks, with neural networks particularly challenged by uninformative features and irregular target functions.[10] That is a benchmark result, not a law. Very large tables, transferable embeddings, multimodal inputs, differentiable end-to-end systems, and pretraining can change the answer.

There is another reason trees feel good in applied work: the baseline is fast to falsify. I can fit a sensible CatBoost or histogram-boosting model, inspect validation slices, and learn whether the table contains predictive signal before building a cathedral around it. A good baseline is not merely a number to beat. It is an instrument for finding out whether the project deserves another week.

A split you can read is not automatically an explanation

A small decision tree is globally inspectable. I can print every path and tell you the complete fitted rule. A forest or boosted ensemble with thousands of leaves is different. Each component is made of readable if-statements, but the sum is no more readable by eye than a novel becomes simple because every sentence uses familiar words.

The first trap is built-in impurity importance. Add up how much each feature reduced the split criterion, and features with many possible thresholds tend to receive more opportunities to look useful. Continuous measurements and high-cardinality identifiers can beat a genuinely predictive binary feature through sheer audition count. Training-set importance also rewards overfit splits.

Permutation importance asks a better predictive question: if I scramble this column in held-out data, how much does performance fall? But correlated features complicate it. If two columns carry the same information, scrambling one leaves its twin intact and makes both look dispensable. Scrambling a feature can also create impossible rows, such as a pregnant value attached to an incompatible demographic or yesterday's balance detached from today's balance.

Partial dependence averages predictions while varying one feature. Individual conditional expectation keeps one line per row. Accumulated local effects avoid some extrapolation into empty combinations. All are descriptions of the fitted model under interventions we choose; none automatically identifies a causal effect.

TreeSHAP efficiently decomposes a tree ensemble prediction into feature contributions based on Shapley valuesShapley value. Borrowed from cooperative game theory, it averages a feature's marginal contribution over possible coalitions of other features. The mathematical allocation is exact for a chosen value function; the causal meaning still depends on how “missing” features are modeled..[11] The contributions add to the difference between the prediction and a baseline, which is genuinely useful for debugging and case-level explanation. But correlated variables force a choice about what it means for a feature to be absent. Different background distributions can allocate credit differently while explaining the same prediction.

My current rule is to triangulate. Use held-out permutation importance for reliance, SHAP or path contributions for individual debugging, partial-dependence-style plots for shape, and subgroup error analysis for consequences. If the decision is high stakes, add domain review and a causal design. One colourful beeswarm plot is not due process.

Where the forest ends

Trees do not extrapolate

Suppose a regression tree has seen temperatures only from 10 to 35 degrees. Beyond the largest training value, every hotter point follows the same rightmost branches and lands in the same leaf. Its prediction goes flat. A boosted ensemble adds many flat functions; outside all learned thresholds, the sum is still flat.

This is extrapolationExtrapolation means predicting beyond the support of the training inputs. Interpolation fills gaps among examples already represented. Trees are excellent interpolators of irregular structure and poor at continuing trends beyond observed thresholds., and it matters whenever the trend itself is the object: growth curves, physical response, demand under unprecedented prices, or climate regimes outside the sample. A linear or mechanistic component may be safer, perhaps with trees modeling residual deviations.

Smooth structure is expensive

An axis-aligned tree approximates circles, rotations, and smooth additive relationships with stairs. Enough depth or boosting rounds can make the stairs fine, but a model that already encodes smoothness may need far less data. Kernel methods, generalized additive models, linear models with good transformations, or neural networks can be the better bias.

Images, audio, and language also have reusable structure. A tree sees a million pixels as a million unrelated columns unless somebody engineers the locality. Deep models learn shared representations, and pretraining lets one example teach many tasks. Throwing raw text embeddings into boosted trees can be an excellent downstream baseline; asking a forest to discover syntax from token IDs is not.

Probabilities need inspection

A leaf probability is a fraction of training labels, then bagging or boosting transforms and averages such quantities. Ranking can be excellent while the numbers are overconfident or underconfident. CalibrationCalibration asks whether events predicted with probability 0.7 happen about 70 percent of the time. It is distinct from discrimination: a model can rank positives beautifully while assigning unreliable probabilities. should be checked with reliability diagrams and a proper scoring rule such as log loss or the Brier score, preferably on data not already used for early stopping. Platt scaling or isotonic regression can help if fitted on a separate calibration set.

Finally, trees do not repair the data-generating process. Leakage, time travel, selection bias, concept drift, censored outcomes, and label policy all survive a beautiful validation score. Because boosters are so good at finding weak shortcuts, they can make a flawed experiment look unusually convincing.

The order I actually try things

After grinding these methods for years, my tree workflow has become less heroic and more boring. That is a compliment.

1. Fix the split before the model. If deployment predicts the future, validation must move forward in time. If rows share users, households, devices, patients, or matches, group them. No ensemble can cross-validate away leakage.
2. Fit a small tree. Not because it will win, but because it exposes suspicious thresholds, missingness shortcuts, and target bugs in a form I can read.
3. Fit a Random Forest or Extra Trees baseline. They are forgiving, parallel, and reveal whether nonlinear interactions matter without a delicate boosting schedule.
4. Fit one serious booster. CatBoost is my first choice for messy categories; histogram gradient boosting, LightGBM, or XGBoost are natural for mostly numeric large tables. I use early stopping and tune leaf complexity before obsessing over tiny regularization differences.
5. Compare under the real metric and budget. PR-AUC for rare positives, pinball loss for quantiles, calibration for decisions, latency and model size for production. Accuracy is not a diplomatic solution to incompatible costs.
6. Inspect slices and stability. I vary seeds, folds, and time windows; examine subgroup errors; check whether important features survive perturbation; and compare train-validation gaps.
7. Earn complexity. Stacking, target encoding, monotonic constraints, custom objectives, and neural hybrids enter only after a clean baseline shows what they must improve.

What this story does not say

It does not say trees always win on tables. “Tabular” is a storage format, not a single statistical regime. A table can contain ten billion examples, images embedded into vectors, time-dependent panels, spatial fields, or repeated measurements with known structure. The best model depends on which regularities deserve to be shared.

It does not say scaling is literally irrelevant. Threshold partitions survive monotone transforms, but histogram binning, numerical precision, regularization, distance-based preprocessing, and interpretation can still change. Extreme outliers can also consume bins and make plots useless.

It does not say feature importance explains the world. It explains some aspect of one fitted predictor under one perturbation or allocation rule. Causality needs assumptions about how the data were generated, not merely an ensemble with good test AUC.

And it does not say one library is universally best. XGBoost, LightGBM, CatBoost, scikit-learn's histogram boosting, Random Forest, and Extra Trees occupy overlapping but distinct engineering trade-offs. A fair comparison holds folds, search effort, and compute budgets steady.

The model that kept teaching me how models think

I began with trees because they worked. I stayed because every extension teaches a different machine-learning lesson without hiding the mechanism.

A single tree teaches greedy optimization and the bias-variance trade-off. Pruning teaches that a model should pay rent for complexity. Bagging teaches that instability can be averaged into strength. Random Forest teaches that diversity is useful only when it decorrelates competent errors. Extra Trees teaches that less optimization can generalize better. Rotation Forest teaches that coordinates decide what a simple learner finds simple. AdaBoost teaches that errors can be turned into attention. Gradient boosting teaches that a model can descend through function space. XGBoost teaches that curvature and systems engineering both matter. CatBoost teaches that the order in which information becomes available is part of the algorithm.

That is an absurd amount of machine learning hiding inside repeated if-statements.

The college grinding mattered because it changed what I saw. At first I saw a catalogue: decision tree, random forest, AdaBoost, gradient boosting, XGBoost. Separate chapters, separate exam questions. Eventually I saw one continuous argument. A tree is expressive but unstable. Average it. The averages are too correlated. Randomize the features. Parallel correction leaves bias. Correct sequentially. First-order corrections ignore curvature. Use second derivatives. Categories leak target information. Give the rows an artificial past.

Every method is the previous method noticing one of its own bad habits.

That may be why I still reach for trees whenever a new table lands in front of me. They do not ask me to believe that scale will rescue a vague problem. They force concrete questions. Which threshold? Which rows? Which loss? Which interaction? Which mistakes are shared? Which information was available at prediction time?

I grinded tree-based methods in college because I loved how often they won. I love them now because I understand more clearly how they can lose.