PAPERS-10-004 · Paper — Random Forests

The paper

The paper “Random Forests” is a landmark work that introduced a powerful ensemble method for decision trees, establishing a new baseline for tabular data analysis. It is widely cited and has become a default tool for many data scientists.

Before this paper

Before this work, practitioners relied on single decision trees, which are easy to interpret but highly unstable—small changes in the data could produce very different trees, leading to overfitting. The community needed a way to improve predictive stability without sacrificing interpretability. Existing methods like bagging existed but did not fully address the correlation between trees. Consequently, building accurate, robust models on structured data was challenging.

The key idea

The key insight was to grow many de‑correlated decision trees and combine their predictions, letting the individual errors cancel each other out. By introducing randomness both in the training data and in the feature selection at each split, the trees become diverse. Averaging (for regression) or voting (for classification) then yields a more stable and accurate model. This simple idea turned ensembles into a powerful, easy‑to‑use tool.

How it works

First, a bootstrap sample—drawn with replacement—from the full training set is created for each tree. Then, when growing a tree, at every split the algorithm considers only a random subset of the available features, rather than all of them. This restriction forces each tree to explore different splits, reducing correlation between trees. The tree is grown to full depth (or until a stopping criterion) without pruning. After all trees are built, predictions are made by averaging the outputs of all trees for regression tasks or by majority vote for classification. Additionally, because each bootstrap sample leaves out about one‑third of the data, those out‑of‑bag samples provide a built‑in estimate of prediction error.

See it — a forest votes

flowchart LR
  A["Bootstrap samples"] --> B["Many trees,<br/>random features"]
  B --> C["Each tree votes"]
  C --> D["Average / majority"]

What they showed

The authors demonstrated that random forests are accurate, robust, and resistant to overfitting, while requiring little tuning. They also showed that the method yields useful feature‑importance scores and provides a free out‑of‑bag error estimate from the left‑out samples.

What it changed

This work made bagging and tree ensembles a default baseline for tabular data, changing the standard workflow of data scientists. It showed that a simple ensemble of randomized trees could outperform many more complex models, encouraging widespread adoption of ensemble methods. As a result, random forests quickly became one of the first tools tried on structured datasets, shaping modern machine‑learning practice. Next: Gradient Boosting.

The one-line takeaway

Random forests turn many weak, diverse trees into a strong, reliable predictor with minimal effort.

Read it / meet it

Reading the original paper gives insight into the elegant simplicity behind a method that still powers countless applications today. Its ideas live on in modern libraries and in the everyday practice of data scientists tackling structured data problems.