ML-01-012-P · Primer — Decision Tree Regressor: Piecewise Predictions

The idea in one sentence

A decision tree regressor slices your input space into rectangular regions with yes/no questions, then predicts one flat, constant number inside each region.

Why it exists

Linear regression draws a single straight line through your data — great when the relationship really is a line, but helpless when it bends, jumps, or plateaus. Real-world signals often have thresholds: a house past 3 bedrooms suddenly costs more, a machine past 80°C suddenly fails faster. The decision tree regressor was built to capture exactly these "it depends" patterns without you specifying the shape in advance. It asks simple questions like "is the temperature above 4.5?" and answers with a stored average for whatever group you land in. That makes it easy to read, easy to explain to a non-expert, and a foundational building block for the powerful ensembles (random forests, gradient boosting) you will meet later in the AIM track.

How it works

The tree looks at your training data and searches for the single split — one feature, one threshold — that best separates high-target rows from low-target rows, then repeats the search inside each resulting group. Each split carves the input into cleaner regions; the tree keeps splitting until it hits a stopping rule like max_depth. When it reaches a leaf (a final region), it simply memorizes the average target value of the training points that fell there. At prediction time, a new point answers the yes/no questions, walks down to one leaf, and gets that leaf's stored constant back — this is why the output is a staircase, not a smooth curve. So predictions are piecewise-constant: flat within a region, jumping only at split boundaries.

import numpy as np
from sklearn.tree import DecisionTreeRegressor

# Tiny inline dataset: hours studied -> exam score
X = np.array([[1], [2], [3], [6], [7], [8]])
y = np.array([50, 55, 60, 85, 88, 90])

model = DecisionTreeRegressor(max_depth=1)  # one split -> two regions
model.fit(X, y)

print(model.predict([[2.5], [7.5]]))  # -> [55.  87.66666667]
# Split lands at hours=4.5: low group averages 55, high group averages ~87.67
print("Train R^2:", round(model.score(X, y), 3))  # -> 0.962

The one gotcha

Because each leaf returns a fixed average, a decision tree cannot extrapolate and its prediction never smoothly changes — it is flat, then it jumps. Two consequences bite beginners: (1) feed it an input far beyond your training range (say 100 hours studied) and it will confidently repeat the highest leaf average it ever learned, not a higher number; and (2) if you let max_depth grow too large, the tree makes a tiny region for almost every single training point, memorizing noise — that is overfitting. Control it with max_depth (start small, like 2–4) and watch validation error, not just training error.

See it — one split turns numbers into a staircase

flowchart TD
    A["New point:<br/>hours studied"] --> B{"hours <= 4.5 ?"}
    B -->|"yes"| C["Region 1 leaf<br/>predict 55.0"]
    B -->|"no"| D["Region 2 leaf<br/>predict 87.67"]
    C --> E["Constant number out<br/>(piecewise prediction)"]
    D --> E

The one-line takeaway

A decision tree regressor answers a few yes/no questions and hands you the average of whichever region you land in — simple, readable, piecewise-constant predictions.

Where you'll meet it

You will see DecisionTreeRegressor as the honest baseline for any tabular prediction task — pricing, demand, sensor readings — and, more importantly, as the single "tree" that gets stacked hundreds deep inside random forests and gradient-boosted models, the workhorses of modern applied ML. Master the staircase here, and the ensembles later feel obvious.

Named concepts: DecisionTreeRegressor, region-constant predictions, max_depth for regression