A decision tree is one of the easiest machine learning models to understand, because it works like a flowchart of questions. It is also the building block of powerful methods such as random forests and gradient boosting, so it is worth learning well.
How a tree makes a prediction
Imagine predicting whether a student will complete a course. The tree might ask: “Attended more than 80% of classes?” If yes, “Submitted the first project?” Each answer leads down a branch until a leaf gives the prediction. You can print the tree and explain every prediction, which business teams appreciate.
How the tree is built
At each step the algorithm tries possible questions on every feature and picks the one that separates the classes best. For classification it measures this with Gini impurity or entropy; for predicting numbers it uses the reduction in error. It repeats the process on each branch until a stopping rule is reached.
The big problem: overfitting
A tree allowed to grow without limits will keep splitting until it memorises the training data, including its noise. It then scores almost perfectly on training data and badly on new data. Control this with:
max_depth: limit how many questions deep the tree can go.min_samples_leaf: require a minimum number of examples in each leaf.- Pruning: remove branches that add little value.
In Python
from sklearn.tree import DecisionTreeClassifier
model = DecisionTreeClassifier(max_depth=4, min_samples_leaf=20, random_state=42)
model.fit(X_train, y_train)
Trees do not need feature scaling, which makes them convenient for a first model.
From one tree to many
A random forest trains many trees on random samples of the data and averages their votes, which reduces overfitting. Learn single trees first, then forests.
Related: overfitting and underfitting explained, logistic regression explained and why accuracy is not enough.
Machine learning models like this one are covered with projects in our Data Science course in Bangalore.
Book a free demo class at our BTM Layout centre, and ask a counsellor for batch details.
