Gradient boosting models, including XGBoost, LightGBM and CatBoost, win many competitions on tabular data and are widely used in industry for tasks like churn prediction and credit scoring. The core idea is simpler than the name suggests.
Forests vs boosting
A random forest builds many trees independently and averages them. Boosting builds trees one after another: each new tree focuses on the mistakes the previous trees made. Together, many small, weak trees form a strong model.
How it works, step by step
- Start with a simple prediction, such as the average value.
- Calculate the errors (residuals) of the current model.
- Train a small tree to predict those errors.
- Add that tree’s predictions to the model, scaled down by a learning rate.
- Repeat for hundreds of trees.
“Gradient” refers to the mathematical method used to decide which errors to fix next, but you can use the model without the calculus.
The settings that matter most
- n_estimators: how many trees.
- learning_rate: how much each tree contributes. Smaller values (0.05 to 0.1) with more trees usually generalise better.
- max_depth: keep trees shallow, often 3 to 6.
- Early stopping: stop adding trees when the validation score stops improving.
In Python
from sklearn.ensemble import HistGradientBoostingClassifier
model = HistGradientBoostingClassifier(learning_rate=0.1, max_depth=4, early_stopping=True, random_state=42)
model.fit(X_train, y_train)
XGBoost and LightGBM offer similar interfaces with extra speed and options.
Watch out for
- Overfitting with too many deep trees and a high learning rate. Always use a validation set.
- Data leakage: powerful models exploit leaked information; check that no feature reveals the answer.
- Explainability: use feature importance or SHAP values when you must explain predictions.
Learn the foundations first: decision trees, then random forests, and overfitting and underfitting.
Machine learning models like this one are covered with projects in our Data Science course in Bangalore.
Book a free demo class at our BTM Layout centre, and ask a counsellor for batch details.
