For tabular data, tree-based models — particularly gradient boosting — are usually the right answer. Modern gradient boosting libraries (XGBoost, LightGBM, CatBoost) win Kaggle competitions, power production systems, and outperform deep learning on most tabular tasks.
Knowing when and how to use tree-based models is essential ML knowledge.
Recursively split data on features to predict outcome.
A tree is a series of if-else questions:
Splits are chosen to maximize information gain (or minimize impurity).
Many trees trained on bootstrap samples; predictions averaged.
Each tree:
Result: less variance than single tree, often less bias.
Random Forests are an excellent strong baseline.
Build trees sequentially, each correcting errors of the previous.
Mathematical framing: each tree fits the gradient of the loss on previous predictions.
The original popularizer. Highly optimized; many features.
Microsoft's implementation. Often faster than XGBoost; comparable quality.
Uses leaf-wise growth (vs depth-wise).
Yandex's implementation. Native categorical handling, ordered boosting.
Often best for categorical-heavy data.
All three give similar quality with reasonable tuning.
Reasons deep learning struggles on tabular:
Tabular data is heterogeneous: mix of numeric, categorical, ordinal, with different scales. Trees handle naturally.
Non-smooth target functions: real-world rules are step-like ("if age > 65 AND income < X then..."). Trees represent these directly.
Less data per task: deep learning needs lots of data. Tabular tasks often have 10K-1M rows, where trees excel.
Engineered features matter: domain knowledge encoded as features works well with trees.
Robustness to outliers: trees aren't affected by extreme values like neural networks.
This is why financial, healthcare, and e-commerce ML systems run on gradient boosting.
Common tunables:
Tune from there.
Convert categories to binary columns. Standard.
Map categories to integers. Use only when order is meaningful.
Replace category with mean target value. Risk of leakage; needs cross-validation.
CatBoost and LightGBM handle categoricals natively. Often outperforms manual encoding.
Trees naturally produce feature importance:
Useful for:
Caveat: importance scores can be misleading with correlated features.
Per-prediction feature attribution. Tells you why a specific prediction was made.
For tree-based models, SHAP can be computed exactly and efficiently.
Use for:
Deep learning may beat trees when:
For most tabular ML in industry: trees are still the right answer.
Tune on a single random split; overfit to it. Use CV.
Features that wouldn't be available at prediction time. Common with engineered features.
Deep trees on small data overfit. Reduce depth or get more data.
Regression objective when classification is the task. Subtle but happens.
Majority class dominates. Use class weights, scale_pos_weight, or focal loss variants.
Without early stopping, you have to manually tune number of trees.
Different feature engineering pipelines drift. Use a shared pipeline.
Tree-based models are reliable, fast, and accurate. They should be the default for tabular data unless proven otherwise.