Language and image models dominate the headlines, but a large share of the machine learning that businesses actually run works on tables: customer records, transactions, sensor readings, clinical measurements. On that kind of data, a decades-old family of methods called gradient-boosted decision trees keeps proving hard to beat. This explainer covers what gradient boosting is, why it suits tabular data, what the main comparison studies found, and how newer tabular foundation models are starting to change the picture.
What counts as tabular data
Tabular data is information arranged in rows and columns, where each row is an example (a loan applicant, a patient visit) and each column is a feature with its own meaning and unit: age, income, blood pressure, a category such as country. Unlike pixels in an image or words in a sentence, the columns have no natural order or neighborhood. Swapping the order of two columns changes nothing about the underlying information. Features also mix types: numbers on wildly different scales, categories with many values, and missing entries.
Those properties matter because neural networks gained their reputation on data with strong structure that they can exploit, such as the grid of an image or the sequence of a sentence. Tables offer little of that.
How gradient boosting works
A decision tree asks a series of yes-or-no questions about features (“is income above this threshold?”) and ends in a prediction. A single tree is easy to understand but not very accurate. Boosting builds many small trees in sequence. Each new tree is trained to correct the mistakes of the ensemble so far, and the final prediction is the sum of all the trees’ outputs. “Gradient” refers to the way the method measures the error: it fits each new tree to the gradient of the loss function, the same mathematical idea behind training neural networks that we explain in our piece on gradient descent, but applied to functions made of trees rather than to numeric weights.
Practical libraries made this fast and robust. The XGBoost paper describes a scalable system for tree boosting that handles sparse data and parallel computation. LightGBM, presented at NeurIPS 2017, introduced techniques to make training more efficient on large datasets. CatBoost added dedicated handling of categorical features. All three, along with the gradient boosting implementations in scikit-learn, are widely available and well documented. Our glossary entry on XGBoost gives the short version.
What the comparison studies found
Two widely cited studies examined whether neural networks designed for tables beat boosted trees.
Shwartz-Ziv and Armon, in “Tabular Data: Deep Learning is Not All You Need,” compared several recently proposed deep models with XGBoost across multiple datasets. Their question was whether the deep models should be recommended for tabular problems. In their analysis, the new models did not clearly outperform XGBoost on datasets other than those used in their own papers, and XGBoost needed far less tuning time.
Grinsztajn, Oyallon and Varoquaux, in “Why do tree-based models still outperform deep learning on tabular data?”, assembled a standard set of 45 datasets from varied domains and built a benchmarking method that accounts for both fitting models and searching for good hyperparameters. Their reported result is that tree-based models remain state of the art on medium-sized data, roughly up to 10,000 samples, even without accounting for their speed advantage. They also investigated why, identifying properties of tabular data that favor trees. Read the paper for the full list; the main themes are that tables contain many uninformative features, that the important patterns often do not vary smoothly, and that the columns are not rotation invariant.
Both studies are preprints available on arXiv, and benchmark conclusions depend on the datasets chosen, so they are best read as strong evidence for a default, not a law.
Why trees suit tables
Three intuitions explain much of the gap.
- Irrelevant features are cheap for trees. A tree simply never splits on a useless column. A neural network that mixes all inputs together has to learn to ignore it.
- Sharp thresholds are natural for trees. Many real rules are step-like, such as eligibility above an age or a price breakpoint. Neural networks prefer smooth functions and need more capacity to approximate jumps.
- Feature meaning matters. Rotating a table’s columns, which a fully connected network can effectively do, destroys the meaning of each feature. Trees treat each column on its own terms.
Where deep learning still helps
Deep learning earns its place when tables are only part of the problem. If the inputs include text, images or time series, a neural network can learn representations from the raw signal and combine them with the structured columns. Very large datasets, multi-task settings and the need to share a learned representation across many problems also favor neural approaches. Interpretability expectations vary too: trees offer feature-importance tools, but a boosted ensemble of hundreds of trees is not as readable as a single tree.
Tabular foundation models
A newer line of work tries to bring the pretraining idea from language models to tables. A 2025 paper in Nature, “Accurate predictions on small data with a tabular foundation model,” describes a model pretrained on a large number of synthetic datasets that can make predictions on a new small table in a single forward pass, without training on it. The authors report strong results on small datasets compared with gradient-boosted trees. Whether such models hold up across larger, messier industrial data is an active question, and the paper’s own scope concerns small data, so boosted trees remain the standard baseline.
A practical workflow
For a new tabular problem, a sensible order is: start with a simple baseline such as logistic regression, then try a gradient-boosting library with default settings, and tune only after cross-validation shows the model is worth improving. Compare against a neural or foundation-model approach only if the data includes unstructured inputs or the baseline plateaus. Throughout, guard against leakage between training and test rows, which inflates results far more than the choice of algorithm. For a gentler introduction to the vocabulary, see our beginner’s guide to machine learning.
What to watch next
- Independent evaluations of tabular foundation models on larger, real-world datasets.
- Updates to the benchmark suites used to compare tree and neural methods.
- New releases of XGBoost, LightGBM and CatBoost, which keep adding speed and features.
- Tools that combine tree models with learned embeddings of text or images.
- Standard practice for reporting tuning budgets, which strongly affects benchmark fairness.
Key takeaways
- Gradient-boosted trees build many small trees in sequence, each correcting the previous errors.
- Large comparison studies found tree ensembles still match or beat deep tabular models on medium-sized datasets.
- Trees handle irrelevant features, sharp thresholds and mixed column types well.
- Deep learning is the better choice when tables are combined with text, images or other raw signals.
- Tabular foundation models are promising on small data but have not displaced boosting as the default baseline.
Frequently asked questions
Is XGBoost still worth learning?
Yes. It and similar libraries remain a strong default for tabular problems, and the studies we reviewed show they are hard to beat with less tuning effort.
Why not just use a neural network for everything?
On tables, neural networks often need more tuning and data to match boosted trees, and they offer no built-in advantage because table columns lack the spatial or sequential structure that neural networks exploit.
What is a tabular foundation model?
It is a model pretrained across many datasets so it can predict on a new table with little or no task-specific training, as described in the Nature paper cited above.
