For more than a decade, one rule of thumb has held almost everywhere in applied machine learning: if your data lives in tables, use gradient-boosted trees. XGBoost, LightGBM and CatBoost have powered credit scoring, churn prediction, demand forecasting features, fraud detection and countless other business models. Deep learning transformed images, text and speech but repeatedly failed to beat well-tuned trees on ordinary tabular data.
That rule is now being seriously challenged. A new class of tabular foundation models, pre-trained once and then applied to new datasets without conventional training, has posted benchmark results that match or beat tuned tree ensembles on small and medium datasets, often in seconds. This article explains what these models are, what the evidence shows, where they fit in enterprise machine learning, and how to evaluate them without falling for hype.
Why Tables Resisted Deep Learning
Understanding why trees dominated helps explain why TFMs are a genuine shift. Business tables mix numeric, categorical and missing values; individual columns carry very different meanings; relationships are often irregular, with sharp thresholds rather than smooth patterns; and datasets are frequently small. Decision trees handle all of that naturally by splitting on thresholds and ignoring irrelevant features. Neural networks, by contrast, tend to prefer smooth functions, are sensitive to uninformative features and need careful preprocessing and tuning. Every few years a new deep architecture claimed to close the gap, and independent benchmarks usually showed tuned gradient boosting still ahead. TFMs take a different route: rather than training a network on one small dataset, they amortize learning across millions of synthetic datasets, so the network arrives already knowing how tabular patterns usually behave.
What Is a Tabular Foundation Model?
A tabular foundation model (TFM) is a neural network, typically a transformer, pre-trained on a huge number of datasets so that it learns how to learn from tables. When you give it a new dataset, you do not train it in the usual sense. Instead, you pass your labeled training rows as context together with the rows you want predictions for, and the model produces predictions in a single forward pass. This is in-context learning, the same idea that lets large language models follow examples in a prompt, applied to rows and columns.
The best-known family is TabPFN. A paper published in Nature in January 2025, “Accurate predictions on small data with a tabular foundation model,” described TabPFN v2, trained on millions of synthetic datasets generated from simulated causal processes. According to the abstract, on datasets of up to 10,000 samples it outperformed previous methods, and in one classification comparison it beat an ensemble of strong baselines tuned for four hours in 2.8 seconds. The authors also highlighted capabilities beyond prediction, including fine-tuning, data generation, density estimation and reusable embeddings.
How Fast the Field Moved in 2025–2026
Since that paper, the field has expanded rapidly. In his February 2026 overview of the state of tabular foundation models, Christoph Molnar lists more than a dozen models released in 2025 alone, from academic labs and companies including SAP, AWS and others. Several trends stand out:
- Larger datasets. The TabPFN-2.5 technical report (November 2025) states support for datasets of up to 50,000 data points and 2,000 features, a twentyfold increase in data cells over TabPFN v2. Other models such as TabICL target hundreds of thousands of rows, and some research models aim at millions.
- Stronger benchmark results. The TabPFN-2.5 report claims it leads the TabArena benchmark, substantially outperforming tuned tree-based models and matching AutoGluon 1.4, a four-hour tuned ensemble. Against default XGBoost it reports win rates of 100% on small-to-medium classification datasets and around 85–87% on larger classification and regression datasets.
- Deployment options. The same report describes a distillation engine that converts the foundation model into a compact neural network or tree ensemble, keeping most of the accuracy while greatly reducing latency.
- Diverse licensing. Molnar notes that the leading TabPFN version uses a non-commercial license requiring payment for commercial use, while TabICL v2 is fully open source. Licensing is now a real selection criterion.
A healthy caveat applies. Molnar points out that TabArena involves researchers from the TabPFN lab, so independent benchmarks are important when interpreting leaderboard positions. Vendor and lab claims should be validated on your own data.
Tabular foundation models change the economics of building a first model: strong baselines in seconds without tuning. They do not remove the need for good features, honest validation and production discipline.
Where Tabular Foundation Models Shine
- Small and medium datasets. When you have hundreds to tens of thousands of labeled rows, TFMs often deliver strong accuracy without hyperparameter tuning, which is where trees typically need careful tuning to avoid overfitting.
- Rapid prototyping. A credible baseline in seconds lets teams test whether a use case is predictable before investing weeks in feature engineering.
- Many small models. Organizations that maintain a separate model per region, product line, plant or customer segment can replace hundreds of tuning jobs with a single pre-trained model used in context.
- Uncertainty estimates. TFMs produce predictive distributions, which can be useful for risk-aware decisions and active learning.
- Scientific and industrial domains where labeled data is expensive, such as lab experiments, quality inspection outcomes or rare-event engineering data.
Where Gradient Boosting Still Wins
- Very large datasets. With millions of rows, gradient boosting remains fast, cheap and highly accurate, and many TFMs either cannot ingest that much context or become expensive.
- Strict latency and cost budgets at inference. In-context prediction carries the training set at inference time and typically benefits from GPUs. Trees score in microseconds on CPUs. Distillation narrows this gap but adds a step.
- Mature tooling and governance. Model risk teams know how to validate, explain and monitor tree ensembles. SHAP explanations, monotonic constraints and regulatory precedent are well established.
- Commercial licensing simplicity. Open-source tree libraries carry permissive licenses, while some top TFMs require commercial agreements.
- Heavy feature engineering pipelines. Where most of the value lies in domain features and time-aware aggregations, model choice matters less than data work.
Practical Considerations for Enterprise Adoption
Infrastructure
Plan for GPU availability for training-free inference on larger datasets, or for distillation workflows that produce CPU-friendly models. Measure end-to-end latency, including the cost of passing context rows, before committing to real-time use cases.
Data Privacy
In-context learning means training rows are present at prediction time. If you use a hosted API, your training data is sent with each request. Self-hosting or distillation avoids this, which matters for personal or regulated data.
Explainability and Model Risk
Model-agnostic explanation techniques such as SHAP and partial dependence still apply, but validators will want to understand a model whose “training” is a forward pass of a pre-trained network. For high-stakes decisions, consider using TFMs as challengers or for distillation into interpretable models rather than as the production champion until your model risk framework covers them.
Licensing
Check whether the model’s license permits your intended commercial use. Molnar’s overview illustrates the split between models with non-commercial terms and fully open-source alternatives. Treat this like any other third-party software review.
How TFMs Fit Into the ML Lifecycle
It helps to think of tabular foundation models as a new component in an existing lifecycle rather than a new lifecycle. Data preparation, feature engineering, leakage checks and validation design stay exactly as important as before. What changes is the modeling step and, in some cases, deployment.
- Exploration. Use a TFM as an instant baseline to judge whether a target is predictable and which features matter, before committing to a full project.
- Model development. Run the TFM alongside tuned tree ensembles in your experiment tracker, logging the same metrics and artifacts so comparisons are reproducible.
- Deployment. Choose between serving the TFM directly (simpler, usually GPU-backed), distilling it into a lightweight model, or keeping it as a challenger in shadow mode.
- Monitoring. Drift detection, performance tracking and alerting are unchanged. Note that “retraining” a TFM can mean simply refreshing the context rows, which makes updates fast but still requires validation before release.
Quick Decision Checklist
A TFM is worth piloting for a use case when most of the following are true:
- The labeled dataset is in the hundreds to tens of thousands of rows.
- Tuning effort for the current model is significant or repeated across many similar models.
- Batch scoring or moderate latency is acceptable, or distillation is an option.
- The license of the candidate model fits your commercial use.
- Model risk policy allows a challenger or champion built on a pre-trained network.
- You can reproduce production validation exactly, including time-based splits.
A Step-by-Step Pilot Plan
- Pick two or three existing models with modest dataset sizes and clear business metrics, such as a lead-scoring model, a quality-defect classifier or a regional churn model.
- Reproduce the champion’s validation setup, including time-based splits and the exact metric used in production decisions.
- Run one or two TFMs at default settings alongside the tuned champion and a default gradient boosting baseline.
- Compare accuracy, calibration, latency, cost and stability across several random seeds and time windows.
- Test distillation if latency or CPU deployment matters, and measure how much accuracy survives.
- Review explainability and licensing with model risk and legal teams.
- Decide per use case: adopt as champion, use as challenger, use for prototyping only, or reject.
A Hypothetical Worked Example
Consider a hypothetical manufacturer with 40 plants that predicts whether production batches will fail final quality inspection. Each plant has its own model because processes differ, and each dataset contains only a few thousand labeled batches. A small data science team spends much of its time retuning gradient boosting models as processes change.
In a pilot, the team evaluates a tabular foundation model on ten plants using the same time-based validation as production. On most plants, the TFM at default settings matches or slightly exceeds the tuned champions on recall at the operating threshold, with better-calibrated probabilities. On two plants with unusual sensor configurations, the champions remain better. Latency is acceptable for batch scoring at the end of each shift, but the team distills the TFM into small tree ensembles for plants with edge-only compute.
Legal review confirms the chosen model’s license permits commercial use. The team adopts the TFM as the default for new plants and as a challenger for existing ones, reallocating tuning effort to feature work on sensor data. Monitoring and retraining processes stay the same; what changes is how quickly a strong first model can be produced.
Six months later, the team reviews the results with plant managers. The shorter path from new data to a validated model means plants that change suppliers or equipment get an updated model within days instead of weeks, and the data science team has a clearer view of where domain-specific features, rather than model choice, are the real constraint on accuracy.
Pitfalls to Avoid
- Trusting leaderboard results without reproducing them on your data and validation scheme.
- Using random splits for time-dependent problems, which inflates any model’s apparent accuracy.
- Ignoring license terms on commercial use.
- Underestimating inference cost and latency for real-time applications.
- Sending sensitive training data to hosted inference endpoints without a privacy review.
- Abandoning feature engineering because the model “learns everything.”
Outlook: 2027–2030
Expect tabular foundation models to scale to larger datasets, handle time series and relational data more natively, and integrate into AutoML platforms and data science notebooks as default baselines. Distillation will likely become standard, letting organizations use foundation models for development and lightweight models for production. Licensing competition between commercial and open-source models will shape adoption, as will how quickly model risk management frameworks incorporate them. Gradient boosting will not disappear, but it may stop being the automatic first choice for small and medium tabular problems.
Conclusion
Tabular foundation models are one of the most interesting developments in applied machine learning since gradient boosting became dominant. They make strong models available in seconds for the small and medium datasets that dominate many business problems. Treat them as a powerful new option, not a replacement for disciplined ML: validate on your own data, check licensing and privacy, plan for inference costs, and keep investing in features and monitoring. Used that way, they can shorten time to value and free data scientists to focus on the problems that matter most.
How Hanoi Technology Can Help
Hanoi Technology helps data teams benchmark tabular foundation models against existing champions and deploy the best option with sound validation and monitoring.