Anomaly Detection: Do Tabular Foundation Models Help?

Data scientists know the struggle. Real-world tabular data is rarely just neat rows of numbers. It is messy, heterogeneous, and often packed with free-text columns like job descriptions, customer comments, or clinical notes. In traditional machine learning workflows, these text columns are usually the first victims of preprocessing and get dropped entirely because they are too complex to handle alongside numeric features.

But text carries massive semantic value. In domains like fraud detection, cybersecurity, or predictive maintenance, anomalies often hide right there in the text. Recently, Tabular Foundation Models (TFMs) have sparked a lot of excitement. Other studies showed that turning these text columns into dense vector embeddings can significantly boost a model’s performance.

There is just one catch. Almost all of this evidence comes from supervised learning. But what happens in unsupervised anomaly detection, where we have no labels to guide the model?

Meet the Models

Five foundation models are at the center of this work. What separates them is not size but how they treat numbers and text.

TabPFN-3 and TabPFN Unsupervised are prior-data fitted networks. They are trained on synthetic data drawn from statistical priors and predict through in-context learning in a single forward pass. TabPFN-3 serves as the supervised control and supplies the internal representations that we later feed to the classical detectors instead of the raw columns, a setup we call the enhanced baselines. The unsupervised variant needs no label and goes straight to work as a detector.

FoMo-0D is the odd one out, in a good way. It is the only model here that was designed for tabular outlier detection from the ground up rather than repurposed from a classifier. It runs zero-shot, and it is by far the fastest model in the comparison, which makes it the natural head-to-head opponent in the unsupervised benchmark.

AnoLLM flips the whole approach. It serializes every row into text and hands it to a language model. Free text survives untouched, numbers lose mathematical precision on the way through, and the compute bill goes up sharply.

ConTextTab sits in between. It pairs the PFN architecture with sentence transformer embeddings and was trained on real tables instead of synthetic priors. It cannot detect anomalies, so it takes on the SHAP attribution and a second supervised control.

Why these five

Each model represents one of the three paradigms in the tabular space, and each paradigm makes a different bet on where the signal lives. PFNs bet on distributions and ignore what a column actually means. LLM-based models bet on semantics and pay for it in numeric precision. Hybrids try to do both. If you only test one paradigm, a negative result tells you nothing about the others.

The second reason is less flattering to the field. The pool of candidates is thin. Most tabular foundation models are either not built for anomaly detection or cannot process free text natively, and this work needs models that do at least one of the two. What looks like a curated selection is, in part, simply the set of models that fit the task.

Together the five span the range between statistical exactness and semantic density. That tension is what the experiments are built to expose.

The two datasets

Everything runs on two tables that were picked because they disagree about how much of the story lives in the text.

Fake Job Postings is the text-rich one. It contains job advertisements with five free-text columns, ships with a native fraud label, and about 4.84 percent of its rows are anomalies.

Airbnb Paris is the more numeric one. It covers listings scraped in September 2025, restricted to a single city so that language and column structure stay consistent, with four free-text columns among 43 attributes. It has no native anomaly label, so we built one from the user ratings by treating a 5.0 as normal and anything at or below 3.0 as anomalous, dropping the middle segment. That is a deliberate but arbitrary operationalization, and it turns out to matter for the results.

One number is worth keeping in your head for everything that follows. With anomaly rates below five percent, a random ranking already scores an AUPRC of roughly 0.05. Every value below is measured against that floor, not against 1.0.

The unsupervised dilemma

In a supervised classification task, the label acts like a compass. It tells the model exactly which part of a highly complex text embedding is relevant for predicting the target.

In unsupervised anomaly detection, we take that compass away. The detector must figure out what is normal and what is anomalous purely based on the geometry of the input space. We wanted to find out whether the hyped Tabular Foundation Models and advanced text embeddings still hold their ground when they are flying blind.

The evaluation runs in three stages. First we benchmark the raw unsupervised performance of TabPFN Unsupervised, FoMo-0D and AnoLLM against classical detectors such as Isolation Forest, ECOD, LODA and an Autoencoder. Then we test whether converting free text into vector embeddings gives those classical algorithms a semantic boost. Finally, we run a supervised control to find out whether any shortcoming belongs to the models or to the missing label.

One detail about fairness is worth stating up front. The baseline detectors were tuned by an exhaustive grid search, while the TFMs ran in their default configurations. That asymmetry works against our own hypothesis, and we kept it deliberately so that nobody can wave the results away as a tuning artifact.

The findings: hype vs. reality

The results provided a clear reality check for the current state of foundation models in anomaly detection.

1. David vs. Goliath: The shallow models still win

There is a common expectation that pointing a large pre-trained model at a hard problem naturally produces superior results. In unsupervised anomaly detection, that expectation did not survive contact with the data. No TFM achieved a consistent advantage over the tuned baselines across both datasets.

On the more numeric Airbnb dataset, the two best models were ECOD and Isolation Forest, ahead of every foundation model. The weakest was FoMo-0D, which is essentially the random floor, despite being the one model built specifically for this task. On the text-rich Fake Job Postings dataset, AnoLLM posted the highest point estimate, although the gap to the tuned baselines stayed within one standard deviation. That value also plays by different rules, because AnoLLM is the only model in this experiment that serialises the row itself and therefore sees the free-text columns, while everyone else works on the text-free numeric table.

Beyond accuracy, the decisive factor for production is compute. The statistical detectors finished in well under three seconds. While autoencoder needed around two and a half minutes, TabPFN Unsupervised took several minutes, and AnoLLM took roughly one and a half hours per run on both datasets. The irony is that the fastest model in the entire comparison is a foundation model, since FoMo-0D returns in about 0.2 seconds. Speed is not the problem with TFMs, the detection quality that would justify the slow ones is.

2. The embedding trap and what actually escapes it

Before encoding any text, we checked the premise. A SHAP attribution on ConTextTab, which treats each free-text column as a single attribute, confirmed that the text carries real predictive value. On Fake Job Postings all five text columns landed in the ten most influential attributes, with the company profile far ahead of the strongest numeric one. On Airbnb the picture is more balanced, but the text columns still rank among the top attributes despite making up a small share of the table.

So the information is there. The question is whether an unsupervised detector can reach it. We built three representation pipelines, namely FastText as an efficient word-vector baseline, the sentence transformer all-mpnet-base-v2 for deeper sentence-level context, and TabPFN internal representations as a diagnostic control. Because high-dimensional embeddings handicap distance-based and density-based detectors, we reduced every variant with an unsupervised PCA before concatenating it with the numeric attributes. The resulting pipeline is shown below:

The Results

The result splits along the semantic density of the dataset. On the text-rich Fake Job Postings, FastText with an aggressive PCA lifts every single one of the four detectors above the text-free reference, and for Isolation Forest, the gain is large enough to survive the noise between runs. On the more numeric Airbnb Paris, the exact same recipe drags every detector down. The sentence transformer barely helps anywhere. And across almost every comparison we ran, cutting the embedding down harder beat cutting it gently, which tells you how much of that vector space these detectors experience as pure noise.

So no representation strategy works across both datasets, and the one that does help does not travel. This is exactly where our unsupervised results part ways with the supervised findings of other papers, where text embeddings improved performance on the benchmark as a whole.

The semi-supervised TabPFN representations, which were fitted on the actual label, sharpen the diagnosis. On Fake Job Postings they beat everything else in the comparison, which proves that the detectors are perfectly capable of exploiting an enriched representation once the information arrives in an accessible form. On Airbnb Paris even that label-bearing representation fails for three of the four detectors. The missing label is therefore not the only obstacle, and the second one turns out to be the label itself.

3. It is not the model, it is the missing label

So are TFMs and text embeddings just inherently bad at this? Not at all. We handed the same models the same data with a target label attached and turned the task into supervised classification.

On Fake Job Postings the performance took off. ConTextTab landed close to a near-perfect score, and TabPFN was not far behind. The representations that looked useless in the dark turn out to be highly informative the moment something points at what matters. That is the core result of this work. The bottleneck is not representation quality; it is the transfer into an unsupervised procedure.

Airbnb Paris tells the other half of the story. Even with full supervision, the models hit a ceiling. They rank the listings well, but they cannot pick out the rare cases with any precision. A label built from star ratings separates the classes in a ranking without separating them in the geometry of the input space, and geometry is the only thing an unsupervised detector has to work with. Where your label is soft, no representation will save you.

Discussion, The Road Ahead

While our findings might read like a setback for TFMs in anomaly detection, they point rather precisely at where the field needs to move.

Native text integration. Instead of patching external embeddings onto tabular data as an afterthought, future architectures need to process free text natively. Models that learn these representations end-to-end during training can capture the semantic context of a table far better than anything bolted on afterwards. ConTextTab, trained on real tables and preserving column semantics, is a plausible starting point.

Objectives built for outliers. Anomaly detection lives with extreme class imbalance, and repurposing a classification model for it is a workaround rather than a design. Our results suggest the derivation itself is the problem, since the representations are good and only their unsupervised use fails. What is missing is a way to extract semantic representations without a label.

The scarcity of unsupervised foundation models. Look at the current landscape, and the gap is obvious. Almost everything is aimed at supervised tasks, which leaves a blind spot exactly where industrial anomaly detection sits.

The dataset disconnect. In open-source research, tabular datasets with meaningful free-text columns and a native anomaly label are close to nonexistent. In corporate practice they are everywhere. That scarcity is why this study runs on two datasets and why one of them needed a constructed label, and it is a bottleneck for the whole field rather than a quirk of our setup.

Takeaways for Your Next Project

1. Do not overengineer. For unsupervised anomaly detection, the classics remain the pragmatic choice. ECOD and Isolation Forest are fast, cheap, and hard to beat, and nothing we tested justified paying an hour of GPU time for the alternative.
2. Embeddings are worth it only where the text is dense. Appending text vectors is not a silver bullet, but it is not useless either. On a table where free text carries a large share of the signal, FastText with an aggressive PCA improved detection. On a mostly numeric table, the same recipe made things worse. Check the semantic density of your data before you invest in a pipeline.
3. Interrogate your label before you blame your model. If a supervised control cannot separate the classes cleanly, your unsupervised detector never had a chance. That check costs an afternoon and saves weeks of tuning the wrong thing.
4. Wait for dedicated architectures. The future for Tabular Foundation Models is bright, but it needs models pre-trained for unsupervised tasks rather than classification models pressed into finding outliers.