Deep learning has become the public and private face of artificial intelligence. When one talks casually about artificial intelligence with friends at a party, strangers on the street, and colleagues at work, it is almost always on the exciting models that generate language, create art, synthesize music, and so on. Massive and intricately designed deep learning models power most of these exciting machine capabilities. Many practitioners, however, are rightfully pushing back against the technological sensationalism of deep learning. While deep learning is “what’s cool,” it certainly is not the be-all and end-all of modeling.

While deep learning has undoubtedly dominated specialized, high-dimensional data forms such as images, text, and audio, the general consensus is that it performs comparatively worse in tabular data. It is therefore tabular data where those with some distaste, or even resentment, toward deep learning stake out their argument. It was and still is fashionable to publish accepted deep learning papers that make seemingly trivial or even scientifically dubious modifications – this being one of the gripes against deep learning research culture – but now it is also fashionable within this minority to bash the “fresh-minted new-generation data scientists” for being too enamored with deep learning and instead to tout the comparatively more classic tree-based methods as instead the “best” model for tabular data. You will find this perspective everywhere – in bold research papers, AI-oriented social media, research forums, and blog posts. Indeed, the counter-culture is often as fashionable as the mainstream culture, whether it is with hippies or deep learning.

This is not to say that there is no good research that points in favor of tree-based methods over deep learning – there certainly is.¹ But too often this nuanced research is mistaken and taken for a general rule, and those with a distaste for deep learning often commit to the same problematic doctrine as many advancing the state of deep learning: taking results obtained within a generally well-defined set of limitations and willfully extrapolating them in ways irrespective of said limitations.

The most obvious shortsightedness of those who advocate for tree-based models over deep learning models is in the problem domain space. A common criticism of tabular deep learning approaches is that they seem like “tricks,” one-off methods that work sporadically, as opposed to reliably high-performance tree methods. The intellectual question Wolpert and Macready’s classic No Free Lunch Theorem makes us think about whenever we encounter claims of universal superiority, whether this is superiority in performance, consistency, or another metric, is: Universal across what subset of the problem space?

The datasets used by the well-publicized research surveys and more informal investigations showing the success of deep learning over tabular data are common benchmark datasets – the Forest Cover dataset, the Higgs Boson dataset, the California Housing dataset, the Wine Quality dataset, and so on. These datasets, even when evaluated in the dozens, are undoubtedly limited. It would not be unreasonable to suggest that out of all data forms, tabular data is the most varied. Of course, we must acknowledge that it is much more difficult to perform an evaluative survey study with poorly behaved diverse datasets than more homogenous benchmark datasets. Yet those who tout the findings of such studies as bearing a broad verdict on the capabilities of neural networks on tabular data overlook the sheer breadth of tabular data domains in which machine learning models are applied.

With the increase of data signals acquirable from biological systems, biological datasets have increased significantly in feature richness from their state just one or two decades ago. The richness of these tabular datasets exposes the immense complexity of biological phenomena – intricate patterns across a multitude of scales, ranging from the local to the global, interacting with each other in innumerable ways. Deep neural networks are almost always used to model modern tabular datasets representing complex biological phenomena. Alternatively, content recommendation, an intricate domain requiring careful and high-capacity modeling power, more or less universally employs deep learning solutions. Netflix, for instance, reported “large improvements to our recommendations as measured by both offline and online metrics” when implementing deep learning.² Similarly, a Google paper demonstrating the restructuring of deep learning as the paradigm powering YouTube recommendations writes that “In conjugation with other product areas across Google, YouTube has undergone a fundamental paradigm shift towards using deep learning as a general-purpose solution for nearly all learning problems.”³

We can find many more examples, if only we look. Many tabular datasets contain text attributes, such as an online product reviews dataset that contains a textual review as well as user and product information represented in tabular fashion. Recent house listing datasets contain images associated with standard tabular information such as the square footage, number of bathrooms, and so on. Alternatively, consider stock price data that captures time-series data in addition to company information in tabular form. What if we want to also add the top ten financial headlines in addition to this tabular data and the time-series data to forecast stock prices? Tree-based models, to our knowledge, cannot effectively address any of these multimodal problems. Deep learning, on the other hand, can be used to solve all of them. (All three of these problems, and more, will be explored in the book.)

The fact is that data has changed since the 2000s and early 2010s, which is when many of the benchmark datasets were used in studies that investigated performance discrepancies between deep learning and tree-based models. Tabular data is more fine-grained and complex than ever, capturing a wide range of incredibly complex phenomena. It is decidedly not true that deep learning functions as an unstructured, sparsely and randomly successful method in the context of tabular data.

However, raw supervised learning is not just a singular problem in modeling tabular data. Tabular data is often noisy, and we need methods to denoise noise or to otherwise develop ways to be robust to noise. Tabular data is often also always changing, so we need models that can structurally adapt to new data easily. We often also encounter many different datasets that share a fundamentally similar structure, so we would like to be able to transfer knowledge from one model to another. Sometimes tabular data is lacking, and we need to generate realistic new data. Alternatively, we would also like to be able to develop very robust and well-generalized models with a very limited dataset. Again, as far as we are aware, tree-based models either cannot do these tasks or have difficulty doing them. Neural networks, on the other hand, can do all the following successfully, following adaptations to tabular data from the computer vision and natural language processing domains.

Of course, there are important legitimate general objections to neural networks.

One such objection is interpretability – the contention that deep neural networks are less interpretable than tree models. Interpretability is a particularly interesting idea because it is more an attribute of the human observer than the model itself. Is a Gradient Boosting model operating on hundreds of features really more intrinsically interpretable than a multilayer perceptron (MLP) trained on the same dataset? Tree models do indeed build easily understandable single-feature split conditions, but this in and of itself is not really quite valuable. Moreover, many or most models in popular tree ensembles like Gradient Boosting systems do not directly model the target, but rather the residual, which makes direct interpretability more difficult. What we care about more is the interaction between features. To effectively grasp at this, tree models and neural networks alike need external interpretability methods to collapse the complexity of decision making into key directions and forces. Thus, it is not clear that tree models are more inherently interpretable than neural network models⁴ on complex datasets.⁵

The second primary objection is the laboriousness of tuning the meta-parameters of neural networks. This is an irreconcilable attribute of neural networks. Neural networks are fundamentally closer to ideas than real concrete algorithms, given the sheer diversity of possible configurations and approaches to the architecture, training curriculum, optimization processes, and so on. It should be noted that this is an even more pronounced problem for computer vision and natural language processing domains than tabular data and that approaches have been proposed to mitigate this effect. Moreover, it should be noted that tree-based models have a large number of meta-parameters too, which often require systematic meta-optimization.

A third objection is the inability of neural networks to effectively preprocess data in a way that reflects the effective meaning of the features. Popular tree-based models are able to interpret features in a way that is argued to be more effective for heterogeneous data. However, this does not bar the conjoined application of deep learning with preprocessing schemes, which can put neural networks on equal footing against tree-based models with respect to access to expressive features. We spend a significant amount of space in the book covering different preprocessing schemes.

The objective of this book is to substantiate these claims by providing the theory and tools to apply deep learning to tabular data problems. The approach is tentative and explorative, especially given the novel nature of much of this work. You are not expected to accept the validity or success of everything or even most things presented. This book is rather a treatment of a wide range of ideas, with the primary pedagogical goal of exposure.

This book is written toward two audiences (although you are more than certainly welcome if you feel you do not belong to either): experienced tabular data skeptics and domain specialists seeking to potentially apply deep learning to their field. To the former, we hope that you will find the methods and discussion presented in this book, both original and synthesized from research, at least interesting and worthy of thought, if not convincing. To the latter, we have structured the book in a way that provides sufficient introduction to necessary tools and concepts, and we hope our discussion will assist modeling your domain of specialty.

- - -