Machine Learning and Deep Learning: Deeper Is Not Always Better

How did an older system tell spam from ordinary mail? An engineer wrote the rules: count the promotional words, measure the density of links, look at the sender. Those "features" are the measuring rule, and the machine only measures with it. When people say today that deep learning is "a branch of machine learning", they are right, but the phrase hides what most separates the two. For a primer on what AI means, see the article "What Is Artificial Intelligence, Really? A Plain Explanation Without the Hype".

Who makes the features?

In 1959, in his study of the game of checkers, Arthur Samuel wrote that programming computers to learn from experience should eventually remove much of the need for detailed programming. That is machine learning at its origin. Yet for decades it was held back by an obstacle that LeCun, Bengio and Hinton named in their well-known 2015 review in Nature: conventional techniques were limited in processing natural data in raw form, and building a system required careful engineering of a feature extractor, with skill and domain expertise.

What does "depth" do?

In deep learning the network takes over that engineering: it learns representations at increasing levels of abstraction through multiple layers. The face is the usual illustration: early layers pick up edges and gradients of light, higher ones combine them into corners and shapes, until something like an eye or a nose emerges. The example is approximate, for illustration, not a literal description of every network. As for the word "deep", it means the number of these layers, not depth of understanding.

When is the deeper not better?

The prevailing impression is that deep learning came to displace what preceded it. The evidence is more complicated:

Tables and medium-sized data. In 2022 Grinsztajn and two colleagues compared methods on 45 varied tabular datasets and concluded that tree-based models such as XGBoost and random forests remain among the strongest on medium-sized tabular data (about ten thousand samples). Their finding about this kind of data does not extend to every field; images, sounds and long texts are deep learning's territory.

Shortcuts that pass the test. In their "Shortcut Learning" paper, Geirhos and colleagues showed that networks may adopt decision rules that succeed on usual benchmarks and fail under harder conditions, such as real-world use. This returns us to the root of the problem: the more we hand feature extraction to the machine, the harder it is to know what its judgement rests on.

Sensitive decisions. In her 2019 paper Cynthia Rudin argues that models interpretable by design should be used for high-stakes decisions such as health and criminal justice, instead of explaining a "black box" after the fact, and she holds that after-the-fact explanation may entrench bad practice. This view has opponents among researchers, but it reminds us that a decision touching a person needs a justification that can be stated. For the black box and its biases, see the article "Algorithmic Bias".

Cost. In 2019 Strubell and two colleagues estimated the financial and environmental cost of training several language models, and recommended reducing these costs and making research accessible to a wider range of researchers. The deeper tool has a price beyond its purchase price.

Four questions before you choose

1. What kind of data is it? Large images, sounds and texts lean towards deep; organised tables lean towards trees.
2. How much do I have? With little data, deep learning may not give you what justifies its complexity.
3. What does an error cost? The higher the price, the greater the value of transparency.
4. Must I explain the decision? If so, make interpretability a condition of the choice from the start, not a later stage.

It is not a matter of one approach defeating another, but of fitting the tool to the problem. The wise engineer asks about the problem before asking about the newest networks; the tool is determined by the questions, not by fashion.