The Lasagne of AI: Why the Dish Spoils When the Ingredients Do

Researchers examined ten datasets that the field relies on as reference yardsticks and found labelling errors in their test sets averaging at least 3.3%, and at least 6% in the ImageNet validation set (Northcutt and colleagues, 2021). In other words, the "scales" on which models are weighed were not perfectly true. That is where the kitchen analogy begins.

Ingredients before the recipe

Skilful seasoning cannot save spoiled meat. The same is true of an algorithm: it learns patterns from the data it is given, and if that data is incomplete, biased or mislabelled, it will faithfully learn the error. This is what we mean when we say that the first lesson in this field is the quality of the inputs.

A serious team asks three questions about its ingredients. Is the data sufficient to cover the cases that matter? Is it diverse, so that the people who will use the model are not absent from it? And is it correctly labelled? Our article on algorithmic bias showed how a flaw creeps in from a group missing from the data, and our article on large language models showed how such models are built on a vast volume of text that is hard to inspect in full.

The recipe and its steps

The algorithm is the recipe: a defined way of turning ingredients into a result. The engineer has room for judgement, much as a chef does: trying different kinds of model and tuning what are called hyperparameters (such as the learning rate and the number of layers), as a cook adjusts the oven temperature and baking time.

But order matters in a model as it does in a dish. Cleaning the data before training, splitting it into sets before any experiment, and measuring performance afterwards are steps whose sequence cannot be scrambled; if it is, the result is spoiled even when the ingredients are good.

Three dishes, not one

Engineers usually divide the data into three sets. Google's Machine Learning Crash Course describes them as follows:

• Training set: the examples the model learns from.
• Validation set: used to test the model during training and to guide tuning.
• Test set: a final check carried out when the validation results look promising.

The logic of the split is that testing on new examples is stronger evidence of a model's fitness than testing on examples it has already seen. It is like judging a chef by the palate of a guest who has never eaten his food, not by his own.

The same document warns of two things. First, reusing the test set again and again during tuning pushes the model to fit it ever more closely. Second, duplicated examples across training and test sets inflate the score falsely; one of its examples is spam-email data whose duplicates were not removed before splitting, so that part of the test set found its way into training unnoticed.

Tasting, then back into the oven

After testing come rounds of improvement: the team reviews where the errors fall, adds missing data, and changes the settings or the model. This is no weakness; it is the heart of the method. The wise course, though, is not to exhaust the test set in these rounds, but to keep a portion that neither the model nor the team sees until the very end.

The result has a side that is often forgotten: what we call "accuracy" is simply the count of answers that matched the accepted labels. If the labels themselves contain errors, as Northcutt and colleagues showed, the meaning of the number changes. That is why the researchers gave their paper a title describing such errors as destabilising machine-learning benchmarks: the narrower the gap between two models, the less confidence we can have that it reflects genuine superiority.

What can a reader do?

When a system is announced with "high accuracy", ask three simple questions. On what data was the accuracy measured? Was it new data the system had not trained on? And whom does the measurement represent, and do they resemble those who will actually use the system? Whoever asks about the ingredients is not deceived by the picture of the dish.

There is also excellence in taking the proper means. Whoever takes care over data and evaluation before boasting of the result is closer to scientific honesty, and success in the end is from God.