Learning from Few Examples, or None: Has the Machine Outgrown Data?

Tell someone who has never seen a zebra that it is "a striped horse", and they will recognise one when they meet it. This is what machines have been trying to imitate for years: to recognise what they were never explicitly trained on, relying on what they already know. And the question that keeps coming up is whether artificial intelligence no longer needs data.

Three terms that get mixed up

Few-shot learning is when a model is shown a handful of examples; one-shot learning is a single example; and zero-shot learning is when it sees no example of the task at all and is given only a description of it. The common confusion is that "zero examples" means "zero data". It does not, as will become clear.

An idea older than the hype

In a 2006 paper in the IEEE journal on pattern analysis, Li Fei-Fei and two colleagues showed that a new visual category can be learned from one or a few images if the system reuses what it learned from earlier categories, even dissimilar ones; they tested this on 101 categories. In 2009 Christoph Lampert and colleagues classified categories with no training images at all: each category was described by human-written attributes, such as shape and colour, that the system had learned beforehand from other images. For this they built the Animals with Attributes dataset: more than 30,000 images of 50 kinds of animal and 85 attributes. So the root is old; what has changed is the scale of prior knowledge.

What did large models change?

In the 2020 paper "Language Models are Few-Shot Learners", OpenAI researchers showed that GPT-3 performs new tasks from instructions and examples written in the prompt itself, with no update to its weights and no retraining. In 2021 the company presented CLIP, trained on 400 million image-text pairs, which matched the accuracy of the well-known ResNet-50 on ImageNet without using any of its 1.28 million training examples.

The lesson lies in those very figures: the big data did not vanish; it moved from the front of every task into a general foundation that is built once and used many times. Building that foundation is probably costly, so "few" should not be read as "cheaper" for the system as a whole.

A fragility the headline hides

It is said that choosing the examples is a secondary detail; the research says otherwise. Tony Zhao and colleagues found in 2021 that GPT-3's few-shot performance is "unstable": changing the wording of the prompt, the choice of examples or their order can move accuracy from near chance to near the best results of other methods. Yao Lu and colleagues found that example order alone is enough to produce this spread, and that a good order for one model does not necessarily carry over to another. Reducing the examples raises the weight of each: a single biased or ambiguous example can mislead the whole model.

What is useful here?

• Evaluate on your own cases before relying on it: laboratory results are not enough.
• Choose examples carefully and try different wordings and orders; this is a craft that can be learned.
• Small, high-quality collections in sensitive fields, especially for languages and topics with few resources. This is likely to help Arabic and its contexts, but it needs measuring, not assuming; see the article "Arabic Artificial Intelligence: Do We Make or Consume? Sovereignty and Arabic Language Models".
• Keep a human as judge where mistakes are costly, as in the article "Responsible Artificial Intelligence: Four Principles for Governance and Trust".

Our relationship with the model has shifted a little as a result: from programming it with thousands of silent examples to conversing with it much as one teaches a bright pupil. Choosing the example well remains worth more than having many examples. Do we know how to give the machine the one word that replaces a thousand examples?