Multimodal AI: It Sees and Hears, but Does It Understand?

Two pictures and two captions. The words in the captions are the same; only their order differs. The task is to match each caption to its picture. A person does it in a moment, but the models that researchers examined in the Winoground test in 2022 did not do much better than chance. So how do these models impress us when they describe a photograph or read a chart, and then stumble on a test that looks so simple?

From separate systems to a shared space

For decades AI systems were built for a single job: one classified images, another decoded speech, a third processed text. What is new is a single system that learns a shared representation, so that the word "cat" and a picture of a cat become numbers lying close together in the same mathematical space, and the distance between them can be measured.

OpenAI's CLIP model, published in 2021, showed this. It was trained on 400 million image-and-caption pairs gathered from the internet, after which a user could point to visual concepts with words and the model would classify images it had not been explicitly trained on. Combined with large language models (see the article "How Do Large Language Models (LLMs) Like ChatGPT Work?"), the idea widened: in 2023 the company announced that GPT-4 accepts images and text and produces text.

What helps today

The clearest benefit is making what is seen available to those who cannot see. The Be My Eyes app used GPT-4 to build what it called a "Virtual Volunteer": the user sends a photo from their phone, and the model describes it and answers follow-up questions. A user might photograph the contents of a fridge and be offered dishes to cook, or ask whether an object on the floor is something to trip over. That is the developer's description from the early days of the trial, not an independent assessment of its accuracy.

Fluency is not understanding

The GPT-4 report itself concedes that the model is less capable than humans in many real-world situations, even as it matches human performance on professional and academic exams. Winoground shows one side of that gap: both captions contain the same words, so a system that merely recognises objects will not succeed; it has to grasp the relations between them and their order. The difference between "the dog bites the man" and "the man bites the dog" is not settled by knowing which objects appear, but by understanding who does what to whom. The result concerns the models of that date and should not be generalised to everything since, but it reminds us that a machine's description of a scene can be persuasive and incomplete at once. The article "Three Myths About Artificial Intelligence: Fluency, General Intelligence and Neutrality" explains our tendency to credit with understanding anything that imitates our speech.

This is no fault to charge against the tool's makers; the fault is to load it with what it has not shown it can do. Sensitivity grows when face, voice and text meet in one system, because more can be inferred about their owner. These are personal data that deserve the same discretion and trustworthiness as any other secret; the article "Privacy in the Age of AI" helps here.

Using it wisely

1. Test the tool on cases whose answers you already know before relying on it.
2. Do not let a machine's description of a medical, legal or security image replace a specialist.
3. Do not upload other people's photos or recordings without their permission.
4. Ask the developer for a statement of the tool's limits, and beware of a demonstration designed to dazzle.

We use tools and take the means, so we check and verify, and we put our trust in God for what we do not control.