Does the Machine Understand What It Says? AI in the Mirror of Philosophy

Ask a language model about the meaning of loss, and it answers with a sentence that moves the heart. Did it understand loss? Or did it know how people write about it? The question is not peculiar to the age of language models; the debate began before they existed.

From “Do they think?” to “Do they convince?”

In the journal Mind in 1950, Turing argued that the question “Can machines think?” is too ambiguous to settle by defining the words. He proposed an “imitation game” instead: an interrogator questions two hidden parties, and we ask whether a machine can fool him as often as a person can. Intelligence thus became observable performance, not a hidden essence to be searched for.

The Chinese Room

In 1980, in the journal Behavioral and Brain Sciences, Searle replied with a thought experiment. A man who knows no Chinese is locked in a room; he receives symbols and answers with symbols according to a rulebook. His replies are flawless, and an outsider takes him for a Chinese speaker, yet he understands nothing. The conclusion: syntax (arranging symbols) is not enough to produce semantics (meaning). A famous objection, the “systems reply”, holds that the man does not understand but the whole system, with its books and papers, does. Searle's answer is that if he memorised everything he still would not understand. The debate has not ended in consensus.

In computational linguistics, Emily Bender and Alexander Koller argued in 2020 that a system trained on linguistic form alone cannot, in principle, learn meaning. The argument is debated, but it warns against confusing fluency with understanding. (For how these models work, see the article “Large Language Models”.)

We are inclined to believe

Joseph Weizenbaum built ELIZA in 1966 to imitate a simple therapeutic conversation. Accounts say its users confided their most intimate secrets and attributed understanding and sympathy to it, even after learning that it was a crude programme. This later became known as the “ELIZA effect”. If a programme with simple rules could deceive people, what of a model that writes with high fluency? The more eloquent the speech, the stronger the temptation to think a heart stands behind it.

The hard problem

Philosophy distinguishes explaining what the brain does, the “easy problems” for David Chalmers, from explaining why there is “something it is like” to be in pain or joy, the “hard problem” (1995). In his famous essay “What Is It Like to Be a Bat?” (1974), Thomas Nagel held that knowing the external facts about a creature does not give us its experience from within.

Functionalism, on the other hand, holds that a mental state is defined by its role within a system, not by its material. If a system performed the role of a mind, there would be no principled bar to calling it thinking. Its opponents object that systems might perform the function without experience. So asserting that a machine cannot be conscious is a philosophical position, not an established scientific fact, just as asserting that it is conscious is a claim without evidence. What we do know is that current models predict the next word probabilistically from patterns in their data; that describes what they do, not a verdict on everything that might one day be built.

What our own heritage offers

Al-Kindi, al-Razi, al-Farabi, Avicenna and Averroes discussed the intellect and the soul and its faculties, distinguishing theoretical from practical intellect, and Avicenna argued with his “floating man” that the soul can know itself without the senses. This heritage was not written for machines, but it helps us pose the question in the language of our own culture, not only in imported terms.

Three practical counsels

First, call things by their names: “generates” and “predicts” are more precise than “understands” and “feels”. Second, do not make the machine an authority where only a person will do, in consolation, religious rulings and moral decisions; its eloquence is not the wisdom of one who has lived. Third, teach the young the difference between fluency and wisdom: the first can be imitated, the second is acquired through experience. These are questions that call us to humility: neither deifying the machine nor denying what is unsettled.