Arabic Poetry and AI: A Reader Fooled by Appearances, a Machine Stumbling at Understanding

A language model was asked to write a "short poem in the style of" a given poet, from ten poets including Shakespeare, Emily Dickinson and T. S. Eliot. Its first five outputs for each poet, unselected and unedited, were shown alongside real poems by them to 1,634 participants, who were asked which had been written by a human. Their accuracy was about 46.6%, less than a random guesser would achieve (Porter and Machery, 2024).

Why Did They Like It?

In the same study, other participants (696 of them) rated the machine poems higher in overall quality and on 13 of 14 qualitative measures, with rhythm showing the largest gap. On "originality" no significant difference appeared. The researchers explain this by suggesting that the machine poem is simpler and easier to interpret, so the reader likes it and then takes that liking as evidence that it is human; the reader may also read the complexity of a human poem as noise. This is a proposed explanation, not an established fact. The study also noted that telling participants a poem was machine-made lowered its rating, whether or not it was.

A caution is due: the poems were English, and the model was ChatGPT-3.5, older than today's models. The study shows how easily a quick judgement is fooled, not that the machine equals the poet.

And Arabic?

Al-Khalil ibn Ahmad al-Farahidi (second century AH) devised a system of metres in which he described fifteen, grouped in five circles, and a sixteenth, al-mutadarik, is attributed to al-Akhfash as an addition. This discipline tempts one to assume that the computer will quickly master Arabic poetry. But the "Fann or Flop" benchmark (EMNLP 2025) says otherwise about understanding. It is a test set of poems from 12 eras, from classical metrical verse to contemporary free verse, measuring grasp of meaning, interpretation of metaphor, awareness of prosody and cultural context. Its result is that most models stumble at understanding poetry despite their strength on usual Arabic tests.

This does not mean the machine will never improve; the result is a snapshot of where we stand today, and models change quickly. But it reminds us that hitting the form is one thing and grasping what lies behind it is another.

A Measure Worth Revisiting

The two experiments together show that "does this sound like poetry?" is a weak measure. A hasty reader settles for sound and fluency, and may prefer what comes easily over what taxes and enriches. If the researchers' explanation holds, then as machine texts multiply, public taste may drift towards the simple and familiar, and dense poetry that needs reflection may be neglected. That is a conclusion we warn of, not one that has been measured.

The better question is: what does the text say about what we do not know? In what experience and language does it say it? Does it hold up when we reread it after a while? These are questions a human can answer, and ones on which models stumble today.

What Do We Do?

Arabic poetry is a record of language, memory and taste, and the machine can serve it if put in its place:

• Documenting texts: verified digital corpora in which versions and metres are fixed, serving as a reference for learning and analysis, not a field for imitation.
• Teaching taste: young people learning prosody, imagery and the poetic lexicon, so that they recognise what a trained ear distinguishes and a hurried ear misses.
• Disclosure: stating plainly when a machine was used in a text; honesty in attribution is part of trustworthiness, and a poem must not be ascribed to anyone but its author.
• Disciplined use: employing the tool for indexing, collation and suggestion, with the final judgement remaining with the literary scholar and the editor.

The danger is least in the machine writing a metrically correct line. It is greatest in our growing used to a quick reading that never asks: who said it? What was he saying? What did he mean? Whoever learns these questions will be fooled by neither a machine poem nor a spurious one.