AI as a Trusted Servant, on One Condition: Where It Helps and Where It Misleads

Researchers in Germany and the Netherlands tested 27 radiologists on 50 mammograms, showing the doctors suggestions from an AI system, some correct and some wrong. What happened when the suggestion was wrong? That question sums up what "trusted servant" means.

Where benefit is proved

No reasonable person denies that this technology bears fruit. In October 2024 the Nobel Prize in Chemistry was shared between David Baker, for computational protein design, and Demis Hassabis and John Jumper, for protein structure prediction. The latter two used the AlphaFold2 model on a problem about fifty years old. Al Jazeera reported that it calculated the structures of all human proteins and predicted those of nearly all of the roughly 200 million proteins researchers have identified, and that more than two million people in 190 countries have used it.

In medicine, our article "What Has AI Actually Proved for Our Quality of Life?" covered the Nature study on breast screening: a system that reduced false alarms and missed cases on past records, whose authors concluded that it supports later clinical trials. These are real fruits in defined tasks.

When reliance becomes a danger

But benefit leads to an old psychological habit: believing the machine because it has so often been right. Researchers call it "automation bias". A team led by Thomas Dratsch of the University Hospital Cologne measured it and published the result in Radiology in 2023: 27 radiologists read fifty mammograms with classification suggestions from an intelligent system. When the suggestion was wrong, the accuracy of the less experienced doctors fell to below 20%, and that of the experienced ones (over 15 years' experience on average) fell from 82% to 45.5%.

An important qualification: this was a laboratory test with a limited number of doctors and images, not a record of clinical practice. Yet the lesson goes beyond radiology: experience alone does not protect against being swept along by an answer that looks confident. And it is the more dangerous when the machine is right most of the time, because the eye stops checking.

What makes a servant trustworthy?

Trustworthiness here is not a virtue we ascribe to the machine; it consists of arrangements an institution puts in place. In light of this evidence, the following are suggested (our inference, not the text of the study):

• An independent human opinion first. The doctor or auditor reads the case and records an assessment before seeing the machine's suggestion.
• A statement of the system's limits. Knowing where it errs and how often, rather than presenting its answer as a verdict.
• Periodic auditing of the system's performance in its real setting, not in the setting where it was tested.
• Clear accountability: who signs the decision? The answer is always a person or an institution, never an algorithm.

These meanings connect with the AI ethics principles issued by the Saudi Data and Artificial Intelligence Authority (SDAIA) in their second version in 2023. There are seven: fairness, privacy and security, humanity, social and environmental benefits, reliability and safety, transparency and explainability, and accountability and responsibility. Systems are classified by level of risk: low-risk systems are not obliged to comply though it is recommended, limited- and high-risk ones must comply with additional controls, and unacceptable-risk systems are prohibited. These general principles are treated in our article "Responsible Artificial Intelligence: Four Principles for Governance and Trust".

Informed trust

None of this means hesitating to use the machine; whoever abandons beneficial means has fallen short. We use it as one uses a trusted servant: we thank it for its skill, ask about its limits, and know that responsibility rests with us. The source of trustworthiness in the machine is its makers and its custodians; in the human being it is God and then conscience, and we hope for a good outcome for those who combine taking the means with reliance on God.