Explainable AI: When the Decision Falls Silent

You apply for financing and the refusal arrives within seconds. You ask why, and are told: 'the system decided.' Had a judge or a doctor said it, we would call it plain injustice. So why do we accept it from an algorithm?

The black box is no conspiracy

Deep models do not run on hand-written rules. They draw out patterns from millions of examples, so a decision comes to rest in the interplay of vast numbers of numerical parameters spread across stacked layers. Nobody made this opacity on purpose; it is a side effect of the way learning works. That led researchers to open an entire field, 'explainable AI'. The US Defense Advanced Research Projects Agency (DARPA) says its programme in this area sought models that are easier to explain while keeping a high level of performance. For the technical background on machine learning and deep learning, see our article 'Real AI: What It Is and What It Is Not'.

An explanation that persuades but is not true

The gravest point is that a handsome explanation may be false. In its 2021 document, the US National Institute of Standards and Technology (NIST) set out four principles of explanation: the system gives reasons, the reasons are understandable to those they are addressed to, they are accurate, and the system knows the limits of its knowledge. The document distinguishes decision accuracy from explanation accuracy: a system may be right in its decision while its explanation describes a route it never took.

Nor are these merely theoretical worries. A research team (Slack et al., 2019–2020) showed that LIME and SHAP, two widely used explanation tools that estimate each factor's influence by testing altered inputs, can be fooled. One builds a classifier that stays biased on real data and then arranges matters so that the two tools produce innocent-looking explanations; the researchers tried this on several datasets, among them the well-known COMPAS criminal-justice data. A biased system can thus wear an explanation that looks fair.

Along the same lines, Cynthia Rudin distinguishes two things: explaining an opaque model with an external tool, and building a model that is interpretable by design. She argues that settling for after-the-fact explanation in high-stakes decisions, such as healthcare and criminal justice, risks entrenching bad practices and doing serious harm, and she calls for models that are interpretable from the outset. This is one researcher's position, not a settled consensus, but it reminds us that 'having an explanation' is not the same as 'having understanding'.

Explained to whom?

Explanation has no single form. What an engineer needs to repair a model differs from what a doctor needs to trust a recommendation, and from what a person harmed by a decision needs in order to contest it. That is why NIST made understandability a principle in its own right, and noted that simplification suited to one audience may omit details an expert needs. The right question is: explained to whom, and for what purpose?

What principles and laws say

This requirement now appears in authoritative documents. The OECD's transparency and explainability principle calls for clear information about the factors and logic behind an output where feasible and useful, and for enabling those affected to challenge it. Article 86 of the EU Artificial Intelligence Act provides that a person affected by a decision based on a high-risk system of a kind listed in a particular annex may ask the deploying body for a clear explanation of the system's role in the decision. In the Kingdom, the AI Ethics Principles issued by the Saudi Data and Artificial Intelligence Authority (SDAIA) include 'Transparency and Explainability' and 'Accountability and Responsibility'.

What can be done in practice?

Behind these principles lie specific practices that any organisation can demand or adopt:

• Documentation: a card for each model stating its data sources, limits and weak spots, along the lines proposed by Mitchell and colleagues in 'Model Cards for Model Reporting' (2018).
• Choice at procurement: in weighty decisions, prefer models that are interpretable by design where they serve the purpose, rather than a cosmetic tool laid over an opaque one.
• A human in the loop: nobody's fate is left to an automated judgement without human review able to overrule it.
• Testing the explanation itself: ask not only 'does the system have an explanation?' but 'does it match what actually happened?'

Among the foundations of justice is that the person judged should know the grounds of the ruling and have a way to object. That is precisely what automated systems need when they touch people's rights: a decision that can be understood can be appealed. Nobody asks the machine to be human; only that it stand to be held accountable, as everyone does who rules on another's affairs.