Conscious AI Adoption: Five Principles Before the Rush

Between 16 and 31 July 2024, McKinsey asked 1,491 participants in 101 countries what separates organisations that see a financial effect from generative AI from those that do not. It tested 25 factors. The one most strongly associated with impact on profit was the redesign of workflows, not the number of tools or the size of the spend. Yet only 21% of the organisations using generative AI said they had fundamentally redesigned any of their workflows. The association is statistical and does not prove causation, but it hints that the most useful question is not "how many tools have we bought?"

When a machine is added to a structure not built for it

The usual habit is to drop a tool into existing work and count the gain at once. That sum has a hidden line. In a 2015 paper, D. Sculley and colleagues described what they called "hidden technical debt" in machine-learning systems: they are quick to build, but they carry continuing maintenance costs from entangled components, hidden feedback loops and dependence on data. The first gain may be a loan rather than a profit.

The irony of automation

In 1983 the psychologist Lisanne Bainbridge wrote that automating most of a job creates new and severe problems: people are left with what the machine cannot do, they get fewer chances to practise their skills, and watching a machine becomes tiring work. She concluded that operators need more training, not less, to be ready for the rare but crucial interventions.

The literature on automation bias points the same way. In a 2010 review, Parasuraman and Manzey found that over-reliance on automated aids persisted despite training and instructions. This is why Article 14 of the EU Artificial Intelligence Act requires high-risk systems to be designed so that the human overseer can recognise the tendency to over-rely on outputs, and can disregard, reverse or stop the system.

Five principles

npsai, the initiative that supports the non-profit sector, presents an adoption methodology built on the following pillars, which we read here in the light of the evidence above:

1. Purpose before tool. Begin with the human problem, then ask whether this technology is the best answer.
2. Explanation where a decision touches a person. The OECD principles call for transparency, explainability and accountability; a fateful decision that its owners cannot explain should not be accepted.
3. Monitored, gradual rollout. Stages that can be measured and reversed, not a single move with no way back.
4. Preserving human competence. Train people alongside the adoption of the tool, so that judgement does not wither under dependence.
5. Human value as the final test. Every tool is put to one question: how much sovereignty over the decision is left to the person?

Not a call for slowness

Refusing a tool whose benefit is proven in a given field is not wisdom; it forgoes real benefit. A sound approach distinguishes: where the machine is shown to excel, it leads; where judgement, empathy and responsibility are needed, the person keeps the decision; where the two complement each other, the collaboration is designed on purpose. And this distinction should rest on measurement rather than intuition.

How do we know the awareness is real?

The authors of the methodology propose measuring the success of adoption on three dimensions rather than one: how much efficiency it raised, how much value it preserved, and how much sovereignty over their decisions it left to people. This is a proposed view, not a proven result, and we found no independent evaluation of this particular methodology, so judging it requires trial and measurement in each organisation.

Nor is every discourse about "conscious adoption" sincere. Awareness shows in the decision taken when a quick profit conflicts with a lasting value, not in principles hung on walls. And a decision that affects people is a trust held by its owner, however clever the tool.