AI Return on Investment: How to Measure It Without Fooling Ourselves

A chief executive tells the board that the AI platform has delivered a "dazzling return". Do you believe the figure? Better to ask one question: compared with what? A figure whose birth we do not know silences the questions that should have been asked before it was written.

Why does the effect arrive late?

In its classic form, return is net benefit divided by cost. The formula was born in a world where a machine is bought at a known price and produces countable units. Artificial intelligence is closer to a general-purpose technology like electricity. The economist Paul David showed in a 1990 paper that factories which replaced the central steam engine with a single electric generator saw small gains, and productivity did not rise sharply until the factory was redesigned with a small motor at each machine.

Erik Brynjolfsson, Daniel Rock and Chad Syverson then proposed the "productivity J-curve": general-purpose technologies require complementary intangible investments in processes and skills, which are hard to measure, so productivity is estimated below its true level in the early years and above it later, when those investments bear fruit. Hence whoever measures three months after adoption is mostly measuring the cost of learning, not the value of output.

The average hides variation

Brynjolfsson, Danielle Li and Lindsey Raymond studied the introduction of a generative assistant in customer support, using data on 5,179 agents. Issues resolved per hour rose 14% on average, and 34% for novices and the low-skilled, while the effect for experts was small. Whoever counts the average alone hides that the tool helps one group and barely helps another.

In a randomised trial by the organisation METR in 2025, 16 expert developers of open-source software each worked on about 15 tasks from their own repositories, with tasks randomly assigned between allowing and forbidding AI. Completion time rose 19% when it was allowed, the opposite of what they expected before the trial (a 24% reduction), and afterwards they estimated they had saved 20%. The sample is small and early-2025 tools have since changed, so we read it as a warning against relying on subjective impression, not a general verdict on the tool.

In the Harvard experiment with the Boston Consulting Group (758 workers), users of the tool were 19 percentage points less likely to produce a correct solution on the task outside the model's capability, and were faster and better inside its limits. So the question about the scope of the task is part of the calculation.

An hour saved is not a riyal

Many suppose that an hour saved is profit. Yet a study by Humlum and Vestergaard of 11 exposed occupations (25,000 workers in about 7,000 workplaces in Denmark) found average time savings of about 3%, with no significant effect on wages or recorded hours in any occupation. That is, saved time does not become money unless it turns into extra output that is sold or an actual cost that is removed.

The full cost

The price of the subscription is usually counted and what lies beneath it forgotten: cleaning and governing data, computing and its variable bills, training staff, legal and security risks, and maintaining what was built in haste. We found no documented public figure for the share of these costs, so we cite no percentage and keep to the principle: a calculation that omits the full life-cycle cost is misleading by nature.

Five rules for honest measurement

1. Compare: a team using the tool and a similar team not using it, in the same conditions. A rise in sales after rollout may be due to the season or a campaign.
2. Set the indicators before you begin: whoever chooses, after the results appear, the measures that flatter them is selecting evidence, not measuring impact.
3. Combine leading with lagging indicators: such as output quality, completion time and customer satisfaction, before the budget shows anything.
4. Count the full cost over the life cycle, visible and hidden.
5. Tie each project to a clear goal before launching it, stop early what does not meet its goal, and do not be ashamed to admit it.

In our institutions the "fear of being left behind" may push us to spend before measuring. It is better to learn how to measure honestly, for that is an asset more useful than any model we buy. This presents frameworks for understanding, not a recommendation to spend or to refrain; whoever needs a financial decision should consult a licensed specialist.