Did Your Staff Really Learn? How to Measure the Impact of AI Training

In 2025 the Saudi Ministry of Education announced that more than a million citizens had been trained in artificial intelligence and had received certificates under the Samai initiative, which brings together the Ministries of Education and of Human Resources and Social Development and the Saudi Data and Artificial Intelligence Authority (SDAIA). It is a large number, and it tells us how many people the programme reached. It does not tell us what changed in their work after they left the classroom. That is the question many organisations avoid: how do we know that training bore fruit?

Four levels, not one

In a series of articles published between 1959 and 1960, Donald Kirkpatrick set out the best-known framework for evaluating training. It distinguishes four levels: the trainee's reaction, then what was learned, then what changed in behaviour at work, then the effect on the organisation's results. Measurement easily stops at the first level because it is the cheapest: a short questionnaire at the end of the course. But satisfaction with the room and the instructor does not show that the employee has become better at the job.

Use is not impact

Stanford's 2025 AI Index reports that the share of organisations saying they use AI rose from 55% in 2023 to 78% in 2024. That is wider use, not evidence of the value it produced. The World Economic Forum's 2025 jobs report says that 63% of the employers it surveyed regard skills gaps as the biggest barrier to transforming their businesses through 2030, and that, on their expectations, 59 of every 100 workers will need training by that year. If spending on training is going to widen, it is wise to know what it yields.

What good measurement looks like

A workplace study gives a clear picture of good measurement, even though it is not about a training programme. Researchers followed 5,179 customer-support agents who were given a generative assistant in successive waves, and compared their performance with that of colleagues who had not yet received it. The number of issues resolved per hour rose by about 14% on average, and by about 34% among novices and less skilled workers, while the effect was small among highly experienced staff.

The example holds three lessons: a comparison with a group that has not yet received the tool, a measure drawn from the work itself rather than the classroom, and a breakdown by experience, since the average alone would have hidden the fact that the benefit was concentrated in one group. It is also a single setting, customer support at one company with one tool, so its figures should not be carried over to every organisation.

Five practical steps

• Measure performance before training; whoever does not know the starting point cannot know how far they have come.
• Tie the measurement to a real task: time to complete it, its quality and its error rate, not a test score at the end of the course.
• Measure again after weeks and months, because what is not practised is forgotten, and the tools themselves change.
• Pair the number with the story: what staff and beneficiaries say about how their work has changed explains the figure.
• Redesign the programme in the light of what the measurement shows; otherwise it becomes a report left in a drawer.

The limits of measurement

Not everything useful can be counted. Self-confidence, initiative and the skill of putting a good question to a tool may not show up in a spreadsheet, and it is a mistake to neglect what we do not measure. Measurement also has an ethical limit: it is better for an organisation to measure its programme than to monitor its people in ways that intrude on their privacy (see the article "Privacy in the Age of AI: How Your Secrets Are Inferred, and How to Protect Them"), and to tell staff what is measured and why.

Money spent on training people is a trust held by whoever spends it, and part of honouring that trust is to ask, after spending, what it yielded. If the answer is disappointing, knowing it is better than a comfortable belief with nothing behind it.