Why Your AI Adoption Numbers Are Lying to You

It's an oldy but a goody, what doesn't get measured, doesn't get done, but I think that only tells us half the picture because if you are measuring the wrong things you still aren't getting the whole picture.

You've seen the quarterly slide. Licences deployed: 4,000. Training completed: 87%. The room nods. The programme looks healthy. But three months later, the same teams are doing the same work the same way, & someone asks whether the renewal is worth it.

This is the shelfware cycle, & it kills more AI programmes than technical failure ever does. A tool gets bought, rolled out with enthusiasm, measured by procurement metrics, & then left to gather dust because nobody tracked whether anyone's actual work changed. The next budget ask lands on a table still scattered with the debris of the last one.

I've watched this pattern repeat across a dozen organisations in the last two years. AI adoption is new, they're at the beginning of this cycle, so lots of things like renewals haven't come up yet. But that's going to be the thing that people are going to look out for in the not too distant future. A lot of organisations have overestimated or underestimated the capability of AI & therefore implementation is very scattered, best practice isn't necessarily all clear, & people are still waiting to see the returns. Most AI adoption strategies measure the wrong thing entirely. They count access instead of behaviour. They celebrate completion rates on e-learning modules while the people who completed them go straight back to their old workflows.

Moveworks made a point recently that stuck with me: when AI acts as what they call an "agentic front door" to the enterprise, resolving issues & surfacing information before employees even raise a ticket, adoption stops being a change management problem. It becomes a natural outcome. The tool earns its place by being useful. I don't fully agree that it stops being a change management problem, because every shift in business is a change management problem & those frameworks are useful lenses for helping an organisation adapt. But the framing contains a cost that's easy to miss.

An AI system that acts on behalf of employees before they ask only works if the data governance, the permissions model, & the trust architecture underneath it are solid. Roll it out before those foundations exist & you've created something that's technically adopted but operationally dangerous, with no clear human accountability built in. The adoption number goes up. The risk register should too.

AI is a mirror of the organisation that it's in. It only depends on the organisation in terms of the data it collects, the processes that it has in place, the way that it operates. That is going to reflect whether you get good results or bad results from AI, which is why culture change, change management, adoption, all start with actually fixing the ground floor rather than trying to start up top by implementing systems, because these systems are just going to magnify what already exists within the organisation.

So what does honest measurement actually look like?

It starts with behaviour change. If you deployed a generative AI assistant for your procurement team, the question is whether cycle times shortened, whether the quality of first drafts improved, whether they stopped doing the manual comparison work the tool was supposed to handle. Those are harder to measure than licence activations. They're also the only metrics that will survive a CFO's scrutiny when renewal comes around.

The second piece is instrumenting from day one. If you can't articulate what behaviour you expect to change before you deploy, what you've got is a procurement decision with a training plan bolted on. Most organisations look at things as systems, & they implement the system & they expect that the system is going to change the behaviour, when quite often the behaviour needs to change in order to be able to operate the system. That's something that a lot of organisations miss the picture on.

I talk to Heads of L&D who feel this tension acutely. They know completion rates are a weak signal. They also know that their stakeholders want a number, preferably a big one, preferably soon. The pressure to report something encouraging can crowd out the slower, harder work of tracking whether the organisation is actually operating differently.

The way through is to pick metrics that would embarrass you if they went the wrong way. If your AI writing assistant has 3,000 active users but average document quality hasn't shifted, that's a finding. If your customer service copilot is being used daily but resolution times are flat, that's a finding too. AI has some of its most effective usage when it runs silently in the background. That doesn't run on adoption, that doesn't run on my usage of the tool, that runs on the information that I receive. If my insights are better, if my performance is better, if my understanding of a subject & my subject matter expertise is better, then that is an insight that has been gained through AI adoption. It's a much harder metric to measure. Honest metrics create accountability in both directions: they tell you what's working & they tell you what to stop funding.

My prediction: within 18 months, "behaviour change evidence" will be a standard line item in enterprise AI renewal business cases, right alongside cost savings & productivity gains. The signal to watch is whether procurement teams start asking vendors for embedded analytics that track workflow change rather than usage counts. If that shows up in RFP templates by mid-2028, the shelfware cycle starts breaking. If it doesn't, we'll keep buying tools & celebrating logins while the real work stays untouched. I'd change my mind if vendors find a way to make usage depth visible enough that raw access metrics become genuinely meaningful, but I haven't seen that yet.

The organisations that get this right will be the ones that treated measurement as a design decision from the start, & began building real AI literacy into how their people actually work.

Frequently Asked Questions

How do I know if my organisation's AI adoption is actually working?

Look at whether workflows have changed, not whether people logged in. Track cycle times, output quality, & task elimination rates for the specific work the AI was meant to improve. If those metrics haven't moved after 90 days of active deployment, your adoption is superficial regardless of how many licences are in use.

What are the most common mistakes in measuring AI adoption?

The most common mistake is treating licence deployment & training completion as proof of adoption. These are input metrics that tell you about access, not behaviour. Other frequent errors include waiting months before setting up measurement, relying on self-reported survey data, & failing to define expected behaviour changes before rollout.

What does "shelfware" mean in the context of enterprise AI?

Shelfware refers to software that has been purchased & technically deployed but is not being meaningfully used. In AI programmes, this typically happens when tools are rolled out without clear use cases tied to daily work, so employees revert to old habits & the investment sits idle until someone questions the renewal.

How should I justify AI spending to my CFO or board?

Lead with behaviour change data rather than adoption counts. Show whether the AI tool has shortened specific processes, improved output quality, or eliminated manual tasks. CFOs are more persuaded by evidence that work has genuinely changed than by a slide showing high login rates, especially when renewal costs are on the table.

What is an "agentic front door" approach to AI adoption?

An agentic front door is an AI system that surfaces information, resolves issues, & takes action on behalf of employees rather than waiting to be asked. The idea, discussed by Moveworks, is that when AI integrates into existing workflows & solves problems automatically, adoption becomes a natural outcome rather than a change management challenge requiring sustained effort.