Thank you for opening up the latest article in 'The Intelligent Commit' newsletter, my thoughts, research and analysis on Artificial Intelligence and how AI is reshaping the software development lifecycle. I hope you find it insightful, subscribe, and engage.

A few weeks ago, my paper Building the Foundations of Successful AI Adoption landed and a question was asked around the source of the headline-grabbing statement that 95% of organisations are getting zero return from their AI investments. On the same day, a journalist asked me a fantastic question about why AI programs are still being measured by productivity gains and cost savings, and whether organisations should shift to measuring AI's impact on long-term enterprise value. These two questions feel intrinsically linked.

tldr;

  • The headlines are directionally true. Many organisations are spending heavily on AI, and most are struggling to show anything meaningful for it.
  • While the numbers look alarming, researchers are using a range of different definitions of what "success" actually means.
  • When things go wrong, the root cause is almost never the technology itself; it is typically how people, teams, and organisations behave around it.
  • Researchers found a significant number of initiatives never had any realistic chance of reaching the real world because they only existed to reassure boards that 'something was happening'. They were designed to fail.
  • Simply dropping AI into an existing way of working invariably changes very little; the underlying business processes have to change too.
  • Judging AI investments too quickly, or on the wrong criteria, will almost always make them look worse than they are.
  • Chasing the wrong numbers can actively steer an organisation in the wrong direction, as at least one major brand discovered the hard way.
  • The question to ask is whether the way we judge AI's success was ever fit for purpose, rather than whether AI is living up to C-suite expectations.

Are AI projects really failing or do we need to revisit how we measure and define success?

While writing the paper, I collated and managed a library of over 100 primary sources, academic and industry research. I've spent the last few days reading back over select studies looking for patterns, approaches to measurement, what marks out the more successful companies, and whether there were any common failure modes and thematic reasons. It's an interesting data analysis challenge, but a bit dry. However, I will persevere and if you survive reading to the end, hopefully you will take away some insights and a counter-narrative to consider. First, I need to cover off 'success' and 'failure'.

In summary, those 'failure' headlines are real and consistent, although it was clear that the definition of 'success' differs across researchers.

  • MIT reported that despite an estimated $30-40 billion investment, 95% of organisations saw zero measurable financial return.
  • Gartner found only half of AI projects make it to production.
  • BCG found 3 in 4 organisations failed to deliver scaled value.
  • PwC's 2026 CEO Survey found over half of executives reported they neither increased revenue nor reduced costs from AI over the prior twelve months.

The spread from roughly 45% to 95% failure rate is not noise or contradiction; it is the direct consequence of different yardsticks being applied to the same question.

So while failure is often real, the failure rate depends entirely on what gets measured, and this is where the conversation gets interesting. What do these surveys actually class as success or failure? MIT NANDA requires demonstrable profit-and-loss impact within six months of pilot completion. That is a deliberately high bar. Gartner tracks whether a project reaches production at all, which is a much lower one. McKinsey looks at contribution to operating profit, while PwC measures revenue and cost impact against specific thresholds.

What is not in dispute is that adoption is soaring by every headline, metric, and report going. We are no longer facing organisational reluctance or individual skepticism, and as MIT reported, a lot of money is being spent.

So the question remains, why aren't 'returns' following?

When things go wrong, the root cause is almost never the technology itself - it is how people, teams, and organisations behave around it.

Positively, the research mostly agrees on why AI projects struggle. BCG summarise it best when they suggest 'success' is 10% algorithms, 20% technology & data, and 70% people, processes, and cultural transformation, a conclusion that is repeated independently in studies by McKinsey, Accenture, Deloitte, RAND, and many more.

The single most documented barrier to success across all research is "pilot purgatory"; that is, organisations who remain stuck in proof-of-concept loops, with little to show for in production and no clear path for how to get there.

Given many researchers use this as a primary marker of 'success' and 'failure', it is important to dig behind the headlines to investigate why pilots are getting stuck, not only that they are.

Alarmingly, the detail suggests that many pilots were never designed to reach production in the first place. IDC's research of nearly 3,000 decision-makers found that a significant proportion of pilots were driven by board-level urgency and funded to demonstrate activity rather than to solve a defined business problem.

When a proof of concept exists to show that 'something is happening' rather than to build something scalable, the subsequent failure to scale is not a surprise but an almost inevitable outcome.

Even where intent is genuine, pilots typically use clean, carefully prepared data, while the messy reality of how that organisation actually stores and manages its data is often overlooked. Does this resonate? Think back to how your AI solution survived its first contact with production reality.

MIT research describes AI solutions as static "science projects" rather than evolving systems. Solutions are being built in isolation from the organisation's existing systems, creating painful integration challenges when it comes time to scale, with API misalignment, missing security controls, and incompatible architectures.

The cost of production is also dramatically underestimated. The AI model itself is typically a small fraction of any solution (McKinsey puts it at roughly 15%), with the bulk of effort and cost going to integration, change management, and ongoing operations. Most organisations budget for the new technology and then discover the rest late. Unsurprisingly, if these costs aren't considered then the ROI and business case later collapses.

Finally, BCG found that 60% of companies fail to define any financial KPIs for AI value creation at all, and McKinsey's 2025 survey found that fewer than 20% track well-defined KPIs for Generative AI solutions. Without measures in place from the start, even if the pilot is successful, it may be hard to justify continued investment.

There is a compounding effect to consider too. Any type of failure, whether intentional or not, can drain workforce confidence. In turn, that makes the next one harder to fund and harder to staff, which makes it more likely to fail.

Simply dropping AI into an existing way of working changes very little - the underlying processes have to change too.

Beyond 'pilot purgatory', other common failure modes reinforce organisational patterns.

  • RAND identified misunderstanding the problem as the most common root cause, with teams building solutions without a defined business problem.
  • At the leadership level, BCG found that nearly all C-suite leaders at high-performing firms are deeply engaged with AI strategy, compared to just 8% at the so-called AI laggards.
  • And finally, a strong predictor of whether AI moves the earnings needle is not the model, the data, or the budget. It is whether anyone redesigned the business process. McKinsey found only 21% of organisations had attempted to do so.

Understanding why projects fail is only half the picture, though. Even where intent is genuine and execution is solid, there is a more fundamental question about whether the way we judge success was ever fit for purpose in the first place.

There is an argument that traditional ROI metrics are structurally unsuited to AI in the first place. 78% of executives already acknowledge traditional metrics do not fully capture AI's impact.

Regulars at Yorkshire DevOps or Software Engineering Leadership conferences will have heard me talk on the challenges, the pros, and the cons of measuring 'effort > output > outcomes > impact'. It's hard, I get it.

AI costs arrive immediately while the real benefits take time to compound, which means measuring success after a few months is almost always going to give you an unfair picture. If value shows up as people becoming better at their jobs (for example, analysts producing better-informed or deeper research, or managers making faster decisions), that's different to having a line item on a cost spreadsheet.

And it is hard to point to AI as the sole cause of any improvement, because it rarely lands in isolation; it is introduced alongside improvements to data quality, team structures, workforce training, and process redesign. Which, of course, it has to.

Fortunately, KPMG reports that 78% of executives already acknowledge traditional metrics do not fully capture AI's impact, which suggests they can now sense this gap, even if they have not yet closed it.

Positively, some studies are getting closer to a better measurement framework. ISACA breaks ROI into three distinct layers; the direct financial return, the strategic positioning benefit (are you more competitive and adaptable?), and the people benefit (are your teams more skilled and innovative?), each tracked over a different timeframe.

And MIT Sloan and BCG suggest that the most forward-thinking organisations are going further, using AI to make their performance metrics smarter, turning measurement into a competitive advantage rather than just a scorecard.

Having the right measurement of success really matters.

Klarna offers a cautionary tale of what happens when measurement stays too narrow. Having initially celebrated AI doing the work of 700 customer service agents, projecting $40 million in profit improvement, they were forced to reverse course and rehire human representatives after discovering that its purely cost-focused AI measurement had obscured deteriorating customer experience.

In summary, the 'failure' headlines are not wrong, but the data is incomplete. The real divide forming across industries is not between organisations that have adopted AI and those that have not. The organisations capturing disproportionate value right now are those that take the questions of governance, measurement, and organisational change seriously.

And if 78% of executives already sense that their metrics are missing the picture, should the question be whether AI is failing, or whether we need to change the way we keep score?

Thank you for reading my latest article. Please share your experiences and engage with the discussion. I am not always right, but I try to be insightful and thought-provoking.

  • Are you tired of reading about AI project failures and scares? Ever considered why there may be so many negative reports (and yes, some are because the authors have a magic bean they want to sell you).
  • Is your feed full of AI success stories that you're struggling to replicate? Maybe you just don't know where to start or how to scale.
  • Do you recognise 'pilot purgatory', are you also stuck in non-production with no clear pathway to successful live workloads? (Amazon suggests 13 pillars that must be considered as part of operationalising any AI workload).
  • Or maybe, everything is great and you don't know what all the fuss is about.

Please reach out if you'd like to continue the discussion.