In today’s world, digital transformation is an inevitability for any business, but its track record means it probably inspires more dread than excitement. It should be promising sweeping innovation, new efficiencies, and a competitive edge. And though such grand aspirations should never be easy, it has become notorious for projects that balloon in scope, exceed budgets, and miss deadlines.
The statistics are sobering. Research suggests that 18% of IT projects experience cost overruns of 50% or more, with an average overrun of 447% for those projects. This can’t just be bad luck—it’s a structural issue with how organisations plan and execute transformation initiatives.
The problem lies in how we think about risk, uncertainty, and decision-making. But there’s a way forward, and it starts with understanding exactly what’s going wrong.
The Fat Tails of Digital Transformation
Most of us are accustomed to thinking in terms of normal distributions, where outcomes cluster around an average, and extreme values are rare. This works well for many everyday things, like human height or the time it takes to drive to the shops. But digital transformation projects don’t follow normal distributions—they follow fat-tailed distributions.
In a fat-tailed world, extreme outcomes—like a project running 447% over budget—aren’t rare exceptions. They’re surprisingly common and disproportionately impactful. A small percentage of projects account for the majority of total overruns, delays, and failures.
Here’s why this is hard for people to grasp: in a normal distribution, we expect things to balance out. If one project has a massive cost overrun, we assume the next one will revert to the average. In a fat-tailed world, that’s not how it works. The next project is just as likely to blow past its budget as the last one.
Imagine rolling a dice where five sides say “5% over budget” and one side says “400% over budget.” In a normal distribution, that 400% side would be so rare it hardly matters. But in a fat-tailed distribution, it shows up far more often than you’d expect, and when it does, it skews the entire outcome.
This is why traditional project planning often fails. It assumes risk is predictable and distributed evenly across projects. In reality, digital transformation is inherently risky, and extreme outcomes are baked into the system.
Optimism Bias
If fat tails are the structural problem, optimism bias is the human one. Optimism bias is the tendency to believe we’ll do better than average—that we’ll succeed where others have failed. It’s why most people think they’re above-average drivers, even though statistically, they can’t all be right.
In digital transformation, optimism bias shows up in subtle but dangerous ways. Organisations systematically underestimate complexity, downplay risks, and assume everything will go smoothly. Stakeholders overpromise outcomes to secure buy-in, setting up unrealistic expectations. Even if they’ve been burnt in the past, they think this time will be different.
Here’s an analogy: imagine you’re on a plane. You want the pilot to be a realist, carefully calculating risks, preparing for turbulence, and planning alternative routes. But you also want the flight attendant to be an optimist, keeping passengers calm and reassured.
In too many digital transformation projects, the roles are reversed. Planners act like flight attendants, presenting overly rosy scenarios, while execution teams are left to scramble like pilots reacting to unanticipated turbulence.
The Risks of Breaking New Ground
Digital transformation often means venturing into uncharted territory. Organisations want to be pioneers, leading their industries with cutting-edge technologies and innovative business models. But breaking new ground comes with a unique set of challenges that are easy to underestimate.
Integrating advanced technologies such as AI, cloud platforms, or IoT into existing systems isn’t just a technical challenge—it’s a web of dependencies, legacy systems, and stakeholder coordination. Each of these elements introduces uncertainty, and the interplay between them can make even small missteps cascade into significant delays or failures.
Adding to this complexity is the intense pressure to deliver results. Digital transformation projects are often branded as “strategic,” which ensures they receive attention from the highest levels of the organisation. While this is a necessity to deliver transformative outcomes, it creates immense pressure to prove ROI quickly. This urgency often leads to rushed decisions, overly aggressive timelines, and inadequate risk assessments. The focus shifts from doing it right to doing it fast, and that almost always backfires.
On the flip side, past failures can breed hesitation. Many organisations, scarred by projects that went wildly over budget or delivered underwhelming results, become overly cautious. They avoid taking bold steps, fearing a repeat of prior disasters. This reluctance to act decisively can be just as damaging as overconfidence, leading to missed opportunities and half-hearted transformations that fail to deliver meaningful change.
The result is a paradox: businesses want to be bold and innovative, but they often miscalculate the risks involved or shy away from action altogether. Without a clear, evidence-based approach to planning and risk assessment, organisations are left navigating uncharted waters without a reliable map.
Reference Class Forecasting: A Realist’s Approach
Reference class forecasting (RCF) offers a way out of this paradox. It’s a forecasting method that grounds predictions in the actual outcomes of similar past projects, known as the “reference class.” Instead of relying on optimistic assumptions or gut feelings, RCF asks:
What happened in projects like this before?
What were the key risks, costs, and timeframes?
How do the unique aspects of our project change these expectations?
By drawing on real-world data, RCF provides a reality check that’s particularly vital in digital transformation, where projects often involve new and complex technologies. It forces organisations to confront the realities of similar initiatives and adjust their plans accordingly.
Why RCF is Essential
RCF is invaluable for several reasons. First, it counteracts optimism bias by shifting the focus from what we hope will happen to what actually happened in the past. This doesn’t kill ambition—it grounds it in evidence, enabling teams to prepare for the inevitable hurdles.
Second, RCF addresses the fat-tailed nature of digital transformation projects. By highlighting outliers in the reference class, it ensures organisations are prepared for extreme outcomes. If 15% of similar projects faced budget overruns exceeding 100%, those risks can be built into the forecast, making failure far less likely.
Third, RCF helps organisations strike the right balance between boldness and caution. By showing what’s possible—and what’s probable—it enables leaders to take calculated risks without venturing into reckless overconfidence or paralysing fear.
Finally, RCF builds confidence in decision-making. Stakeholders are more likely to align behind plans that are anchored in historical evidence. When leaders can point to past outcomes to justify their strategies, it reduces uncertainty and creates a stronger foundation for moving forward.
A Path to Success
Digital transformation doesn’t have to be synonymous with failure. By recognising fat-tailed risks, countering optimism bias, and embracing evidence-based planning like RCF, organisations can dramatically improve their odds of success.
In a world of rapid technological evolution, success belongs to those who plan like realists but execute with optimism. It’s time to embrace the complexity of digital transformation, and start building plans that actually work.
Because when it comes to digital transformation, the only thing worse than not starting is starting with a plan that’s doomed to fail.