Why Data Driven Decision Making Fails
We have all heard of data-driven decision making, the admirable desire to make better decisions through data. The conventional justification is that given the world’s complexity, organizations need data to make decisions.
Such ideas exist across organizations from governments to companies to hospitals. In fact, they are wrong in principle and misleading in practice. Data-driven decision making puts the emphasis in the wrong place, and means organizations focus on the wrong thing.
So what’s Wrong ?
Let’s go back in time to understand why. For thousands of years, it was thought that understanding, and therefore decisions, came from divine mandate. About 400 years ago, philosophers realized that collecting data to create understanding was a good thing.
However, they also thought data alone was sufficient to establish how the world was and predict what would happen next, a process called induction. They thought a wider understanding of what was going on didn’t matter. Notice this is the same claim made for data-driven decision making, but we know a wider understanding does matter.
Will stars appear in the sky because they did yesterday? Well, yes, for a while. But at some point, they will burn out. What was an obvious extrapolation is, suddenly, no longer true.
This view changed with the philosopher Karl Popper, who said we don’t extrapolate inductively from data, because that’s impossible. In fact, we guess what’s going on, then find data to falsify that theory. This is a crucial change. Suddenly, the focus is the theory, not the data. This means the theory can be very different from an extrapolation from data.
So, the first stage of decision making was divine truth is everything, the second stage was data is everything, and the third stage was theories are proposed and refuted through data.
Businesses have gone from stage one to stage two over the past few decades. Shifting from CEO mandates to looking at data is simple enough, but we still haven’t made the leap from data to theories.
Decision Intelligence Is The Future
Decision intelligence is one of the most important contemporary applications of artificial intelligence (AI). This next generation of technology will have far-reaching consequences for how organizations make decisions, and it comes at a much-needed time.
We need to move beyond the spreadsheet hell of data-led decisions. We need a revolution in how our leaders approach decision making.
And, just as Popper taught us, we need to move to decisions based on understanding and that means true decision intelligence.
