Last week, I talked about Complex systems and our limits to understanding. And when we do find ourselves in such situations, we are prone to fill the gaps or work through our limits of understanding with stories. Stories by themselves are not wrong – in fact, we are probably the only species on the planet that has created a collective consciousness through stories. The question that I am curious about is: how we as individuals and decision makers, parse stories and statistics. And as anyone who has worked in a corporate environment knows only too well, there are more than a fair proportion of decisions that are influenced by anecdotal stories. Michael Lewis in his book, The Undoing Project, does a fantastic job of chronicling the work of Amos Tversky and Daniel Kahneman, who together pretty much rewrote the book on Economics.
A brief history
As always, it is instructive to look at the history of statistics. The idea of counting (population, armies, livestock etc) has been around for a long time with evidence from several civilizations. Statistics – which is really the science of analyzing samples and making inferences about the population as a whole – is a more recent discipline. In 18th century, a mathematician by the name of John Arbuthnot1 published a paper on what can be arguably called the first instance of using data to fit into a story. He examined birth records in London for 82 years (1629 to 1710) and the human sex ratio at birth (males/females) and discovered that in every year, the number of males born in London exceeded the number of females. Assuming that the probability of male and female birth is equal, the probability of the observed outcome would be (1/2)82, which is a really small number. And that led him to conclude that there is divine providence at work: “From whence it follows, that it is Art, not Chance, that governs”. I don’t want to get into the logical fallacies in this argument – but he does get the credit for starting statistics as a formal science.
What does it mean to us?
The story, like most of statistical inferences, points to an important feature of all data analysis – you create a story from the analysis and how you tell the story is the lens through the rest of the world looks at the analysis. And so, I don’t believe that it is a case of Stories vs. Statistics (Art vs. Science) – it is actually an interplay of story-telling and statistics (Art and Science). As a decision maker in an organization or in daily life, it is futile to expect complete objectivity – a much better idea is to be aware of the biases and as always, treat all analyses outputs with healthy skepticism. While I don’t want to get into an exhaustive list of cognitive biases, here are a few of my observations/what I have seen over the years:
- As anyone who has read fiction knows, we are often willing to suspend disbelief when it comes to fiction. And conversely, we tend to be more skeptical when it comes to statistics – I often think we tend to suspend belief in order to avoid the cognitive trap of attributing of objectivity to a result just because it is presented as data. And so, when you are presented with an analysis story, make sure that you apply the skeptical filter, triangulate with other analyses, look at the error metrics – in other words, engage and ask the hard questions.
- We tend to be more rigorous with error metrics when it comes to data analysis – as any data scientist would tell you. We need to be thinking of this idea of errors more broadly with stories and statistics – and more importantly, understand the mindset of the managers and the business context:
- Type-I (False Positive): we tend to observe what is not there. The most common problem here is that of generalization – beware of using anecdotal stories to draw broad conclusions and drive decisions. And also stands to reason that people who don’t particularly trust stories or who think about the prospect of making a Type-I error might prefer statistics to stories. And so, when you are looking for someone in your Risk team, you might want to check for someone who prefers statistics.
- Type-II (False Negative): we tend to miss what is there. This is the blind spot problem – where we are presented analysis that looks at a specific hypothesis and shut out other possibilities. And so, people who like stories or wish to avoid Type-II errors might prefer stories to statistics. There are situations when your Supply Chain team might have to make inventory stocking decisions with very limited data points – e.g. during a product launch when the cost of a stock-out tends to be very high. In such situations, it is better for the planner to trust her intuition on the basis of a few anecdotal data points as opposed to waiting for a fully data driven approach.
- The one thing that stories do is to give rise is the possibility of the ‘conjunction fallacy’2. Stories tend to lure us down the path of assuming that specific conditions are more probable than a single general one. Decision makers in organizations are especially prone to fall into this trap – make very specific conclusions based on the story that gets told from data analysis. This tends to happen in a fast-moving environment with a lot of noisy data – e.g. the web analytics teams might look at a sudden spike in funnel drop-offs for a specific customer segment and product level to infer a behavioral trend across all segments and products.
And before I close, no harm in stating the obvious: we are all but human and are likely to prefer a good story to dry statistics. We are not going to change any time soon – at the same time, it is important for managers to be aware of their own biases when they are presented with stories from data. Like I said, it is less about Stories vs. Statistics – and more like Stories and Statistics.
References:
Further reading:
The Undoing Project by Michael Lewis is a fantastic read. This is a great review by two very accomplished economists (Cass Sunstein and Richard Thaler): https://www.newyorker.com/books/page-turner/the-two-friends-who-changed-how-we-think-about-how-we-think
Celebrating the learning and curious mindset 
Leave a Reply