#56 – Problem Solving: Insights

The bright Data Scientist in your team has spent several intense days with a problem statement and comes to you with a bunch of Analysis and charts. She is all excited with her work – and your first question, ‘Where are the insights?’ I find some managers use this almost reflexively as a way to get their teams to dig deeper; at the same time, you can see the Analyst thinking, ‘What does he mean by that?’

This brings up an important question: What is an Insight? And perhaps more importantly, how do you get an Insight? I find that teams rarely, if at all, do a good job at the definition. And that in turn, leaves the business and data teams talking past each other when it comes to coming up with meaningful analysis. In this and the next post, I will try to define what an insight is and how is it that some tend to be better than others in generating insights in a variety of problem situations. But first, a detour and a story.

In the history of Astronomy and Mathematics, Johannes Kepler occupies a very important place. He and his mentor Tyco Brahe together made two foundational contributions that triggered an explosion in Physics and Mathematics that soon followed starting with the likes of Galileo, Newton et al. There is a fascinating story of how Kepler arrived at his seminal contribution – the planetary laws of motion. It started with Kepler’s attempts to calculate Mars’ orbit. After several iterations, he came up with a model that generally agreed with Tyco’s observations, except that the model deviated from the data at several points. By then, it was generally accepted that the planetary orbits were not circular with the Sun at its center – the question in some sense, was: if it is not a perfect circle, where exactly is the Sun in relation to the planets? The prevailing religious view of the cosmos was that the Sun was the source of motive force in the Solar System – Kepler extended that to assume a ‘magnetic soul’ of the, which weakens with distance, causing faster or slower motion as planets move closer or farther from it. And in what must have been a flash of inspiration, he hit upon the idea of the elliptical orbit and his two celebrated laws of planetary motion: all planets move in ellipses, with the Sun at one focus [First Law] and planets sweep out equal areas in equal times [Second Law]. And as they say, the world of Physics was never the same again.

Psychologists have been studying ‘insights’ for a while now: how do you define one? How is it that some are able to see what others don’t? One definition that I particularly liked is from Gary Klein who has one of the leading researchers in this space: An Insight is a discontinuous discovery. In my experience, Insights are the discovery of new patterns, which could come from:

1. Combining seemingly disconnected ideas to create something that is fundamentally different. Steven Johnson has written extensively about serendipity and slow hunches² that often lead to breakthrough ideas – one of the examples being the invention of GPS.
2. Study data points that do not fit the pattern – and use them to draw insights. Kepler did that – instead of focusing on the accuracy of his model, he focused on the data points that did not fit the model: and eventually, led him to develop the laws of planetary motion
3. Study patterns over and over again – and draw insights by abstracting from the patterns. Some of the most important discoveries in medicine have come from the careful compilation of observations and the ability of a few to abstract the underlying causes. Michael Gottlieb helped identify AIDS as a disease by studying a series of patients over a 2 year period – the key thing here being he published his observations early so that he could get other physicians to report similar cases. At the time, this went against the conventional wisdom of publishing in medical research journals of reporting only after rigorous testing, something that would have taken several years.

In all these cases, the real insights came from an approach that essentially pushed thinking to turn a discontinuous discovery.

One final point: it is useful to draw a distinction from intuition – which is typically the use of patterns already learned. This is an important one: we have all seen the crusty old manager who has been around for so long so that he has been through ‘all possible scenarios’ and is wont to rely on intuition to fit historical patterns to the problem at hand. Which may not work – especially when you run into situations that are so different from anything you have encountered in the past, and it is no loynger good enough to rely on intuition to re-use a pattern from the past. After what we have been through starting 2020, this is no longer a ‘what-if’ scenario.

References:

1. Gary Klein and his work: Recognition Primed Decision Model – more on this in the next post
2. Where good ideas come from: Steven Johnson
3. The discovery of AIDS