#59 – Learning: Exploring the ‘Adjacent Possible’

Isaac Newton, when asked how he was able to work out some of the fundamental laws in Physics, is said to have responded: “By standing on the shoulders of giants”. A couple of centuries later, a reported asked Einstein if he believed that his stunning contributions to Physics were ‘by standing on the shoulders of Newton’. To which Einstein is said to have retorted – ‘actually, by standing on the shoulders of Faraday and Maxwell’. The humility of these extraordinary geniuses is striking – but that is not the point of this post. This post is about the fact that what seems to be flashes of original genius is probably a series of incremental steps, culminating in a game-changing outcome. And this seems to happen through a series of explorations and discoveries built on existing knowledge. The physicist/author Stuart Kauffman calls it ‘the adjacent possible’¹

This is nowhere more apparent than in Science – where just about every path-breaking discovery and invention has been a series of such adjacent possible moves. And every once in a while, someone comes along who makes that incremental breakthrough contribution that sets off a whole new chain. Newton gave the world particle physics – his laws worked spectacularly well in explaining the interaction between objects at a distance. What they did not tackle was how the forces ‘traveled’ through space (and as it turned out later, time). Along came Faraday – he was less of a theoretician but a brilliant and more importantly, imaginative experimental scientist. He conjured up the idea of a field that ‘carried’ the force between objects; Maxwell took that idea up and proved it mathematically. That set the stage for Einstein to formulate the Theory of Relativity that finally explained the behavior of object interactions in space and time. In this case, Faraday was a critical link in this chain – this is yet another instance of how science makes progress by exploring the adjacent possible, building on the ‘shoulders of giants’.

The real question then is: how can we enable Learning by exploring the adjacent possible? This is a topic that has been researched extensively – my focus is specific to Data. Over and over, I have seen two patterns:

1. Every team ends up creating fresh datasets for each analysis effort – it is often unavoidable, as it is (and should be) driven by the problem you are trying to solve. However, it also stands to reason that the problems are more often than not, related – and given that, there is no reason why the datasets should not be re-used. Lots to be done here.
2. At a more fundamental level, there has always been a separation of code and data. Over the years, Software Engineering practices have given modular coding practices, code repositories etc. that are designed to enable re-use across projects by engineers. That same rigor has never translated to data

Why is this issue relevant now? As enterprises are beginning to realize, this mindset worked fine so long as the analyses were offline, point-in-time exercises – for a given question, generate the insights and move on. That is changing (and rightfully so) as we move into a world where the volume and frequency of data analyses continues to grow exponentially and more importantly, data driven analysis is increasingly integrated into the decision-making process. Integrating and automating data driven actions into the operating workflows requires a whole level of rigor in managing data.

In the next post, I will explore the idea of ‘Data as Code’ – what does this mean, why is it important and how to make it happen in the Enterprise data ecosystem.

PS: Michael Faraday trivia: he never received any formal education in science – he was completely self-taught, that too while he was apprenticing at a bookbinder which gave him access to books on science that he used to read and compiled his own notes. And critically, he sent the notebook to the Sir Humphry Davy, the celebrity scientist of his time – who was impressed by Faraday and got him an appointment in his laboratory. Yet another story of the role of serendipity in scientific progress!

Further Reading:

1. Stuart Kauffman is a theoretical biologist, and perhaps best known for coining the term: The Adjacent possible
2. Excellent story of how Faraday and Maxwell revolutionized Physics: Book by Nancy Forbes
3. This is of interest to anyone who is remotely connected with Enterprise data: Data as Code