Complex Systems: Limits to our understanding

Over the last several weeks, I have been writing about decision making, with the focus being how to get better at navigating VUCA environments. This has been rooted in the paradigm of reasoning and a deterministic view of the world. Which to be sure, works pretty well in most situations. And then we got hit by the pandemic – and among many things, it seems to have laid bare the limits to our pretense of understanding a deterministic world. A brief journey into the history of scientific thinking. This is obviously a rich area in philosophy and religion and I do not even pretend to scratch the surface – and I don’t want to get into that now. And as always, let’s start with a bit of history.

Determinism and Chaos in Science

Newton did way more than being a prodigious mathematician and physicist. Newtonian laws gave a massive boost to the determinists – his laws led the world to embrace the idea of determinism – after all, if you can neatly described all known physical phenomena with his laws, where is the ‘invisible hand’? And they do work very well in our everyday world. In the 19^th century, the French mathematician Laplace theorized that Newtonian laws, derived from the present conditions, could predict anything that might happen in the future. And he mathematically proved it and in popular lexicon, it was called the ‘Laplace Demon’ – an ‘intellect’ that given all the data about the current state of the natural universe, could compute all the possible outcomes into the future. This is a thought that continued to persist through mid-20^th century, even through Einsteinian physics, quantum mechanics et al. Although it was pretty obvious in the scientific circles that Newtonian physics fell short in several conditions, the idea of determinism clung on, particularly in our everyday phenomena.

And then in 1961, Edward Lorenz stumbled on something that has changed our thinking of systems ever since. Lorenz was a mathematician who was interested in weather patterns – and using computers which had begun to show up around that time, he built a prediction model that would take various parameters as inputs and spit out a weather forecast. One evening, as the story goes, he ran the model and stepped out for a cup of coffee. He came back and on a whim, wanted to run the model again – only that this time, he rounded up the data to the 3^rd decimal place (down from 5 places in the original model). And to his shock, the model started diverging and eventually, the output was a completely divergent picture of the weather phenomena. This eventually led to a series of papers, and his coining of the term ‘butterfly effect’: systems that are highly sensitive to small changes in initial conditions and even an infinitesimal change can lead to completely different outcomes. Extending this, it quickly became apparent that it is impossible to analyze all initial conditions with the accuracy necessary to predict all possible outcomes.

This led to a flurry of research into Complex Systems – and as often happens, some clever researcher coined the term ‘Chaos Theory’. And then Hollywood did the needful to make all of this fully mainstream: in Jurassic Park, Jeff Goldblum’s character talks about the ‘butterfly that can flap its wings in Peking and in Central Park, you get rain instead of sunshine’

The study of Complex systems, in my mind, is just getting started – and what is fascinating to me is where this will take us: will we come to accept that the universe is inherently unpredictable or it is just that we cannot yet see deeply enough into the factors that shape all events? For now, Chaos Theory is built on the premise that a seemingly random process is may in fact be generated by a deterministic function that is not random

So what?

All this fine for a weekend rumination on science and the nature of the universe, but what can we all learn from all of this, especially when we are tasked with trying to make decisions based on educated guesses of the future (aka forecasts)? We have to come to accept that we are dealing with Complex systems – how do we better navigate through this complexity? Here are 3 takeaways:

1. Wider data collection: Your team obviously cannot capture all the factors in their entirety: but can they keep doing better incrementally? Have they added new features, interpreted the current features better over time?
2. Better modeling: Does your team rely on just one model or have they adopted ensemble models? That is becoming de regeur with most prediction models. The interesting question here could be: have your teams tried to experiment with (slightly) different initial conditions and converge on a consensus forecast?
3. Focus on shorter-term prediction models: Conversely, you should treat long range forecasts with suspicion. And know that the traditional models such as linear time-series regression do not work very well with chaotic time series data. And the current pandemic situation and the uncertainty around pretty much every business is a classic petri-dish for chaotic data. Your teams might have to look harder to explore other alternatives

Quick note: I have not used ‘chaos’ in its typical dramatic sense: confusion, disorder. I have kept it to its formal scientific use. Thought I would clarify in these weird, polarized times.