18th century Europe was a particularly exciting time if you were in the seafaring business. There were multiple ventures, several with vast capital investment attempting some very bold goals (most of these ventures would be outright illegal or egregious by today’s standards or ethical standards of any age – colonial ambitions, slave trade et al. But that is a topic for another day and blog). There was a fundamental problem in all these sea expeditions – how do you determine where exactly you are in the ocean in the absence of any landmarks? This had huge practical implications – losing your way could be fatal to the sailors, and terrible returns for the investors. Some of the best minds of the day, including Sir Isaac Newton, were called upon to solve it, without any success. And then someone from the British government hit upon an idea: why not institute a prize for solving this problem and open it up as a challenge? It was called the Longitude Prize and was to be given to the person who could devise a method to determine the exact longitude of a ship at sea. There were in fact, several prizes, in increasing amounts, for solutions of increasing accuracy. This was a masterstroke – it motivated many people from diverse professions to jump in. And in the end, it was John Harrison, a clockmaker who won the prize. There is a fantastic book, Longitude: The True Story of a Lone Genius Who Solved the Greatest Scientific Problem of His Time by Dava Sobel1 that chronicles this story.
One of the reasons that this worked (and has been replicated in multiple cases) is because of a key construct – what Steven Johnson has described as the alignment of 3 peer networks:
- The network that is incentivized to solve the problem
- The network that is incentivized to work on a solution
- The network that is available to take the solution and scale it up
And bringing these networks together requires two major design considerations:
- A prize that is reflects the economic value of the problem being solved. Network-1 should be willing to put that up – in case of the longitude challenge, it ran into millions (in today’s currency) and was offered by the ship owners and the investors
- Open sourcing the idea – for Network-3 to step in and scale the solutions emerging from Network-2, this is critical. And this was the masterstroke that really helped the Longitude challenge to take off. Clock manufacturers were ready to jump in and scale up John Harrispn’s idea once they had access to the design
In other words, this is a classic case of an intervention where the market failed to come up with a solution. With an important difference – the intervention is less to do with a market failure and more of a market accelerator that can nudge a solution which otherwise, would have taken years to find.
This approach has been replicated over the years, most recently the X-prize. To be sure, crowdsourcing is not always the best option – the biggest risk being that of being inundated with frivolous solution ideas, which then sucks up precious resources into validating and seeking out a workable solution. Nevertheless, there certainly is a lot of merit in going beyond the core networks to seek out solutions and at the very least, foster collaboration among groups that would otherwise have no reason to come together to solve a problem
Going beyond core networks
All this is fine as a lesson in history, but what good would it to do in an organization, you might ask? I think this could be a template for scaling problem solving in organizations – especially when you are looking to solve large, knotty problems that tend to span across functional boundaries. Hackathons have been around for a while – but their effectivity in harnessing the problem-solving talent in the organization has been patchy at best.
Here are two ideas that have worked for me in the past:
- Prize based competitions: The goal here is not just to create a prize for the best submission of a solution blueprint or even a prototype, but get the winning team to go all the way and implement the solution in a production environment. Add the constraint that the team must be cross-functional which would hopefully draw on the long connections. These could be one-off contests to get teams together to solve the really hard problems – and to motivate the teams, you need to have a substantial prize (not just a dinner coupon!) Take the problem of improving call deflection rate by predicting call intent. In any technology company, this is a cross-functional issue cutting across Customer Support, Product Management and Sales and requires a combination of business process, data and data science capabilities. It might be a worthwhile idea to launch a prize competition with clearly defined success metrics to judge the prize outcomes. These work best when treated very differently from hackathons (which often tend to be long on hype and short on substance)
- Solution marketplaces: While it is absolutely essential to explore prize based competitions, it is also obvious that we don’t always have the luxury of creating structured prize based incentives. The idea here is to devise and execute a platform approach that enables the sharing of assets – say code snippets, data sets etc. across teams. Technology plays a pivotal role here: can you enable an enterprise wide platform to do the sharing – in other words, just as github is to centralize code repositories and in turn, enable a sharing ethos, can you extend the idea to all the stages of your problem-solving process? Especially for algorithms and datasets. Of late, there is a powerful idea around data sharing: Feature stores2 which allow teams to create a ‘marketplace’ of datasets – teams publish features, training/validation datasets etc. that can be used by other teams in related projects. Generally speaking, these kinds of peer networks do not need extrinsic motivation – the idea of ‘social honor’ of a peer consuming an asset created is often a powerful motivator. The trick is to reduce the technology friction of sharing assets – and then these mechanisms can go a long way in developing smaller ‘peer networks’
References:
Celebrating the learning and curious mindset 
Leave a Reply