Recently, Deep Learning has revolutionised a multitude of technical fields, most notably Computer Vision and Natural Language Processing. The implications for business are enormous, with the ramifications of these change still filtering through the mostly technology-shy world of business. There are signs that Combinatorial Optimisation might be the next technical field to undergo Revolution-By-Neural-Network. If this comes to fruition, then the impact on business will be seismic.

Combinatorial Optimisation

Combinatorial Optimisation (CO) sounds very dry and technical when defined; It is the process of finding an optimal object from a set of objects. The Travelling Salesman Problem, a classic and well-studied problem in Computer Science, falls into this category, as do problems as diverse as vehicle routing and workforce scheduling.

CO problems are so ubiquitous in industry that it’s fair to say that every business of at least a medium size is impacted by them (whether they are aware of it or not), as well as a good many smaller ones. For some problems, such as those found in supply chain management, entire industries exist merely to assist with small parts of that overall challenge.

There are, of course, many methods currently employed to deal with problems arising in CO. Any number of software companies and consultants offer solutions in supply chain management, for example. So why is it that solving these issues with Deep Learning would be such a revolutionary event?

AlexNet Moments

To understand this, we need to look at the recent history of Computer Vision (CV) and Natural Language Processing (NLP). As recently as a few years ago, solutions to both CV and NLP problems possessed the following characteristics:

1. Highly specialised algorithms.
2. A “big bag of tricks” approach.
3. Scale is a problem.
4. A great deal of subject-matter expertise.
5. A general lack of success in solving real-world problems.

Today, both NLP and CV problems are addressed with Deep Learning, with the following set of characteristics:

1. Very general algorithms.
2. Consistent approach.
3. Scale is an advantage.
4. A great deal of Deep Learning expertise.
5. Huge success in solving real-world problems.

Former specialists with years of inherited wisdom about how to approach CV and NLP were not particularly happy with this sea change, but the results speak for themselves: High quality, scalable NLP and CV are available to any company that wants them.

The turning point for CV was the 2012 release of AlexNet - a Convolutional Neural Network (CNN) that won the ImageNet Large Scale Visual Recognition Challenge. After AlexNet, the only methods to win image recognition contests were based primarily or solely on CNNs. Similarly, in 2017, the publication of the Transformer architecture for NLP problems in a paper entitled “Attention Is All You Need” created a paradigm shift in NLP model performance. The events that unlock a new, Deep-Learning-based approach to a field are called “AlexNet moments”.

The Bitter Lesson

The particular Neural Network architectures of CNNs and Transformers per se are not very important. As I have previously written, what is important is that both derive their power from three critical characteristics:

1. The method is general.
2. The model architecture allows for scalable computation.
3. The quality of the model scales well with compute.

These “Bitter Lesson Conditions” derive their name from Richard Sutton’s seminal essay The Bitter Lesson, which begins:

The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin.

The observation is simple, but the implications are profound; when we find a method that satisfies these three conditions, we revolutionise a field. At the time of writing, Automatic Speech Recognition (ASR) is going through an AlexNet moment because we have discovered methods that satisfy the Bitter Lesson Conditions.

The Next Frontier

Returning to the realm of Combinatorial Optimisation, we can see that solutions to CO problems presently have the following characteristics:

1. Highly specialised algorithms.
2. A “big bag of tricks” approach.
3. Scale is a problem.
4. A great deal of subject-matter expertise.
5. A general lack of success in solving real-world problems.

These are the exact conditions that pertained in CV and NLP problems prior to their respective AlexNet moments. What CO currently lacks is an algorithm that satisfies the Bitter Lesson Conditions.

Excitingly - or worryingly, depending on your perspective - there are signs that this may be changing. Researchers are actively working towards scalable Deep Learning approaches to CO problems. For now, the problems are toy problems and classical algorithms are still the go-to solution for difficult cases, but it is hard to believe that the same forces that have rapidly transformed stubborn domains such as CV, NLP, and ASR will not eventually crack the CO nut.

Back to Business

The upshot for business is that while CV and NLP may be critical technologies, they are small beer compared to the world of CO. Combinatorial Optimisation problems are so core to the operations of so many businesses, that the explosion of solutions and software unleashed by an AlexNet moment would be Cambrian in scale, and the impact on the business landscape incalculable.

Given how fundamental such processes are to many businesses, and given also the scale of the commercial advantage that early adopters will obtain, I hold out some mild hope that this may be the technology that finally breaks the “IT is overhead” attitude of many large enterprises. With enough determination from management, recent advances in CV and NLP can be kept at arm’s length from the core operations of the business, and it is apprently still possible for an executive in 2021 to truly believe that technology is best bought and stuck in the corner, far away from where the action truly happens. The business-critical nature of CO problems will, with any luck, force executives to confront the reality that ongoing effective use of technology is now a real strategic imperative, not a cost to be born.

Neural Networks in the Boardroom

It’s a long bow to draw from experimental methods that may be just over the horizon to my personal vendetta against the exclusion of technologists from key strategic positions in traditional companies, but the ingredients for an AlexNet moment are all there - not just for Combinatorial Optimisation, but for business itself.