Overcoming Barriers to Innovation

At Semafind, we interact with a lot of companies from different industries and notice a common pattern when it comes to innovation. There are almost always the same invisible barriers that stop the flow of information and stifle out-of-the-box thinking. In this post, we will cover three main ideas and how to overcome them.

Let’s first define what innovation in this context refers to. The Cambridge dictionary defines innovation as “the creating and use of new ideas or methods” which aligns nicely to technological innovation and the use of new techniques such as machine learning and data science. Innovation is about doing something new, perhaps untested before, within the context of a problem statement. It is less about adding incrementally new features to an existing product or service that closely follow on from the status quo.

So what goes wrong?

It works, why innovate?

One of the most common responses we get is that the business is working, it has customers and they have a feature roadmap which nowadays often includes a vague integrate-with-AI step. It may completely be sensible to think that the business should just invest more money in marketing and short-term growth.

The issue is that a working product and additional features are in, what we call, the zone of proximal development. It is actually a term used in learning theory to define “the space between what a learner is capable of doing unsupported and what the learner cannot do even with support”. This is very familiar with businesses who understand their current product and what they can do next with a bit of help from their competitors or the information available on the internet. It’s within their reach, the developers can imagine what’s next, the executives can calculate what’s next and all of this nicely sits in a narrow channel of incremental adjustments.

For example, last year, we were in talks with companies demoing a chatbot based company assistant as the retrieval augment generation (RAG) was starting to prove useful. One of the responses we got was that regular fuzzy search works and their Elastic search backend served their needs perfectly. Fast forward to today and Elastic search is covered with AI, RAG and vector search.

How can you break this cycle? We recommend imagining that the product is being rebuilt from scratch for the problem statement to ask what would you do differently? The answer to that question often reveals bottlenecks, fundamental issues and more that the business is aware of but cannot easily address due to their current state, product, customers and market. That is the point when innovation starts since going back to the problem statement outside the confines of an existing product or service beckons new ideas and new technology.

It's unknown, we’ll wait.

A common misconception with innovation is that because it is fundamentally new and potentially has unknown results, business owners cannot make an informed decision on whether or not to pursue it until one of their competitors does it. So it is often left alone and the focus shifts on marketing and making incremental changes again until it is too late. It is perceived as a bigger risk and deemed unnecessary. Yet we would argue innovation is an essential part of any business and should be motivated internally rather than external pressures.

The most interesting example must be the recent ubiquitous integration of chatbots in every business and product. Few companies integrated large language models well into their services and the rest, we argue, fumbled down like when Air Canada’s chatbot gave an incorrect discount.

Motivating and trying new ideas internally sounds very simple but for most businesses it is actually a waiting game. The first step in all our meetings at Semafind is ideation and it is targeted to address this barrier. We want to know what is happening inside the company that might give an edge over their competitors.

We already do it.

The final piece of the puzzle is that sometimes companies truly believe they are already innovating. Their principal engineers are doing cutting edge work, the marketing team has incredible conversion rates and the product or service is unique in every way. While it sounds exciting, the reality is often an off-the-shelf data analysis tool that trains a machine learning model from a data lake coupled with a nice dashboard.

This isn’t to say that using existing tools cannot be novel, especially if applied in new domains. For example, using existing computer vision methods to detect and identify the spread of Asian hornets in the UK would be very novel. Younger companies and startups who are closer to their problem statement often have to innovate in order to bring a new solution to the market. However, once a critical threshold is crossed the shadow of the initial novelty looms over the business and lends itself to the “We already do it.” state-of-mind.

An easy way to gauge whether your company is still pushing new boundaries is to ask whether there is enough material for a peer-reviewed paper that can be published. It is not actually about publishing anything. Instead, knowing that you may be able to indicates there is new knowledge at hand. This framework also gives a scientific grounding to innovation and helps separate it from new product features versus truly novel ideas. Unfortunately, the specialised vocabulary and complex methodologies employed in research can be daunting for businesses unfamiliar with the field. Navigating the intricate web of scientific terms, statistical analyses, and theoretical frameworks can be overwhelming, posing a significant barrier to understanding and utilising research findings. At Semafind, this is where we try to help boost that confidence and innovate together.