By Ayazhan Zhakhan, co-founder and CTO at Dropbase.
getty
Artificial intelligence (AI) entered the main scene in the early 2010s and in the following years captivated our imagination by producing new and exciting results almost every month. There were models for images, audio, video, and it seemed like it would soon take over the world. AI-based startups were launching everywhere with the promise of capturing the market of the “new electricity.” Now, almost 10 years after the hype started, few promises were realized. What went wrong?
The first thing to understand is that AI has many flavors. The ones that took over our imagination were primarily machine learning (ML) models, specifically deep learning–based classification models. They attempt to predict the outcome based on a sample. For example, given a picture of a cat, they will label it as “cat.” They simply map input to output, and they’re really good at it.
It’s also important to point out that AI companies are different from traditional software companies. On the surface, they both work with software. At the core, however, AI startups are centered around data and predictions, while software companies focus on product and experience. AI industry comes with its own set of challenges, and design principles perfected in the software development world are only partially applicable to solve them. To better understand these challenges, let’s explore the process of developing an ML model a bit further.
ML models are built by sampling the real world as accurately as possible and training the ML model (which is simply a glorified mapping function) to give predictions based on those samples. If the training dataset was well sampled, then predictions from a model should be transferable to the real world. For example, if we collect one million pictures of cats and teach the ML model to recognize them as cats, next time we take a picture of a random cat, we’re hoping that this sample will look a lot like the cats in our sample dataset and our model can correctly label it as “cat.”
MORE FOR YOU
Having a good dataset is crucial since an ML model can only be as good as the dataset it was trained on. There are many ways to collect a dataset: scraping the internet, collecting internal records, purchasing from data providers or generating samples synthetically. To train a production-ready model, however, you need a sufficiently large dataset with diverse representation. Depending on the industry you might also need to address privacy and ethics concerns.
The next step is to train a model. Think of training as wiring a bot’s brain so a set of attributes are associated with a specific label—like associating ears and whiskers with cats. Luckily, we don’t have to do this manually. Instead, we show many pictures of cats and “not cats” to a bot and ask it to label each. When it gets it right, we reward it. When it’s wrong, punish. We repeat this process until the bot “learns” the difference. This might sound absurd, but thanks to the ingenuity of mathematicians and computer scientists, it works.
There is a catch, however. First, we need to know which model to use for a given task. Second, we need to configure it. Unfortunately, there is no better way of doing this except for the method of trial and error. Each training iteration could take anywhere from hours to days, and there are many parameters to tune, making the possible number of experiments enormous.
And it doesn’t stop there. Even after your model is fully trained, you have to constantly retrain it so it stays relevant in this constantly changing world. Say you trained a natural language model in 2019, when “corona” meant beer in most cases. In 2020, you have to update it so it also associates “corona” with Covid-19—which means you have to gather a new dataset that has samples with “Covid-19” and “corona” in it. Depending on the industry, you might have to retrain your model from once a year to once every hour.
Say you go through all these troubles and have a functional, always-up-to-date model. Now for the hard part—convincing people to use it. As humans, we’re reluctant to adopt anything we don’t understand. When it comes to ML, though it may seem easy to understand how it works, it’s difficult to grasp the exact path a model took to arrive at a specific decision. Furthermore, humans are much more sensitive to mistakes made by machines than by humans. Just like people don’t trust self-driving cars until they are orders of magnitude times better than humans, they won’t trust your model until it significantly outperforms humans. You might think this is irrational, but imagine being denied immigration status by a machine versus by a human.
Finally, because machine learning is such a young field, there are no established practices available. Those who can train a usable model, struggle to deploy it in production. Those who can deploy it, struggle with a shortage of specialized hardware. This is further compounded by a shortage of qualified talent. Development and deployment of a useful ML model requires the combined effort of a data scientist, data engineer, software engineer and infrastructure engineer.
Because of these reasons, most companies that were able to take advantage of AI were large tech corporations that could afford to bear the cost of the development and maintenance of ML models.
However, that doesn’t mean that no progress has been made in the field. Most of the advancements were happening in the background, and benefits are subtle. ML helped to reduce the time it takes to deliver your package. It also protected your bank account from fraudulent transactions and recommended that new show that you loved. The field is growing rapidly. Many of the technical challenges around training and deployment are being addressed by open-source projects. The rise of transformers dramatically reduced the cost of training models. Community-driven projects make it easy to share datasets and pre-trained models. Despite all the challenges, the ecosystem is flourishing and the future is promising.
