One morning in 2019, Adebayo Alonge was preparing to enter a hotel room in Cape Town, South Africa, to present his startup's AI-based solution to a serious healthcare problem in Africa: detecting counterfeit medicines, which claim the lives of thousands of people across the continent every year.

RxScanner device for monitoring fake medicines

AI device for monitoring fake medicines

The RxScanner is a handheld spectrometer that scans a pill with infrared light, then sends the molecular profile (molecular composition) of the drug to an AI model equipped with a pharmaceutical database. Within seconds, the AI identifies the medication by its molecular profile or flags it as counterfeit.

Pharmacies were using the system in more than a dozen countries, including Ghana, Kenya, Myanmar, and Nigeria, Alonge's home. But that morning in South Africa, the system did not work. 'I was shocked,' Alonge says. The spectrometer was connected to the AI model, but the data center was 14,000 kilometers away, and the internet speed was limited. IEEE Spectrum reported Alonge as saying: 'Our server was in the United States, and it was taking me over 5 minutes to get a single scan result.'

So, Alonge immediately asked his engineers to shrink the AI model into a smaller, low-power, offline version that could run entirely on his Android phone. They accomplished this in two hours, saving the demo. More importantly, this work resulted in a new version of his device capable of verifying the authenticity of a pill in places that lack high-speed internet, computers, or even reliable electricity. Alonge also became a proponent of this type of 'TinyML'.

TinyML for Global Healthcare. TinyML differs significantly from large language models (LLMs), hyperscale data centers, and multi-billion-dollar investments, and it is far removed from the debates over AI consciousness occurring in wealthy nations. But for millions of people around the world, the only AI that matters—and often the only type available—is 'TinyML'.

According to a World Bank report released in November, only 0.7 percent of internet users in the world's poorest countries used ChatGPT, compared to a quarter of internet users in more advanced countries.

Ajay Banga, President of the World Bank, said at the World Economic Forum in Davos last January: 'Most people discuss AI from the perspective of machine/generative learning. But that requires immense computing power, electricity, massive data, and qualified personnel to manage it. Outside the developed world, with the possible exception of India and China, very few countries have these components.'

In contrast, small AI systems can provide useful, and even life-saving, services to populations in areas that lack all these resources, as Banga explained.

Monitoring Agricultural Pests and Heart Disease

In India, where government AI plans call for the development of more small AI systems, many of these systems work for the benefit of farmers.

For example, a drone-based system, developed by Bala Murugan and his colleagues at the Vellore Institute of Technology in India, captures images of cashew plants and identifies those showing spots indicative of disease quickly. All processing is done on the drone itself, so there is no need for an on-site computer or a connection to a central server.

Using small language models trained to solve a specific problem, and sometimes running on cheap, low-power hardware, other TinyML applications have been developed to identify ant infestations on grape farms in Uruguay, detect the presence of malaria-carrying mosquitoes in several countries, and perform electrocardiograms from an Arduino device in parts of Brazil lacking more complex equipment.

Marcelo Jose Rovai, a professor at the Institute of Engineering and Information Systems at the Federal University of Itajubá in Brazil, who participated in all three projects, says: 'This is the most important field in AI today. It is growing very fast.'

Small Low-Power Models

For Alonge, Rovai, and other proponents of these trends, TinyML is not just a 'promising trend,' as the World Bank report described it, but it may be, in the long run, the optimal form of AI that will impact the lives of the greatest number of people and remain sustainable after some large models have become too expensive for most users.

Alonge says: 'I believe the future of AI does not lie in one massive model at the center of the system, but in millions of small, precise models distributed widely, where each model solves a specific problem in a specific context.' This is partly because a large segment of humanity—including residents of some wealthy and developing nations—cannot access the latest cutting-edge models. But he adds that the reason is also that these models are unsustainable.

There is no precise definition for the term 'TinyML,' but it is usually used to describe language models that do not exceed a few billion parameters. (Compare that to state-of-the-art models that may have over a trillion parameters.) This small size is enough to run them directly on a phone or a Raspberry Pi computer. This allows these applications to work on devices without needing a connection to a data center, consuming only a few watts of power, which is often derived from a battery or a solar panel.

Rovai says that despite their small size, these models are not fundamentally different from large AI models. Many small language models have been created by 'pruning' large models, or removing parameters unnecessary for the task. The result is a system that is less capable overall, but still excellent at the specific task for which it was pruned, says Rovai.

As for other small models, they are created through 'distillation.' Rovai says these systems are trained to simulate a large model, until their performance approaches that of the 'teacher.' In other cases, the precision of the larger model is reduced, for example, so that a model running on a 32-bit architecture can run on 8-bit designs. In cases where a machine learning application is used to classify data or predict patterns (such as ant invasions), it is trained from scratch on a small device, not derived from a larger model at all.

Running on a phone