Why do small, on-device AI models matter for ordinary businesses?

Small AI models now run on ordinary phones and laptops. For a business that means summaries, rewrites and translation with no per-request fee, text that stays on the device, and features that work offline. The catch: they know less than cloud models and need recent hardware, so plan on 16 GB of RAM in a business PC.

On-device AI means a small language model runs on the phone or laptop itself rather than on a cloud server. For an ordinary business it matters for three practical reasons: routine jobs such as summarising, rewriting, translating and sorting text carry no per-request fee, the text does not leave the device, and the feature keeps working when the connection drops. The trade-off is that small models know far less than the big cloud models, so they suit narrow, repetitive tasks rather than open-ended questions.

This stopped being a lab curiosity during 2025. Apple and Google now ship small models inside the operating system or the browser with no charge per use, and Microsoft has set a hardware standard for local AI on Windows. Here is what is available, what it is good for, and what it means for your next hardware purchase.

fewer parameters needed to score above 60% on the MMLU test, 2022 to 2024: 142x; fewer parameters needed to score above 60% on the MMLU test, 2022 to 2024
parameters in Apple's on-device model, stored at about 2 bits per weight: ~3B; parameters in Apple's on-device model, stored at about 2 bits per weight
memory the smallest Gemma 3n model can run in, according to Google: 2 GB; memory the smallest Gemma 3n model can run in, according to Google
RAM a PC needs before Microsoft counts it as a Copilot+ PC: 16 GB; RAM a PC needs before Microsoft counts it as a Copilot+ PC

What changed: small models got good enough

The Stanford AI Index 2025 puts a number on it. In 2022 the smallest model to score above 60% on MMLU, a broad general-knowledge benchmark, was Google's PaLM with 540 billion parameters. By 2024 Microsoft's Phi-3-mini reached the same mark with 3.8 billion, a 142-fold reduction. A model that size fits in the memory of a recent phone or laptop.

Compression did the rest. Apple's on-device model has roughly 3 billion parameters squeezed to about 2 bits per weight. Google says its open Gemma 3n models run in as little as 2 GB and 3 GB of memory, accept text, image, audio and video input, and support 140 languages for text.

None of this makes a small model as knowledgeable as a large cloud model. It makes it good enough for office work: tidy this email, summarise this call, pull the invoice number out of this document.

Where on-device AI already runs

On-device AI built into mainstream platforms, April 2026
Platform	What is built in	Charge per use
Chrome on desktop	Summarizer API on Gemini Nano, plus Translator and Language Detector APIs, all local and stable since Chrome 138	None
Android	ML Kit summarisation, proofreading, rewriting and image description on Gemini Nano, on supported phones	None per call
iPhone, iPad and Mac	Foundation Models framework in iOS 26, iPadOS 26 and macOS 26, giving apps Apple's on-device model	Free, says Apple
Windows Copilot+ PCs	An NPU of 40+ TOPS for Microsoft's local AI features	Built into the PC
Your own hardware	Open models such as Google's Gemma 3n, run with your own software	Hardware and setup time

In practice, if your staff use Chrome on a decent computer, a developer can add summarise and translate features to your web app, and if your customers carry a recent iPhone or a supported Android phone, summarise and rewrite features to your mobile app, all without sending that text to a server.

Two limits are worth knowing. Chrome's built-in APIs are desktop only for now, and for the Gemini Nano features its documentation asks for 22 GB of free storage plus either a GPU with more than 4 GB of video memory or 16 GB of RAM and four CPU cores. Language coverage also varies: Chrome's built-in translator handles Hindi, Tamil, Kannada and Telugu but not Malayalam. Test the languages your customers actually use.

The cost argument, stated honestly

Cloud AI is already cheap per request. The AI Index found that the price of GPT-3.5-level performance fell from $20 to $0.07 per million tokens between November 2022 and October 2024, a drop of more than 280 times. So on-device AI is rarely about saving money on a single request.

It is about habit: a cloud bill grows with every summary, every staff member and every customer who taps a button. On-device features have no meter running: Google says its ML Kit GenAI APIs carry no additional cost for each call, and Apple describes its framework's inference as free of cost. You pay once, in hardware and development time.

A hypothetical example: a clinic in Kochi wants a short summary of every patient call for the front desk. In the cloud, the bill rises with call volume and each transcript goes to an outside provider. On a capable office laptop, the summaries cost nothing extra and the transcripts stay on the machine. The second is simpler to explain to a patient.

The privacy story gets simpler, not automatic

Chrome's documentation is unusually plain: after the one-time model download, no data is sent to Google or any third party when the model is used. Google's Android team says input, inference and output are processed locally and that the features work without a reliable internet connection, which matters wherever mobile data still drops out.

This weighs more now that India's Digital Personal Data Protection Rules are notified. According to MediaNama, they were published in November 2025, with core obligations on businesses, including reasonable security safeguards and breach notification, applying 18 months later. Every cloud service that touches customer data is one more party in that chain. Local processing shortens it.

Buying hardware in 2026: memory is the spec that matters

If you are replacing PCs because Windows 10 support ended on 14 October 2025, buy with local AI in mind. Microsoft's Copilot+ PC standard is a useful yardstick: an NPU capable of 40 or more trillion operations per second (TOPS), 16 GB of RAM and 256 GB of storage. Machines below that can still run local AI workloads, just not Microsoft's Copilot+ features.

The complication is price. In December 2025 IDC reported that a memory chip shortage, driven by AI data centres, had major PC makers confirming price rises. By February 2026 it expected PC shipments to fall 11.3% this year, warned that some makers would cut base RAM instead, and said AI PCs have so far failed to deliver what was promised. So do not pay extra for an AI sticker, and do not accept 8 GB of RAM to save money either.

16 GB of RAM as the floor for any business machine you expect to keep for four years.
An NPU of 40+ TOPS if you want Windows Copilot+ features.
At least 256 GB of SSD storage. Chrome will not download its model unless 22 GB is free.
The memory configuration in writing. In a shortage, the same model name can ship with less.

Know what small models are bad at

That candour is worth borrowing. Small models are weak at world knowledge and current facts. They are strong at transforming text you hand them: Apple's own list, summarising, extracting entities, refining text and short dialogue, reads like an office job description.

The sensible design is hybrid: routine, private, high-volume work on the device, genuinely hard questions to a frontier cloud model, with consent where personal data is involved. The same logic runs through last month's look at open-weight and API models.

It is not designed to be a chatbot for general world knowledge.
Apple Machine Learning Research, on its own on-device model

What to do this quarter

List your repetitive text jobs: call summaries, email drafts, enquiry tagging, pulling fields out of documents. These are the on-device candidates.
Check what your team already has. Recent desktop Chrome, Apple Intelligence iPhones on iOS 26 and supported Android phones may already carry a model your software can call.
Set 16 GB of RAM as the floor in your next hardware order, confirmed in writing.
Ask every AI vendor where processing happens, on the device, in their cloud or both, and test any language feature in Malayalam before you rely on it.

If you would like a second opinion on which of your workflows suit local AI, our AI and automation team can go through them with you.

Sources

Facts checked against these sources, as of 16 April 2026.

All insightsAI