Mobile silicon now ships with neural processing units fast enough to run useful models locally, and Apple's Core ML and Google's ML Kit have matured around that hardware. In 2026, on-device AI stopped being a demo reel and became a practical architecture choice with clear win conditions.
Hardware caught up
Dedicated neural engines now handle real workloads: image segmentation, transcription, and small language models at usable speeds. The constraint has shifted from raw capability to model size, memory pressure, and battery impact.
Framework choice by task
Core ML on iOS and ML Kit on Android cover most standard tasks with hardware acceleration. For custom models, both platforms support importing common formats with conversion steps that require testing against real devices, not just simulators.
Budget battery and thermals
Sustained on-device inference heats phones and drains batteries in ways users notice immediately. Throttle background inference, batch requests, and measure power draw on mid-range hardware before committing to an always-on AI feature.
What on-device does best
The strongest use cases are latency-critical, privacy-sensitive and repeated: text completion while typing, live camera features, transcription, semantic photo search and offline assistants. Running locally eliminates network round trips and per-call API costs, and keeps sensitive data on the device by construction — an increasingly compelling property under privacy regulation. The capability ceiling is real: compact models handle narrow tasks well, not open-ended reasoning.
The engineering checklist
Success depends on the boring parts: quantized models sized for low-memory devices, warm-up strategy so the first inference does not jank the UI, background scheduling that respects battery policy, and graceful degradation to cloud endpoints on unsupported hardware. Model updates ride app releases or on-demand delivery, which means prompt-and-model versioning must be coordinated. Instrument inference latency and failure rates the same way you monitor cloud calls — silent on-device failures are invisible otherwise.
Key takeaways
- On-device wins for latency, privacy and repeated narrow tasks.
- Quantize, warm up, and degrade to cloud on weak hardware.
- Version and ship models deliberately; monitor on-device inference.
Final thoughts
The NPU era makes the device part of the inference fleet, and teams that architect hybrid routing — local for the fast and private, cloud for the heavy — will ship AI features at unit economics their cloud-only competitors cannot match.