Tech developers are shifting artificial intelligence from distant cloud data centers to personal devices like phones and laptops to achieve faster processing, better privacy, and lower costs. This on-device AI enables tasks that require quick responses and keeps sensitive data local. Experts predict significant advancements in the coming years as hardware and models improve.
The reliance on cloud-based AI, such as Anthropic's Claude, involves sending prompts to remote data centers, which can introduce delays of seconds—unacceptable for urgent tasks like alerting a user to an obstacle in their path. Privacy is another concern, as sensitive information like health or financial data travels through multiple untrusted systems. To address these issues, companies are increasingly processing AI on devices themselves, eliminating the need for internet connectivity and reducing costs by avoiding payments to data center operators.
This shift has been underway for years. As early as 2017, iPhones used on-device AI for face recognition via a neural engine. Modern implementations, like Apple's Apple Intelligence with about 3 billion parameters, handle specific tasks such as summarizing messages or visual recognition from screenshots. Google's Pixel phones employ the Gemini Nano model on the Tensor G5 chip to power features like Magic Cue, which pulls relevant information from emails and messages without manual searching.
Experts highlight the challenges and benefits. Mahadev Satyanarayanan, a Carnegie Mellon professor, likens ideal on-device computing to the human brain, noting that while nature evolved it over a billion years, humans aim to achieve similar efficiency in five to ten years through advanced hardware and specialized models. Vinesh Sukumar, head of generative AI at Qualcomm, points out system differences for compact devices like smartwatches, often requiring offloading to the cloud—but with safeguards like user permission and secure handling to protect data.
Apple's Private Cloud Compute exemplifies privacy measures: it processes offloaded data only on company servers, sends minimal information, and stores none. For developers, on-device AI cuts ongoing costs; Charlie Chapman of the Dark Noise app uses it to mix sounds without cloud fees, allowing scalability without financial risk.
Looking ahead, on-device AI excels in object classification within 100 milliseconds but still offloads for detection, segmentation, activity recognition, and tracking. Satyanarayanan anticipates exciting progress in five years, enabling features like trip alerts via computer vision or contextual reminders about conversations.
