On-Device AI: What Apple and Google Are Really Building

In April, the rumor mill shifted. Apple and Google have both been working on on-device AI systems for months, but it became real this month. Gemini Nano is shipping on Android phones. Apple's announced something coming in iOS 18 (the names are still vague, but everyone knows it's coming).

Most people see this as a feature: "Oh, AI in your phone, without network latency."

That's not what's happening. What's happening is more fundamental.

The Infrastructure Shift Nobody's Talking About

For the past three years, on-device inference was technically possible but not practical. You could run a small language model on a phone. It would take 10 seconds to generate a response. The model quality was mediocre. The battery drain was catastrophic.

So everyone went cloud: send the request to an LLM API, get the response back. Simple, fast, high quality. But you're sending every request through someone else's servers. Every request is logged. Every request gives the cloud provider data about what you're doing.

What's changed: model compression and hardware efficiency improved at the same time cloud costs got worse.

Gemini Nano is about 2 billion parameters. It's tiny compared to Claude 3 Opus (which is hundreds of billions). But on specific tasks—classification, local Q&A, content generation with existing context—it's very capable.

And here's the key: it runs on your phone with <5 second latency and minimal battery impact.

That changes the economics of mobile app development entirely.

What This Means for App Development

I spent April rebuilding one of our internal iOS tools to use on-device inference instead of calling our backend API.

Before: User taps a button → app makes API call → cloud processes → response comes back → app updates.

After: User taps a button → app runs model locally → instant response → app updates.

The difference in feel is enormous. The app doesn't have to wait for network. It doesn't have to wait for cloud latency. The user gets instant feedback.

But here's what actually matters for developers:

First, you stop needing servers for certain tasks. We had a service running that cost us about $2,000 a month. It was just running inference on user input. Now that runs on the phone. We killed the service. Saved $24,000 a year just by using on-device models.

Second, privacy becomes a differentiator instead of a compliance checkbox. Your app doesn't need to send user data anywhere. Classification happens locally. Context understanding happens locally. You can show users: "This runs entirely on your device. We never see your data."

Third, you can build features that require frequent inference without cost concerns. Want to auto-complete emails? On-device. Want real-time sentiment analysis of messages? On-device. Want to classify images as you take them? On-device.

The cloud APIs still exist for heavy lifting. But the bread-and-butter features that used to require API calls? They're moving local.

The Enterprise Impact

For Arvension, this opens a door.

We've been building mobile tools for ERP clients. One of the patterns we keep running into: the app needs to classify transactions, tag expenses, understand document context. This requires inference.

In the cloud model, you're either calling Anthropic's API or your own backend. Either way, transaction data is leaving the phone.

With on-device AI, we can do this locally. A field rep photographs a receipt, the phone instantly classifies it, extracts the data, the transaction stays encrypted locally until it syncs with the ERP.

For regulated industries, this is huge. You can comply with data residency requirements. You can guarantee that customer financial data never leaves their device.

The Catch: Model Quality

I need to be honest about the tradeoff. Gemini Nano and the models Apple's shipping are capable, but they're not Claude 3 Opus. They won't win a reasoning competition.

If you need nuanced judgment, complex logic, or handling rare edge cases, you still need cloud.

But for 70% of the tasks I was using cloud APIs for: simple classification, entity extraction, Q&A against local context, intent understanding—the on-device models are more than capable. And they're faster and cheaper.

What I'm Building

We're planning an iOS app for one of our manufacturing clients. The app lets field workers check inventory, log issues, request parts. Previously, the app would call back to the ERP for every smart decision (should we suggest this part? is this quantity unusual?).

Now: the app will run a small model locally. It will answer simple questions from context already on the phone. The cloud API is only for operations that change state.

The result: the app feels responsive, doesn't leak data unnecessarily, and costs 80% less to run because we've reduced API calls by 75%.

By the end of the year, when iOS 18 ships with on-device capabilities, we'll ship an update that goes even further. More inference local. More features that don't require cloud. More privacy.

The Bigger Picture

This shift—from cloud-first to hybrid local-cloud—is how enterprise AI gets practical at scale.

You can't run Claude 3 Opus on every phone. But you can run efficient models for specific tasks. You can keep the heavy reasoning in the cloud for complex decisions. And you can keep user data local for everything that doesn't need cloud processing.

That architecture is coming to every serious mobile app by end of 2024. The companies that move first are the ones who will own the efficiency advantage.