Building Offline AI in Mobile Apps

For years, AI in mobile apps meant "call an API." You typed something, sent it to the cloud, got a response.

That's changing. Apple Intelligence, Google Gemini Nano, Llama 2 on-device—models are getting fast and small enough to run locally.

This changes everything about how you build AI-powered mobile features.

Why Offline AI Matters

Reliability first. No internet? Your feature still works. Users expect their phone to work offline. AI features that require network connectivity violate that expectation.

Privacy second. Data never leaves the device. You're not sending user data to cloud servers. You're not waiting for processing. It's all local.

Latency third. Inference is instantaneous. No "waiting for the API." Type a character and get suggestions immediately.

Speed to feature fourth. No need to deploy servers, manage API keys, worry about rate limits. Model runs on the phone. You ship it.

The cost is lower latency (models are smaller, less accurate) and reduced features (you can't run massive models). But for many use cases, the tradeoff is favorable.

The Pattern in 2025

The winning approach: hybrid. Large model in the cloud for complex tasks. Small model on-device for quick feedback.

Example: autocomplete for forms. On-device model runs as the user types. Suggests based on their history and common patterns. Fast. Works offline. Occasionally wrong.

If the user wants really good suggestions (full AI reasoning, not just pattern matching), they tap "AI assist." That hits the cloud model. Slower. Better results. Requires internet.

Both work together. Users get instant feedback from on-device. They get advanced features from cloud models if they want them.

Building On-Device

The technical landscape is getting better.

iOS: Core ML for on-device inference. Apple's been shipping Core ML models for a while. Now with Apple Intelligence (iOS 18+), there are specific APIs for on-device LLMs. Limited models right now. But expanding.

Android: TensorFlow Lite. Models run on-device. Google's also shipping Gemini Nano on some devices. It's not universally available yet, but adoption is growing.

Cross-platform: ONNX Runtime. Export models to ONNX format, run on iOS and Android. Less optimized than platform-specific APIs, but available today.

The models themselves:

• Phi (Microsoft's small model): 3 billion parameters. Runs fast on mobile. Capable enough for basic tasks.
• Mistral 7B: Larger, but still feasible. Needs more RAM and processing power.
• Gemini Nano (Google): Purpose-built for mobile. Limited release right now. Worth watching.
• Llama 2 variants: Different sizes. The 7B version is feasible on high-end phones.

In 2025, I'm recommending Phi for most use cases. It's small, fast, and surprisingly capable for classification and simple generation tasks.

The Constraints

There's no escaping physics.

Model size. A usable on-device model is 1-4 GB. Your app size is limited. Not every app can carry a 2GB model.

Performance. Inference takes time. A 1B parameter model generating 100 tokens might take 10-30 seconds on a phone. That's instant compared to cloud, but not instant compared to typing a response.

Accuracy. Smaller models make more mistakes. The larger the model, the better the accuracy. But there's a point where you need cloud.

Battery. Inference is compute-intensive. Running it continuously drains battery. This limits when you use on-device models.

Device support. Not every phone can run a 2GB model with reasonable latency. Older devices struggle. You need to handle graceful degradation.

What Actually Works

The products I'm seeing launch with offline AI successfully follow patterns:

Classification tasks. "Is this message spam?" "What category is this transaction?" "Positive or negative sentiment?" These work great on-device because you're not generating text, just picking a category.

Tagging and labeling. Given some text, apply tags. Reading a customer feedback comment and tagging what it's about. Works well on-device.

Recommendations. Based on user history and current context, suggest next actions. These work if you're recommending from a fixed set of options.

Text generation for simple cases. Finishing a sentence. Expanding a note. These work if you're okay with occasional gibberish.

What doesn't work:

• Complex reasoning tasks ("analyze this business situation")
• Factual correctness requirements ("tell me about this person")
• Long-form generation ("write an article")
• Any task where being wrong is costly

The Architecture

A typical on-device AI architecture looks like:

User Input
↓
[Local Model] → 99% of requests handled locally
    ↓ (when needed, send to cloud for better results)
[Cloud API] → 1% of requests that need advanced capability
↓
Results shown in UI

When you need local-only:

• Model ships with app (or downloads on first use)
• Inference runs locally
• Results cached and used for history

When you want cloud fallback:

• Check network connectivity
• User explicitly requests ("AI assist" button)
• Background task for non-urgent processing

The Launch Strategy

Don't ship "ask our on-device AI anything." That won't work because the model is too limited.

Ship specific features:

• Autocomplete for forms (on-device)
• Quick categorization of data (on-device)
• Sentiment analysis of comments (on-device)
• "Get more detail" with cloud model (fallback)

A task management app that uses on-device model to tag tasks as "urgent/normal" works. That's a binary classification, perfect for small models. Same app tries to use it for "write a detailed plan," it fails.

The Future

In 2026, I expect on-device AI to become standard in mobile apps. Not because every app needs it, but because it's the easy default.

Apple Intelligence and similar capabilities (from Google, Samsung, others) will push models into more devices. Developers will have standard APIs instead of custom integrations.

The gap between "cloud AI" and "device AI" will narrow as models get better without getting much bigger.

The products that win will be the ones that use on-device models for what they're good at and seamlessly upgrade to cloud models when needed.

That's the frontier. Not choosing on-device or cloud. Building so both work together, each used where it's best.