LAI: Local AI for Mobile with RAG and Offline Inference

The Mission: AI for Everyone, Everywhere

In a world where artificial intelligence increasingly depends on massive cloud servers and expensive hardware, this project starts from a different question. What if AI could run entirely on the device in your pocket? What if a student in rural Kenya with a basic Android phone could hold the same quality of conversation that a developer in a capital city pays a subscription for?

LAI is the answer to that question. It is a cross platform mobile application designed specifically for rural areas with no internet connectivity, and for people who cannot afford bigger devices or daily data costs. Every model runs on the phone. Every conversation stays on the phone. The app asks the network for exactly one thing: the initial model download. After that, it is entirely self sufficient.

The motivation is simple and it deserves to be stated plainly. Intelligence is not a privilege. It is a human right. This application is a contribution toward making that right real for everyone, everywhere.

What Makes This Special

Real Local Intelligence

This is not another chat wrapper around a cloud API. LAI runs genuine artificial intelligence directly on the device, using two complementary models:

• Gemma 3B, Google’s efficient model, sized and tuned for personal devices.
• TinyLlama, the lightweight champion of constrained hardware, for devices with less memory.

Both models run as quantized GGUF files through an optimized native inference engine. On top of the raw language models sits a vector embedding system that gives the assistant semantic search over the content it has seen, so answers are grounded in context rather than generated from a vacuum.

Technical Excellence

The engineering goal was enterprise grade quality inside a package that runs on a two gigabyte phone. Every layer of the stack was chosen or built with that tension in mind:

• Real time streaming, so responses appear word by word, the way users expect from modern assistants.
• A storage layer built on BLOB columns in SQLite that stores vectors with roughly four times the memory efficiency of standard approaches.
• A modern Material 3 interface, designed to feel current and intuitive rather than technical.
• A native C++ inference backend integrated through JNI, because interpreted code cannot deliver token speed on constrained hardware.
• Smart retrieval over SQLite with vector embeddings, so the assistant finds relevant memory fast without loading the entire database.

Technical Specifications

System Requirements

• Android API 21 or newer (Android 5.0 and up).
• 2 GB RAM minimum, 4 GB recommended.
• 5 GB of free storage for models and app data.
• ARM64 CPU, which covers essentially every modern Android device.
• Internet only for the initial model download.

Performance Metrics

Measured on mid range hardware, the numbers justify the design choices:

• Initial model load takes 30 to 60 seconds per model.
• RAM usage while running stays between 2 and 4 GB.
• Response generation runs at 10 to 20 tokens per second, fast enough for natural conversation.
• Battery consumption is tuned for efficiency, with the native engine doing the heavy lifting.

Architecture Highlights

The stack reads like a careful exercise in knowing when to use which tool:

• Flutter for the cross platform UI layer.
• Drift for type safe SQLite database operations.
• Riverpod for reactive, testable state management.
• ONNX Runtime for hardware accelerated inference.
• llama.cpp for optimized local LLM execution.

Setup Instructions

Step 1: Download the App

Clone the repository and enter the project directory:

git clone https://github.com/AbduljabbarBXR/LAI.git
cd LAI

If you do not already have Flutter, install it from flutter.dev and add it to your system PATH.

Step 2: Setup Android Development

Install Android Studio, enable USB debugging on your device, and connect the device over USB.

Step 3: Download AI Models

The models total about 1.3 GB, so they are distributed separately. Two options exist:

Option A: Direct Download

• Gemma 3B (687 MB), available on Hugging Face.
• TinyLlama (637 MB), available on Hugging Face.

Option B: Git LFS

git lfs install
git lfs pull

For manual setup, create the models directory, download both GGUF files into it, and rename them to match the filenames the app expects.

mkdir -p assets/models/

Step 4: Build and Run

flutter pub get
flutter run

Step 5: First Setup

Open the app. The models load automatically on first launch. Once initialization completes, you can start chatting with your local assistant immediately.

How to Use

Basic Chat

• Send messages from the chat box and tap send.
• Watch responses stream in word by word.
• Switch between Gemma when you want depth and TinyLlama when you want speed.

Advanced Features

• The memory system carries conversation context across messages.
• Vector search retrieves semantically relevant knowledge for grounded answers.
• Offline knowledge works with no connection at all.
• Privacy is structural: all data stays on the device, always.

Troubleshooting

• Slow first launch is normal; models need time to initialize on device.
• On low RAM devices, switch to TinyLlama for a smoother experience.
• Models are large because quality local inference requires them; storage is the honest price of ownership.

Impact and Vision

Real World Applications

Education. Students in rural schools can get AI tutoring without a data bundle. Language learning benefits from local context, and research assistance no longer depends on a connection that may not exist.

Healthcare. Basic medical information queries work in remote areas. Symptom checking becomes possible where clinics are far away, and health workers gain a decision support tool that works when they need it.

Agriculture. Farmers get crop advice, pest identification, weather pattern analysis, and cached market price information, all without leaving the field.

Business. Small businesses can plan with the help of an assistant, automate parts of customer service, and process local language requests.

The Bigger Picture

Every device running this app becomes a node in a network of local intelligence. When villages share agricultural knowledge locally, when students help each other learn without expensive internet, when healthcare workers have instant access to medical AI, the pattern is the same: capability moved to the edge, owned by the people who use it.

This app is not just software. It is infrastructure for digital equality.

Development Details

Code Architecture

The codebase is organized around clear separation of concerns:

lib/
├── database/ # SQLite + vector storage
├── models/ # Data models and types
├── providers/ # State management (Riverpod)
├── screens/ # UI screens and navigation
├── services/ # Core AI and embedding services
└── widgets/ # Reusable UI components

local_llm_plugin/
├── android/ # Native Android implementation
│ ├── cpp/ # C++ inference engine
│ └── kotlin/ # Java/Kotlin interface
└── ios/ # iOS implementation

Key Technologies

• Flutter and Dart for the cross platform application.
• C++ for the high performance inference engine.
• JNI for the native bridge.
• SQLite for embedded storage.
• ONNX Runtime for hardware acceleration.
• Drift for type safe database access.

Innovation Highlights

• A streaming architecture that generates responses token by token.
• BLOB based vector storage delivering a four times memory efficiency improvement.
• MethodChannel communication for an efficient Flutter to native bridge.
• A modular plugin system that is extensible to new models.
• Progressive loading for optimized startup performance.

Future Roadmap

The roadmap is organized into three horizons, each building on the previous one.

Short Term (1 to 3 months)

• Voice input and output support.
• Support for additional language models.
• Image generation capabilities.
• A code execution sandbox.
• Advanced memory management.

Medium Term (3 to 6 months)

• Computer vision capabilities.
• Multi modal conversations.
• Federated learning features.
• Custom model training.
• Distributed AI networks.

Long Term (6 to 12 months)

• Edge computing optimization.
• Community model sharing.
• AI assistant specialization.
• Trust mechanisms for decentralized deployment.
• A complete offline app ecosystem.

Contributing

This project thrives on community contribution, and there is a role for everyone:

• Developers can add features, fix bugs, and optimize performance.
• Translators can make AI accessible in local languages.
• Testers can try the app on different devices and report issues.
• Documenters can improve guides and create tutorials.
• Advocates can share the mission in their communities.

Getting Started

1. Fork the repository.
2. Create a feature branch.
3. Make your changes.
4. Add tests if applicable.
5. Submit a pull request.

Development Guidelines

• Keep code clean and well documented.
• Prioritize performance on low end devices.
• Maintain the offline first architecture.
• Test on real hardware, not just emulators.

Project Statistics

• Development time: three months of intensive work.
• Lines of code: 12,727 across 233 files.
• Technologies used: over 15 frameworks and libraries.
• Platforms supported: Android, with iOS planned.
• Models integrated: two state of the art LLMs.
• Memory efficiency: four times improvement over standard approaches.

Acknowledgments

Special thanks to Google for the Gemma model, Meta for TinyLlama, Hugging Face for model hosting, the Flutter team for the framework, and the llama.cpp community for inference optimization.

The project was inspired by a simple belief: technology should serve humanity, not the other way around. Every line of code was written with the vision of a child in rural Kenya having the same AI assistant as someone in Silicon Valley.

License

MIT License. Free for personal and commercial use. Use it for any purpose, modify and distribute it freely, and commercial use is allowed. Attribution is appreciated but not required.

The only restriction is the honest one: do not use this technology for harmful purposes. It should empower, not exploit.