AI Pantry is an Android app, built as a Droidcon demo, of a fully on-device AI pipeline: you photograph your fridge and your pantry, the app recognises the ingredients and generates recipes that use only what you already have. Three AI engines cooperate, each on the job it was built for.

Inference is 100% on-device and the app does not even hold the INTERNET permission: photos never leave the phone. The models arrive through the install channel — Google Play AI packs for the Play Store release, or embedded in the APK for testing — and never through an in-app download.

• Kotlin
• Jetpack Compose
• Gemini Nano
• LiteRT
• Gemma
• CameraX
• Room
• Koin

The three AI engines

• Gemini Nano (AICore, via ML Kit GenAI) — ingredient detection on the devices that support it (Pixel 9/10, Galaxy S25/S26, …), with a multimodal photo+text prompt. It weighs zero MB in the APK: the model belongs to the system.
• MobileCLIP-S2 (LiteRT) — zero-shot detection on any device: the everywhere-fallback and the default engine.
• Gemma 4 E2B (LiteRT-LM) — recipe generation, the only genuinely generative task.

Demo flow

• Scan — photograph the fridge, then "Scan another" for the pantry; a photo from the gallery works too. The results sheet shows which engine did the work.
• The detected ingredients, with their confidence (anything below 22% is discarded), land in the local inventory stored in Room.
• What can I cook? — the LLM generates 5 recipes ranked by ingredient coverage, with time, difficulty, used and missing ingredients. Opening a recipe generates its step-by-step instructions and variations on demand.

Architecture

Every feature follows Clean Architecture + MVI (data / domain / presentation), with dependency injection through Koin:

• capture/ — CameraX and photo picker; AdaptiveIngredientDetector selects the engine at runtime, on every scan: checkStatus() on Gemini Nano, and a silent fallback to CLIP if Nano is unavailable or fails.
• inventory/ — Room database, ingredient inventory, model status banner, keyword emoji resolver.
• recipes/ — LlmRecipeGenerator with two-stage generation and silent retries, plus a tolerant RecipeJsonParser (small models occasionally emit malformed JSON).
• core/data/ai/ — LlmCatalog, ModelSource (AI packs or bundled assets), ModelRepository for automatic provisioning, LlmEngineHolder for the engine cache and the GPU/CPU policy.

Ingredient recognition without a training set

The MobileCLIP-S2 image encoder (~140 MB, LiteRT) embeds multi-scale crops of the photo — a grid at 50% and 33% of the short side, on a pool of interpreters running in parallel — and compares them by cosine similarity against the text embeddings of 864 ingredients + 6 "distractor" labels, precomputed offline.

The result is that the "classifier" is a text file: one label per line, with aliases and distractor labels that absorb the food-free crops. Adding an ingredient means adding a line and re-running the script that recomputes the embeddings — no training, no dataset, no retraining cycle.

Delivering a 2.5 GB model

Gemma 4 E2B ships in the .litertlm format at ~2.5 GB, with mixed 2/4/8-bit quantisation. Two delivery channels, both without any in-app HTTP download:

• Play for On-device AI — for the Play Store release: 3 fast-follow AI packs of ~0.85 GB each (Play caps a pack at 1.5 GB), reassembled on first launch and then removed.
• Embedded in the APK — for testing: 3 chunks in the assets, since AGP does not package assets over 2 GB. A ~2.9 GB APK to hand out to testers, with zero action required from them.

After a few seconds of "Preparing Gemma 4 E2B…" on first launch, the app is ready and offline forever.

GPU/CPU resilience

The LiteRT-LM runtime is GPU-first, and a broken GPU shows up in two very different ways: a hang in a native call that cannot be interrupted, or an immediate exception on the first inference (for example when there is no OpenCL, where engine initialisation succeeds anyway). LlmEngineHolder catches both — a watchdog with a timeout on a detached job for the hangs, an exception catch for the rest — and marks the GPU as broken persistently for that model, so the following launches go straight to multi-threaded CPU without paying for the attempt again. A micro-inference probe at warm-up triggers the fallback at startup, so the user's first generation never waits for it.

Stack

Kotlin 2.2 · Jetpack Compose (Material 3) · CameraX · ML Kit GenAI Prompt API (Gemini Nano / AICore) · LiteRT (MobileCLIP-S2) · LiteRT-LM (Gemma 4 E2B) · Play AI Delivery · Room · Koin · Navigation Compose · kotlinx-serialization

AI Pantry is my playground for on-device generative AI on Android: model delivery at the gigabyte scale, runtime engine selection, hardware fallbacks and a privacy guarantee enforced by the manifest itself rather than by a privacy policy.

July - September 2026.