Getting Started

Run Gemma and other on-device LLMs in your Flutter app — Android, iOS, Web, and Desktop.

Bring the power of Google's lightweight Gemma language models and other on-device LLMs directly to your Flutter applications. With flutter_gemma you can seamlessly incorporate advanced AI capabilities into your apps — all without relying on external servers.

Gemma is a family of lightweight, state-of-the-art open models built from the same research and technology used to create the Gemini models. The plugin supports not only Gemma, but also Qwen, DeepSeek, Phi, FastVLM, SmolLM and more — see Models for the full list.

Features#

Local Execution: Run Gemma and other LLMs (Qwen, DeepSeek, Phi, FastVLM, SmolLM, …) directly on user devices for enhanced privacy and offline functionality.
Platform Support: Compatible with iOS, Android, Web, macOS, Windows, and Linux.
Desktop Support: Native desktop apps with GPU acceleration via LiteRT-LM, called directly from Dart through dart:ffi — no JVM/JRE bundling. See Desktop Support.
Built-in / System AI: Use the OS's own on-device model, no download — Gemini Nano (Android + Chrome) + Apple Foundation Models (iOS/macOS). See Built-in AI.
Pluggable Engines: Opt-in engine packages — LiteRT-LM, MediaPipe, ONNX Runtime, built-in OS AI — registered via FlutterGemma.initialize(...). See Packages.
Multimodal Support: Text + image input with Gemma 4, Gemma3n, FastVLM, Qwen2-VL, SmolVLM2, and LLaVA-OneVision vision models. See Multimodal.
Audio Input: Record and send audio messages with Gemma 4 and Gemma3n models (Android, iOS device, Desktop).
Function Calling: Let models call external functions and integrate with other services. See Function Calling.
Agent Skills: Give the model a catalog of SKILL.md skills it picks and runs itself — text, JS, native intents, or MCP tools. See Agent Skills.
Speech & Voice (STT + TTS + voice loop): Transcribe audio, synthesize speech, and run a full on-device speech-to-speech voice loop (VoiceSession) via flutter_gemma_speech — STT (moonshine/Whisper/Parakeet) + TTS (Matcha/Qwen3/Inflect) today. See Speech.
Thinking Mode: View the reasoning process of DeepSeek, Gemma 4, and Qwen3 models. See Thinking Mode.
Stop Generation: Cancel text generation mid-process on Android, iOS, Web, and Desktop.
Backend Switching: Choose between CPU and GPU backends for each model individually.
LoRA Support: Efficient fine-tuning and integration of LoRA (Low-Rank Adaptation) weights.
Enhanced Downloads: Smart retry logic with exponential backoff and automatic restart of interrupted downloads.
Android Foreground Service: opt in with foreground: true for large downloads to bypass the 9-minute timeout.
Text Embeddings & RAG: Generate vector embeddings (EmbeddingGemma, Gecko) and run on-device RAG. See Embeddings & RAG.
Web Persistent Caching: Models persist across browser restarts using the Cache API (Web only).

What's new in 1.8#

Package Skills — flutter_gemma ships agent skills for coding assistants; dart run skills@ get --all installs them. See Package Skills.
Whisper output language — getActiveStt(language:) sets a default, transcribe(…, language:) overrides one call. Breaking for custom SpeechRecognizer implementations. See Speech.
FlutterGemma.rag.flush() — persists what you indexed; required on qdrant-edge. Custom VectorStoreRepository implementations must declare it. See Embeddings & RAG.

What's new in 1.7#

Install from Hugging Face — installModel(…).fromHuggingFace(repo) reads the repo's manifest and installs the right variant for the device; resolveHuggingFace inspects it first. See Models → Hugging Face.

What's new in 1.6#

flutter_gemma_onnx — new opt-in ONNX Runtime engine: text generation via ORT-GenAI (OnnxEngine) + embeddings via plain ONNX Runtime (OnnxEmbeddingBackend), both dart:ffi. Device-verified on macOS, Linux, Windows, Android, and iOS (arm64) — plus Web, via Transformers.js (generation) and onnxruntime-web (embeddings). See Packages.
flutter_gemma_builtin_ai on Web — Gemini Nano through Chrome's Prompt API, next to Gemini Nano on Android and Apple Foundation Models on iOS/macOS. See Built-in AI.
BREAKING (flutter_gemma_embeddings 2.0.0): the embedder is now runtime-agnostic — LiteRtEmbeddingBackend moved to flutter_gemma_litertlm (1.5.0). See Migration.

What's new in 1.5#

genai_primitives support — drive an on-device chat with the Flutter team's standard ChatMessage types via package:flutter_gemma/genai.dart (sendMessage/generateContent + streams, covering text, vision, audio, thinking, and tool calls). See genai_primitives.
FlutterGemma is the one canonical entry point. RAG moved onto the FlutterGemma.rag.* namespace (initialize/addDocument/addDocumentWithEmbedding/searchSimilar/removeDocument/stats/clear), and the facade gained model introspection (activeModelSpec/activeEmbedderSpec/activeSttSpec/activeTtsSpec, getModelPath), storage helpers (getStorageInfo/getOrphanedFiles/cleanupStorage/performCleanup), and per-modality uninstallers (uninstallEmbedder/uninstallStt/uninstallTts). FlutterGemmaPlugin.instance is now a documented low-level SPI tier — app code shouldn't need it.

What's new in 1.4#

flutter_gemma_speech — new opt-in on-device speech package: STT (moonshine) + TTS (Matcha) + a full STT → LLM → TTS voice loop (VoiceSession). See Speech.

What's new in 1.3#

ModelFileType.builtIn for OS system models (Gemini Nano / Apple Foundation Models), plus download-reliability fixes. See Built-in AI.

What's new in 1.2#

flutter_gemma_agent — new opt-in agent-skills package (the skillExecutors: registration seam), plus the Android Mali GPU fix. See Agent Skills.

What's new in 1.1#

Declared-column RAG filters — FilterSchema/FilterField + configure, and filterSchema: on initialize; enableHnsw is deprecated. See Embeddings & RAG.

What's new in 1.0#

Modular package split — the monolith is now a small core (flutter_gemma) plus opt-in packages, so your app ships only the native weight it uses: flutter_gemma_litertlm (.litertlm), flutter_gemma_mediapipe (.task/.bin), flutter_gemma_embeddings, flutter_gemma_rag_qdrant, flutter_gemma_rag_sqlite. See Packages.
New FlutterGemma.initialize(...) registration — pass inferenceEngines, embeddingBackends, vectorStore for the packages you added. See Installation.
Every model / session / chat / embedding / RAG API is unchanged — migrating is just adding packages + the initialize call. See Migration.
Two on-device vector stores — flutter_gemma_rag_qdrant (qdrant-edge, fastest on native) and flutter_gemma_rag_sqlite (portable across all six platforms, including Web). Since rag_sqlite 1.1.0 the SQLite store runs exact in-SQLite KNN via the sqlite-vec/vec0 extension, replacing its Dart brute-force + HNSW search.

See the CHANGELOG for the full release history.

Quick Start#

Complete the [platform setup](/docs/installation) before running this code.

1. Install a Model (One Time)#

import 'package:flutter_gemma/flutter_gemma.dart';

// Install model. The URL below uses the .litertlm variant so the same code
// works on Desktop (Windows/macOS/Linux) and mobile/web. For web only, the
// `.task` / `-web.task` variants of the same model also work.
await FlutterGemma.installModel(
  modelType: ModelType.gemmaIt,
  fileType: ModelFileType.litertlm,
).fromNetwork(
  'https://huggingface.co/litert-community/Gemma3-1B-IT/resolve/main/Gemma3-1B-IT_multi-prefill-seq_q4_ekv4096.litertlm',
  token: 'your_hf_token',
).withProgress((progress) {
  print('Downloading: $progress%');
}).install();

**Mobile/Web shortcut:** if you don't target Desktop, you can substitute the URL with the `.task` build of the same model. Desktop targets need the `.litertlm` build — `.task` and `.bin` are MediaPipe-only.

2. Create and Use a Model (Multiple Times)#

// Create model with specific configuration
final model = await FlutterGemma.getActiveModel(
  maxTokens: 2048,
  preferredBackend: PreferredBackend.gpu,
);

// Use model
final chat = await model.createChat();
await chat.addQueryChunk(Message.text(
  text: 'Explain quantum computing',
  isUser: true,
));
final response = await chat.generateChatResponse();

// Cleanup
await model.close();

`Message.isUser` defaults to `false`. Always pass `isUser: true` for user messages, or the model returns an empty response. Always `close()` sessions and models when you're done with them.

System Instructions#

Control model behavior with a system-level instruction:

final chat = await model.createChat(
  systemInstruction: 'You are a concise assistant. Always respond in bullet points.',
);

Platform support:

.litertlm on every platform, native and web: passed as a real system turn.
MediaPipe .task (Android, iOS, web): prepended to the first user message as a fallback.

Changing a runtime parameter reloads the model#

getActiveModel caches one model per active spec, but it compares every runtime parameter — maxTokens, preferredBackend, supportImage, maxConcurrentSessions and the rest. A call that differs in any of them closes the cached model and builds a new one, logging which parameter forced the rebuild:

final quick = await FlutterGemma.getActiveModel(maxTokens: 512);
final deep  = await FlutterGemma.getActiveModel(maxTokens: 4096);
// deep is a NEW model, and `quick` has been closed — using it now throws a
// StateError ('Model is closed…'). Drop the old handle.

So don't hold a handle across a parameter change, and don't vary the arguments on a hot path: each change costs a full weight reload. To serve several conversations from one loaded model, keep the arguments identical and open several sessions instead (below). Closing it yourself first is equivalent:

await quick.close();
final deep = await FlutterGemma.getActiveModel(maxTokens: 4096);

To serve several dialogues from one loaded model, do not reload it — open several sessions instead (next section).

On web there is no reuse at all: every getActiveModel call closes the cached model and builds a new one, identical arguments or not. Get the model once and keep the handle.

On native `.litertlm` (except the NPU) a `maxTokens` below 1024 is clamped up to 1024, so the 512 above would not have been honored in any case. See [Troubleshooting](/docs/troubleshooting).

Concurrent Sessions (`openSession`)#

A single loaded model can serve several independent dialogues at once. openSession() returns a session with its own conversation history, detached from the legacy model.session singleton; openChat() is the same for the higher-level chat API.

The model weights (the big, expensive part — hundreds of MB to several GB) are loaded once and shared across every session; each session only adds its own lightweight conversation context.

final model = await FlutterGemma.getActiveModel(maxTokens: 1024);

final chatA = await model.openChat(); // independent context A
final chatB = await model.openChat(); // independent context B

await chatA.addQueryChunk(Message(text: 'My name is Alice.', isUser: true));
await chatA.generateChatResponse();

await chatB.addQueryChunk(Message(text: 'My name is Bob.', isUser: true));
await chatB.generateChatResponse();

// Each remembers only its own context.
await chatA.addQueryChunk(Message(text: 'What is my name?', isUser: true));
print(await chatA.generateChatResponse()); // "Alice"

model.sessions;               // all live sessions (legacy + open)
await chatA.session.close();  // closing one leaves the others usable

**Concurrent contexts, serialized inference.** The sessions are logically independent, but **only one session generates at a time** — calling `generateResponse()` on a second session while another is still running blocks until the first finishes. Generation is *not* parallel. This is intentional: parallel on-device inference would contend for the accelerator and risk OOM.

Memory: each open session holds its own context (~100–500 MB depending on model + maxTokens). On phones with large models (Gemma 4 E2B+), several concurrent sessions can OOM. Cap the count with maxConcurrentSessions: on getActiveModel(...) — openSession() throws StateError past the cap.

If you only ever have one conversation at a time, stick with the simpler createSession() / createChat() singleton API — you don't need this.

Managing Installed Models#

Uninstalling#

For the active embedder / STT / TTS model, one call deletes every file it owns (e.g. an embedder's model and tokenizer) and clears its persisted identity so it isn't auto-restored on the next launch:

await FlutterGemma.uninstallEmbedder(); // deletes model + tokenizer, clears identity
await FlutterGemma.uninstallStt();
await FlutterGemma.uninstallTts();

To delete an inference model, or any model by filename, use uninstallModel() and clear the active inference identity if you removed the active one:

// Free in-memory handles first if the model is loaded.
await model.close();

// Delete the files + metadata.
await FlutterGemma.uninstallModel('Gemma3-1B-IT_multi-prefill-seq_q4_ekv4096.litertlm');

// Clear the persisted "active model" identity so it isn't auto-restored.
await FlutterGemma.clearActiveInferenceIdentity();

`clearActiveInferenceIdentity()` wipes both the in-memory active spec and the persisted preference. Pair it with `uninstallModel()` only when the deleted model was the active one. The `uninstallEmbedder/Stt/Tts` helpers already clear the identity for you.

Inspecting what's active#

Cheap, synchronous getters return the active model's identity (its spec) without loading the engine — use these to render UI, not getActiveModel() (which loads the runtime). Each is typed to its modality:

final InferenceModelSpec? model = FlutterGemma.activeModelSpec;
final EmbeddingModelSpec? embedder = FlutterGemma.activeEmbedderSpec;
final SttModelSpec? stt = FlutterGemma.activeSttSpec;
final TtsModelSpec? tts = FlutterGemma.activeTtsSpec;

// Absolute on-device path of an installed file (a URL/OPFS handle on web):
final path = await FlutterGemma.getModelPath('Gemma3-1B-IT_..._ekv4096.litertlm');

Storage & cleanup#

Inspect on-device usage and reclaim space left by interrupted downloads:

final info = await FlutterGemma.getStorageInfo();      // StorageStats: files + bytes
final orphans = await FlutterGemma.getOrphanedFiles();  // fragments with no metadata
final removed = await FlutterGemma.cleanupStorage();    // delete orphans, returns count
await FlutterGemma.performCleanup();                    // cancel stale tasks + sweep

Message Types#

// Text only
final textMessage = Message.text(text: "Hello!", isUser: true);

// Text + Image
final multimodalMessage = Message.withImages(
  text: "What's in this image?",
  imageBytes: [imageBytes],
  isUser: true,
);

// Image only
final imageMessage = Message.imagesOnly(imageBytes: [imageBytes], isUser: true);

// Tool response (for function calling)
final toolMessage = Message.toolResponse(
  toolName: 'change_background_color',
  response: {'status': 'success', 'color': 'blue'},
);

// System information message
final systemMessage = Message.systemInfo(text: "Function completed successfully");

// Thinking content (for DeepSeek models)
final thinkingMessage = Message.thinking(text: "Let me analyze this problem...");

// Check if a message contains an image
if (message.hasImage) {
  print('This message contains an image');
}

Response Types#

The model can return different types of responses depending on its capabilities:

chat.generateChatResponseAsync().listen((response) {
  if (response is TextResponse) {
    // Regular text token from the model
    print('Text token: ${response.token}');
  } else if (response is FunctionCallResponse) {
    // Model wants to call a function
    print('Function: ${response.name}');
    print('Arguments: ${response.args}');
    _handleFunctionCall(response);
  } else if (response is ThinkingResponse) {
    // Model's reasoning process
    print('Thinking: ${response.content}');
    _showThinkingBubble(response.content);
  }
});

TextResponse — contains a text token (response.token) for regular model output.
FunctionCallResponse — contains function name (response.name) and arguments (response.args). See Function Calling.
ThinkingResponse — contains the model's reasoning process (response.content). See Thinking Mode.

Next Steps#

Installation — per-platform setup and engine registration.
Models — supported models, file formats, and capabilities.
Migration (0.x → 1.0) — upgrade from the monolith.

Writing this with a coding assistant? dart run skills@ get --all installs flutter-gemma-inference, the skill that teaches it engines, installing a model, sessions and chats, streaming, and the platform setup for all six targets.

Getting Started

Features#

What's new in 1.8#

What's new in 1.7#

What's new in 1.6#

What's new in 1.5#

What's new in 1.4#

What's new in 1.3#

What's new in 1.2#

What's new in 1.1#

What's new in 1.0#

Quick Start#

1. Install a Model (One Time)#

2. Create and Use a Model (Multiple Times)#

System Instructions#

Changing a runtime parameter reloads the model#

Concurrent Sessions (openSession)#

Managing Installed Models#

Uninstalling#

Inspecting what's active#

Storage & cleanup#

Message Types#

Response Types#

Next Steps#

Concurrent Sessions (`openSession`)#