A recipe assistant that answers from your documents, with the network switched off at every step. It embeds a corpus with EmbeddingGemma, stores the vectors in sqlite-vec, retrieves the relevant ones for each question, and hands them to the on-device model as context.
By the end you will have an app that:
The code is short. What takes the hour is the handful of decisions RAG asks you to make, and each step here is built around one of them:
seq256 is notList — three answers, only one of which is speedflutter_gemma_rag_sqlite 1.4.0 requires sqlite3 3.6.0, whose build toolchain wants meta ^1.19.0, and every Flutter 3.44.x pins meta to 1.18.0 exactly. On 3.44 the Step 3 app will not resolve--dart-define=HF_TOKEN=hf_... covers bothWatch out: This codelab continues Getting Started with On-Device LLMs. Its finished app is this one's starter, byte for byte — a CI check enforces it. If you have not done that codelab, step_01_starter still runs on its own; you will just be meeting the download-and-chat code for the first time.
Every step of this codelab exists as a complete, runnable app, so you can join at any point or check your work against the next one.
git clone --depth 1 https://github.com/DenisovAV/flutter_gemma.git
cd flutter_gemma/codelabs/on-device-rag-flutter-gemma
ls
step_01_starter/ the chat app you start from
step_02_embed/ after Step 2 — the corpus, embedded
step_03_store/ after Step 3 — a real vector store
step_04_filters/ after Step 4 — filtered search
step_05_grounded/ after Step 5 — answers with sources
complete/ after Step 6 — the finished app
Open step_01_starter and run it.
cd step_01_starter
flutter run --dart-define=HF_TOKEN=hf_your_token
You get the finished app from Getting Started: it downloads a model once, then chats with it offline. Nothing in it knows anything about your documents — ask it about a recipe and you get whatever was in the weights.
That is the gap this codelab closes. RAG — retrieval-augmented generation — is three moving parts, and the model is only the last one:
Steps 2 to 5 build exactly those three, in order.
lib/recipes.dart is twelve recipes, in plain Dart — no asset bundle, no network. A corpus you can read in one screen beats one you have to go fetch.
class Recipe {
final String id;
final String title;
final String text;
final String cuisine;
final int minutes;
final bool vegetarian;
}
The three fields after text are not decoration. Step 4 turns them into filters, and between them they cover every condition the API has: a string, a number and a bool.
Good to know: Only text is embedded. The vector is built from that string and nothing else, so anything you want the search to match on has to be in it — a title that lives only in a field beside it is invisible to retrieval.
flutter pub add flutter_gemma_embeddings
One package, and it is not an engine. flutter_gemma_litertlm — already in the app — supplies LiteRtEmbeddingBackend, the thing that runs the forward pass. flutter_gemma_embeddings supplies the tokenizers. They are separate on purpose, and Step 2's whole registration hinges on why.
In lib/main.dart:
await FlutterGemma.initialize(
inferenceEngines: [LiteRtLmEngine()],
embeddingBackends: [LiteRtEmbeddingBackend()],
embeddingTokenizers: [GemmaEmbeddingTokenizers()],
// ...
);
Two lines, two packages, and the reason matters.
Which tokenizer an embedding model needs is a property of the model, not of the engine that runs it: EmbeddingGemma wants SentencePiece whether LiteRT or ONNX Runtime executes it, and a BERT-family model wants WordPiece under either. So since flutter_gemma 1.9.0 the backend no longer carries one, and the app says which families it has.
Watch out: Leave embeddingTokenizers out and the first embedding throws a StateError naming the package to add. That is the design working: the alternative — falling back to one family and tokenizing with the wrong convention — returns vectors that are quietly the wrong point in the embedding space, and no test downstream can tell them from good ones.
static const embeddingGemma = EmbedderChoice(
modelUrl: '.../embeddinggemma-300M_seq256_mixed-precision.tflite',
tokenizerUrl: '.../sentencepiece.model',
sizeLabel: '0.2 GB',
requiresToken: true,
);
Two files, and both are required: the .tflite holds the weights, sentencepiece.model turns text into the ids those weights expect. There is no tokenizer baked into the graph — hand over only the first and the install fails.
Good to know: seq256 is the sequence length in tokens, not the embedding dimension. The vectors are 768 long either way; 256 is how much text fits into one forward pass before it is truncated. Every recipe here is comfortably shorter.
await FlutterGemma.installEmbedder()
.modelFromNetwork(e.modelUrl, token: hfToken)
.tokenizerFromNetwork(e.tokenizerUrl, token: hfToken)
.withModelProgress((p) => setState(() => _installProgress = p / 100))
.install();
final embedder = await FlutterGemma.getActiveEmbedder();
final vectors = await embedder.generateEmbeddings(
kRecipes.map((r) => r.text).toList(),
taskType: TaskType.retrievalDocument,
);
install() is idempotent — the bytes are fetched once, and the second run of that button skips straight past the download. Progress is a link in the builder chain, not an argument to install().
One call for the whole corpus rather than a loop: the worker isolate is set up once and the model stays resident between texts.
taskType is the part worth slowing down for. EmbeddingGemma was trained with a different prefix for documents than for queries, and the enum is where that lives:
TaskType.retrievalDocument -> 'title: none | text: '
TaskType.retrievalQuery -> 'task: search result | query: '
Index your corpus with the query prefix and nothing errors. The vectors simply land slightly off, every search afterwards is a little worse, and no exception will ever point at it.
Watch out: This is the one place in the codelab you have to get right by hand. From Step 3 on, searchSimilar(query:) embeds the query for you and uses retrievalQuery by default — so the two halves stay matched as long as you index with retrievalDocument.
lib/embed_page.dart is a new screen — the full file is in step_02_embed, and the parts that matter are above. Two small changes put it in reach.
The Hugging Face token was a private constant in main.dart. Both the model download and the embedder install need it now, and they live on different pages, so it moves to lib/model.dart where both already import from:
// lib/model.dart
const hfToken = String.fromEnvironment('HF_TOKEN');
Then give the chat screen a way in — an action in its app bar:
// lib/chat_page.dart
import 'embed_page.dart';
// ...in the AppBar's actions, before the delete button:
IconButton(
tooltip: 'Recipes',
onPressed: () => Navigator.of(context).push(
MaterialPageRoute<void>(
builder: (_) => const EmbedPage(hfToken: hfToken),
),
),
icon: const Icon(Icons.restaurant_menu),
),
Skip this unless you are running in Chrome — but do not skip it and then run in Chrome, because the Embed button is the first thing that fails.
Embedding in a browser runs through LiteRT.js, which ships as four files in flutter_gemma_litertlm/web/. Copy them into your own web/, the same way cache_api.js was copied in Getting Started:
litert.js litert_embeddings.js sentencepiece.js tensorflow.js
Then load the entry point from web/index.html:
<script type="module" src="litert_embeddings.js"></script>
That is all of it — the WASM runtime underneath is fetched from a CDN, so there is nothing else to host.
Tap the recipes icon in the app bar, then Embed the corpus. After the download you get twelve green ticks and, at the bottom, what an embedding actually is:
768 dimensions
[-0.0147, 0.0412, -0.0038, 0.0221, 0.0095, -0.0176, ...]
That is the whole representation. Two recipes are "similar" when those two lists of numbers point in a similar direction — which is all a vector search ever computes.
Leave the page and come back. The ticks are gone. The vectors lived in a Map in the widget, and that is what Step 3 is about.
A Map and a hand-written cosine loop is where most RAG tutorials stop, and it does work — on twelve documents, once. Three things are wrong with it, and a bigger Map fixes none of them:
double is a float64, so one 768-dimension vector is 768 × 8 = 6 KB — not the 3 KB the model emits. Ten thousand documents is 60 MB of Dart heap, sitting beside an LLM that already wants a gigabyte or two.A vector store fixes all three for the same reason: the vectors stop being a Dart object and become rows in a database that knows they are vectors.
flutter_gemma ships two, behind one interface. This codelab uses sqlite-vec, and the table says why — but the code from here on is written against VectorStoreRepository, so swapping is one line either way.
|
| |
Platforms | all six, including web | five — no web |
Search | exact KNN, always | exact below 10 000 points, approximate (HNSW) above |
Scaling | brute force, linear in N | wins on large corpora |
| no-op on native, drains IndexedDB on web | required — points stay in memory until it is called |
Schema timing | at table creation — a new filter field means re-creating and re-indexing | at write time — declare and re-index |
Field names |
| free-form UTF-8, no |
Good to know: The precision row surprises people. qdrant's fullScanThreshold defaults to 10 000 points — below that it does a full scan and is exactly as precise as sqlite-vec. For a corpus that fits on a phone you are usually not choosing between exact and approximate at all; you are choosing between "runs in Chrome" and "grows better".
The last row is the one that quietly decides the others: the portable set is sqlite's. If you might ever switch backends, stay inside it — which is why this codelab's fields are named cuisine, minutes and vegetarian and not prep-time.
flutter pub add flutter_gemma_rag_sqlite path_provider
await FlutterGemma.initialize(
// ...
vectorStore: kIsWeb ? WebSqliteVectorStore() : SqliteVectorStore(),
);
Two classes, one per platform arm. The native one is sqlite3 over dart:ffi with the vec0 extension loaded; the web one is the same SQLite compiled to WASM with vec0 linked in, keeping its pages in IndexedDB. Both implement the same interface, which is what makes everything after this line platform-independent.
One more file, and it needs no