Embeddings & RAG

flutter_gemma can generate vector embeddings from text (EmbeddingGemma / Gecko on LiteRT, or BERT / MiniLM / WordPiece models via the ONNX backend) and run on-device RAG with a vector store. Two stores are available, both with the same Dart API: qdrant-edge — the fastest store on native (HNSW approximate nearest-neighbour) — and sqlite-vec — a portable, exact store that runs on all six platforms (Android, iOS, macOS, Linux, Windows, Web), and the only store that runs on Web. Your code is portable across both.

Setup#

Embeddings need the flutter_gemma_embeddings package plus a backend that implements it — flutter_gemma_litertlm's LiteRtEmbeddingBackend (or flutter_gemma_onnx's OnnxEmbeddingBackend for ONNX/ORT models, which also runs on Web via onnxruntime-web). RAG also needs a vector store package — flutter_gemma_rag_qdrant (native, fastest) or flutter_gemma_rag_sqlite (sqlite-vec; all platforms, including Web). Register them in await FlutterGemma.initialize(...):

await FlutterGemma.initialize(
  inferenceEngines: const [LiteRtLmEngine()],
  embeddingBackends: const [LiteRtEmbeddingBackend()], // flutter_gemma_litertlm
  vectorStore: QdrantVectorStore(),                    // or WebSqliteVectorStore() on web
);

See Installation for the full registration reference. On web, LiteRtEmbeddingBackend also needs its LiteRT.js loader script in web/index.html — see Installation → Web.

Text embeddings#

The embedding dimension depends on the model and backend. The LiteRT models (EmbeddingGemma / Gecko) generate 768-dimensional vectors; with OnnxEmbeddingBackend the dimension is model-dependent (e.g. all-MiniLM-L6-v2 is 384-dim). The number in a model name (64/256/512/1024/2048) is the max input sequence length in tokens, not the embedding dimension. See Models for the full list.

Install an embedding model#

await FlutterGemma.installEmbedder()
    .modelFromNetwork(
      'https://huggingface.co/litert-community/embeddinggemma-300m/resolve/main/embeddinggemma-300M_seq256_mixed-precision.tflite',
      token: 'hf_...',
    )
    .tokenizerFromNetwork(
      'https://huggingface.co/litert-community/embeddinggemma-300m/resolve/main/sentencepiece.model',
      token: 'hf_...',
    )
    .install();

Generate embeddings#

final embedder = await FlutterGemma.getActiveEmbedder();
final embeddings = await embedder.generateEmbeddings(
  docs.map((d) => d.content).toList(),
  taskType: TaskType.retrievalDocument,
);

On-device RAG / vector store#

All RAG operations live on the FlutterGemma.rag namespace — the canonical entry point. (The store is opt-in: register a vectorStore: in await FlutterGemma.initialize(...), or every rag call except flush() throws a clear "add a RAG package" error — flush() returns without doing anything.)

import 'package:flutter_gemma/flutter_gemma.dart';

// 1. Install an embedding model (any of Gecko / EmbeddingGemma) — see above.

// 2. Initialize the vector store (one shard per database path). On native pass
//    an absolute path: a bare name resolves against the process working
//    directory, which is not writable on Android or iOS. On web a name is enough.
final dir = await getApplicationDocumentsDirectory(); // package:path_provider
await FlutterGemma.rag.initialize('${dir.path}/rag_store');

// 3. Add documents — let flutter_gemma compute embeddings for you
for (final doc in docs) {
  await FlutterGemma.rag.addDocument(
    id: doc.id,
    content: doc.content,
    metadata: '{"category":"science","lang":"en"}',
  );
}

// 3b. Or batch-embed yourself and feed pre-computed vectors via
//     addDocumentWithEmbedding(...) for higher throughput.
final embedder = await FlutterGemma.getActiveEmbedder();
final embeddings = await embedder.generateEmbeddings(
  docs.map((d) => d.content).toList(),
  taskType: TaskType.retrievalDocument,
);
for (var i = 0; i < docs.length; i++) {
  await FlutterGemma.rag.addDocumentWithEmbedding(
    id: docs[i].id,
    content: docs[i].content,
    embedding: embeddings[i],
    metadata: '{"category":"science","lang":"en"}',
  );
}

// 3c. Persist what you indexed while the store stays open (see below)
await FlutterGemma.rag.flush();

// 4. Semantic search, with optional payload-aware Filter
final results = await FlutterGemma.rag.searchSimilar(
  query: 'quantum entanglement',
  topK: 10,
  threshold: 0.0,
  filter: Filter(
    must: [FieldEquals(key: 'category', value: 'science')],
    mustNot: [FieldEquals(key: 'lang', value: 'fr')],
  ),
);

// 5. Maintain the store: remove one document (no-op if the id is absent),
//    read stats, or clear everything.
await FlutterGemma.rag.removeDocument(id: 'doc-42');
final stats = await FlutterGemma.rag.stats();
await FlutterGemma.rag.clear();

Persisting the index: flush()#

Call FlutterGemma.rag.flush() after indexing. What it does depends on the store:

• qdrant-edge — required. New documents stay in memory until the store is flushed or closed, so an index built without either is lost when the process ends — an Android app killed in the background is the ordinary case. close() persists too, but only logs a failed save; flush() throws it.
• sqlite-vec, native — a no-op: every statement is on disk when it returns.
• sqlite-vec, web — drains the IndexedDB storage. On sqlite3 >= 3.4.0 it does not wait for a write batch already in flight (an upstream regression); close() is the stronger drain there.

A store that cannot persist at all (the web in-memory fallback) throws VectorStoreException rather than returning. A custom store that implements VectorStoreRepository must declare flush(); one that extends it inherits an empty default — override it if your store buffers writes.

The Filter API#

Filter supports must / should / mustNot lists of conditions:

• FieldEquals — exact match on a payload field.
• FieldRange — numeric range on a payload field.
• FieldMatchAny — match against any value in a set.

Both stores honor Filter on all platforms, and both need the filterable fields declared up front in a FilterSchema (see below). qdrant-edge promotes exactly the declared fields to payload keys at write time; sqlite-vec creates them as columns at table-creation time. On either store a condition on an undeclared field is dropped, as if it had never been written: the search returns the same hits as filter: null, and a filter mixing declared and undeclared fields narrows only by the declared ones. A missing declaration therefore looks like "my filter had no effect" — too many results, never an error and never zero.

Declaring filter fields#

The sqlite-vec store filters over declared columns. Describe them with a FilterSchema of FilterFields, and pass it either to initialize(...):

await FlutterGemma.initialize(
  vectorStore: SqliteVectorStore(),
  filterSchema: const FilterSchema(fields: [
    FilterField(name: 'category', type: FilterFieldType.string),
    FilterField(name: 'lang', type: FilterFieldType.string),
    FilterField(name: 'year', type: FilterFieldType.number),
  ]),
);

…or at runtime via configure(...) on the VectorStoreRepository — it returns void, so do not await it:

store.configure(FilterSchema(fields: [
  FilterField(name: 'category', type: FilterFieldType.string),
]));

FilterFieldType has exactly three values: string, number, bool.

What a field may be NAMED is decided by the store, and the two differ — the check runs in configure(), so you find out when you hand the schema over, not at the first insert.

SqliteVectorStore is the strict one: a name must match ^[A-Za-z][A-Za-z0-9_]*$ and must not be one vec0 already uses (id, embedding, content, metadata, distance, k). The name becomes a real vec0 column and sqlite-vec's DDL grammar has no quoted identifier form, so doc-type is unrepresentable there rather than merely unescaped.

QdrantVectorStore accepts far more — payload keys are free-form UTF-8 — with two rules of its own: no ., which qdrant reads as a nested-path separator, and none of the keys the store uses itself (__flutter_gemma_id, __flutter_gemma_content, __flutter_gemma_metadata).

So the portable set is sqlite's. If you may ever switch backends, stay inside it. Regardless of store, a schema with a duplicate or empty name is rejected.

A Filter over the declared fields is then applied inside the store; a filter referencing an undeclared field is silently ignored (no-op, never throws).

Platform support#

Both stores expose the identical Dart API, so you can swap one for the other by changing only the vectorStore: you register.