Models

Measured here on an M4 Pro (macOS, Metal), both published artifacts, same prompt and the same session code, backend the only variable:

Nothing fails: the Metal engine is created, activeBackend reports gpu (so it is not a silent fallback), generation runs and emits padding until it reaches the context limit. A gemma-4-E2B-it.litertlm bundle on the same machine, the same code path and the same backend answers correctly with no padding, so this is not the Metal path in general. Earlier revisions of this page said the bundle crashed the LiteRT GPU partitioner because its EMBEDDING_LOOKUP weights stay float32. Nothing in the runs above crashes, so that description does not hold; the repository has published these .litertlm artifacts since 2026-03-31, so it did not describe a later change either. Reproducing across two different quantization recipes also rules the bit width out.

Tracked at LiteRT-LM#1748. Use PreferredBackend.cpu for this model (≈90 s prefill on a 4 B int4 bundle on M-series Macs). If you are converting it yourself, AI Edge Quantizer is the supported route today — the litert-lm quantization CLI announced in that thread never shipped — with published recipes and Model Explorer for the layer regexes (maintainer's pointers).

ModelType reference#

When installing models, specify the correct ModelType:

Usage example:

// Gemma models
await FlutterGemma.installModel(modelType: ModelType.gemmaIt)
  .fromNetwork(url).install();

// DeepSeek models
await FlutterGemma.installModel(modelType: ModelType.deepSeek)
  .fromNetwork(url).install();

// Phi-4 (its own type — parses Phi's tool-call markers)
await FlutterGemma.installModel(modelType: ModelType.phi)
  .fromNetwork(url).install();

Supported models & platforms#

Installation sources#

// Network — .litertlm is the cross-platform default (Android/iOS/Desktop).
// For mobile-only or web-only apps you can substitute a .task URL — and drop
// the fileType, which defaults to ModelFileType.task.
await FlutterGemma.installModel(
  modelType: ModelType.gemmaIt,
  fileType: ModelFileType.litertlm,
)
  .fromNetwork('https://example.com/model.litertlm', token: 'optional')
  .install();

// Flutter assets
await FlutterGemma.installModel(
  modelType: ModelType.gemmaIt,
  fileType: ModelFileType.litertlm,
)
  .fromAsset('assets/models/model.litertlm')
  .install();

// Native bundle
await FlutterGemma.installModel(
  modelType: ModelType.gemmaIt,
  fileType: ModelFileType.litertlm,
)
  .fromBundled('model.litertlm')
  .install();

// External file (native only)
await FlutterGemma.installModel(
  modelType: ModelType.gemmaIt,
  fileType: ModelFileType.litertlm,
)
  .fromFile('/path/to/model.litertlm')
  .install();

Hugging Face#

Install a model straight from a Hugging Face repo. The resolver that reads a repo's deployment manifest ships with the engine, so registering the engine is enough — there is no separate resolver list to maintain.

One call (manifest-driven) — omit file, and fromHuggingFace(repo) resolves the repo's manifest (e.g. litertlm_manifest.json) at install time, installs the right revision-pinned variant for the device, and returns the manifest's overridable runtime defaults on the result:

// The engine carries its resolver — nothing else to register.
await FlutterGemma.initialize(inferenceEngines: [LiteRtLmEngine()]);

final install = await FlutterGemma.installModel(
  modelType: ModelType.general,     // fallback — the manifest overrides it
  fileType: ModelFileType.litertlm, // selects the litertlm resolver
)
  .fromHuggingFace('litert-community/Qwen3-4B-Thinking-2507')
  .install();

final model = await FlutterGemma.getActiveModel(defaults: install.runtime);
// `minOutputTokens` is a FLOOR, not a cap — leave `maxOutputTokens` unset (or
// keep it >= `install.runtime?.minOutputTokens`). Passing the floor as the cap
// would truncate a reasoning model mid-thought.
final session = await model.createSession(
  enableThinking: install.runtime?.isThinking ?? false,
);

.onnx repos install the whole ORT-GenAI directory — the ONNX resolver (carried by OnnxEngine) lists the repo, picks a CPU execution-provider folder (the bundled runtime is CPU-only; pin one with OnnxHuggingFaceResolver(variant: …)) and downloads every file in it; see ONNX. .builtIn repos surface that OS models have no Hugging Face file.

Explicit file — pass file, and fromHuggingFace(repo, file:) resolves …/resolve/<revision>/<file> and installs it directly, for any fileType (no manifest needed; the HF token is applied to huggingface.co automatically):

await FlutterGemma.installModel(
  modelType: ModelType.gemmaIt,
  fileType: ModelFileType.litertlm,
)
  .fromHuggingFace('litert-community/Gemma3-1B-IT', file: 'model.litertlm', token: 'optional')
  .install();

Inspect first — resolveHuggingFace(repo, fileType:) returns the resolved identity plus overridable runtime defaults WITHOUT installing, so you can inspect the variant, its notes, and defaults before committing:

final r = await FlutterGemma.resolveHuggingFace(
  'litert-community/Qwen3-4B-Thinking-2507',
  fileType: ModelFileType.litertlm,
);
// … inspect r.file / r.notes / r.runtime …
await FlutterGemma.installModel(
  modelType: r.modelType ?? ModelType.general,
  fileType: r.fileType,
)
  .fromNetwork(r.url) // r.url pins the resolver's revision
  .install();

The resolver lives in the engine package — LitertlmManifestResolver in flutter_gemma_litertlm reads litertlm_manifest.json — and rides on the engine via HuggingFaceResolverSource, so initialize(inferenceEngines: […]) auto-registers it. Pass initialize(huggingFaceResolvers: [...]) only to override an engine's default (e.g. LitertlmManifestResolver(revision: 'abc123') to pin a commit).

Source capabilities#

Android foreground service (large downloads)#

Android has a 9-minute background execution limit. A foreground service shows a notification and exempts the download from battery-optimization kills, and you have to ask for it — the default configures no notification, and the platform will not start the service without one, so file size alone changes nothing:

// DEFAULT — no foreground service; pass foreground: true to get one
await FlutterGemma.installModel(modelType: ModelType.gemmaIt)
  .fromNetwork(url)  // foreground: null
  .install();

// Force foreground mode
await FlutterGemma.installModel(modelType: ModelType.gemmaIt)
  .fromNetwork(url, foreground: true)
  .install();

iOS uses native URLSession which handles long downloads automatically — no foreground service needed.

Cancelling downloads#

import 'package:flutter_gemma/core/model_management/cancel_token.dart';

final cancelToken = CancelToken();

final future = FlutterGemma.installModel(modelType: ModelType.gemmaIt)
  .fromNetwork(url)
  .withCancelToken(cancelToken)
  .withProgress((progress) => print('Progress: $progress%'))
  .install();

// Cancel from elsewhere (e.g. user pressed a cancel button)
cancelToken.cancel('User cancelled download');

try {
  await future;
} catch (e) {
  if (CancelToken.isCancel(e)) {
    print('Download was cancelled by user');
  }
}

CancelToken cancels all files in multi-file downloads (e.g. embedding model + tokenizer), works on mobile + web, and throws DownloadCancelledException.

Speech models#

On-device speech via the opt-in flutter_gemma_speech package — selectable, profile-driven pipelines (you pick the model with SttModelType / TtsModelType). moonshine-tiny, Whisper, and Parakeet (STT) and Matcha, Qwen3-TTS (multilingual), and Inflect-Nano-v2 (fast) TTS work end-to-end today; kokoro / supertonic TTS voices are follow-ons. Native only (no Web). See Speech for usage. A VoiceSession also chains STT → LLM → TTS into a push-to-talk voice loop — see Speech.

Speech-to-text

Whisper is multilingual — the shipped checkpoints are the multilingual ones (no .en suffix), so all 99 of Whisper's languages are available. Set a default for the recognizer, override it per transcription, or both:

// Default for every transcription on this recognizer.
final stt = await FlutterGemma.getActiveStt(language: 'de');
final german = await stt.transcribe(germanPcm);

// One call in another language — same recognizer, nothing reloaded.
final french = await stt.transcribe(frenchPcm, language: 'fr');

Both are free: the language is one token in the decoder's seed prompt, and that prompt is rebuilt on every transcription. Switching languages never reloads the model or invalidates the recognizer you are holding.

The value is Whisper's own language code without the delimiters, and it defaults to 'en'. It decides the OUTPUT language only — the weights understand the audio either way, so asking for 'en' on German speech returns an English translation rather than an error.

Moonshine and Parakeet have no language token in their decoder prompt and reject the parameter with an ArgumentError rather than ignoring it; both are English-only.

Text-to-speech

Text embedding models#

The LiteRT embedding models below generate 768-dimensional vectors. The numbers in names (64/256/512/1024/2048) indicate maximum input sequence length in tokens, not embedding dimension. (With the ONNX backend the dimension is model-dependent — e.g. all-MiniLM-L6-v2 is 384-dim.) See Embeddings & RAG for usage.

Performance (Android Pixel 8):

• Gecko 64: ~109 ms/doc embedding, 130 ms search (fastest — 2.6× faster than EmbeddingGemma).
• EmbeddingGemma 256: ~286 ms/doc embedding, 342 ms search (more accurate — 300M vs 110M params).

SigLIP 2 text tower (ONNX, manual wiring)#

flutter_gemma_embeddings ships a SigLIP 2 text profile, for putting text into the same space as SigLIP's vision tower — image↔text retrieval rather than document RAG. It is the only embedding profile here that is not installed through installEmbedder().

Text in, vectors out — the plugin does not run the vision tower. You embed the image side elsewhere (or offline) and query it with vectors this profile produces.

Its convention differs from every model above: no leading BOS, a single trailing <eos>, lowercased text, and a fixed 64-token width that lives in the token ids because the int8 export carries no attention_mask. It also ignores the TaskType prefix — the vision tower encodes an image with no prefix, so adding one moves the text vector off the space the two towers share.

SigLIP 2 reuses the Gemma BPE vocabulary, so a tokenizer.json cannot be told apart by its vocabulary alone, and the ONNX tokenizer loader refuses such a file rather than reading it with Gemma's convention and returning a plausible but wrong vector.

To tell whether an export is the one this profile expects, look at two blocks of its tokenizer.json — the same two the refusal keys on:

• "padding" declares a fixed width — "strategy": {"Fixed": …}, not "BatchLongest"
• "post_processor" appends <eos> and prepends no <bos>

The refusal needs both, so there are three outcomes rather than two. A file matching both is SigLIP 2's convention. One that prepends <bos> is an EmbeddingGemma-family file and belongs on the profiles above. A file failing either check — no padding block, "BatchLongest", or a post_processor that is missing, empty, or does not end in a special token — is not classified at all: it is read with Gemma's convention and nothing is raised.

So do not read the absence of an error as approval. A SigLIP 2 export that declares both blocks is caught, but one that dropped either — a re-export, or a tool that strips them — reaches the Gemma path and produces exactly the silently wrong vector this section warns about.

Wire it yourself:

import 'package:flutter_gemma_embeddings/embedding_tokenizer.dart'
    show loadSiglipSentencePieceEmbeddingTokenizer;

and pass that as the tokenizer factory of the ForwardPassDescriptor you give to CommonEmbeddingModel.create. That library is native-only. See the flutter_gemma_embeddings README for the full profile, and ONNX Runtime for why the factory declines to guess.