An offline travel guide. You are abroad, roaming is expensive or missing entirely, and you still want to ask "what should I see in Prague?" That is the reason this app is hybrid rather than one more chat with Gemini — the phone has to be able to answer on its own.
It is a single screen: a chat that streams its replies. Everything else on that screen exists to make where the answer came from visible.
Sources: Paris line under the reply is the only place you can see that the answer was grounded in a guide rather than invented.When you reach the end, this is the demo — worth reading now, so you know what you are building toward:
Sources: Paris line appears, and the answer now leans on the bundled guide instead of the model's memory.Six increments, each a directory you can open and run:
genkit_google_genaigenkit_flutter_gemmagenkit_hybrid (fallback, capability, cascade, budget)Genkit instance and genkit_hybrid's routing strategiesgenkit_flutter_gemma┌──────────────────────────────────────────┐
│ Flutter App │
├──────────────────────────────────────────┤
│ AiEngine │
│ one Genkit, two plugins │
├──────────────────┬───────────────────────┤
│ googleAI plugin │ GenkitFlutterGemma │
│ gemini-3.7-flash│ Gemma 3 1B + │
│ (kCloud) │ EmbeddingGemma │
│ │ (kOnDevice) │
└──────────────────┴───────────────────────┘
genkit_hybrid composes both branches into one routable Model — hybridModel() / cascadeModel() — selected by a PolicyMode: cloud, local, smart, cascade, budget.
The key insight: AiEngine builds a single Genkit instance with both plugins registered, resolves the cloud and on-device models once, and hands them to genkit_hybrid as a Map of branches. For each PolicyMode it composes one branch map into an ordinary Genkit Model and registers it on ai.registry — once, during initialize(). engine.modelFor(policy) just returns that already-registered Model. The chat screen always calls the same ai.generateStream(model: engine.modelFor(policy), messages: [...]); only which Model modelFor returns changes with the policy.
Every step of this codelab exists as a complete, runnable app, so you can join at any point or check your work against the next one.
git clone --depth 1 https://github.com/DenisovAV/flutter_gemma.git
cd flutter_gemma/codelabs/hybrid-ai-flutter-genkit
ls
step_00_starter/ the shell you start from
step_01_cloud_ai/ after Step 2 — Gemini, streaming
step_02_local_ai/ after Step 3 — Gemma 3 1B on the device
step_03_hybrid/ after Step 4 — one AiEngine, two branches
step_04_smart_routing/ after Step 4.5 — images and the routing policies
step_05_embeddings/ after Step 5 — on-device embeddings
complete/ after Step 6 — the finished app, with RAG
Open step_00_starter in your IDE and run flutter pub get. The starter includes:
lib/main.dart — Simple app entry point, no async setup neededlib/screens/chat_screen.dart — Chat UI with TextField, ListView, send buttonlib/widgets/message_bubble.dart — Styled message bubbles (user right, AI left)lib/models/message_model.dart — Simple ChatMessage data classlib/services/ai_service.dart — Abstract AIService interfaceassets/tourist_data/ — 10 JSON files with city descriptions (Paris, Tokyo, New York...)All our AI services implement this contract:
abstract class AIService {
Future<void> initialize();
Stream<String> generateResponseStream(String prompt);
Future<void> dispose();
}
generateResponseStream returns a Stream — responses stream token-by-token for a real-time chat feel.
cd step_00_starter
flutter run
You'll see the chat UI. Messages echo back with a placeholder. Let's replace that with real AI next.
Go to aistudio.google.com, sign in with your Google account, and click Get API key. Copy the key — you'll use it with --dart-define.
No Firebase project, no CLI setup — just an API key.
Model choice: this workshop pins gemini-3.7-flash — the older gemini-2.5-flash is being phased out.
In pubspec.yaml, uncomment the Genkit dependencies:
dependencies:
flutter:
sdk: flutter
cupertino_icons: ^1.0.8
# Step 2: Cloud AI
genkit: ^0.16.0
genkit_google_genai: ^0.3.1
Run flutter pub get.
Create lib/services/cloud_ai_service.dart:
import 'package:genkit/genkit.dart';
import 'package:genkit_google_genai/genkit_google_genai.dart';
import 'ai_service.dart';
// Pass at build time: flutter run --dart-define=GEMINI_API_KEY=AIza...
const String _apiKey = String.fromEnvironment('GEMINI_API_KEY');
class CloudAIService implements AIService {
Genkit? _ai;
@override
Future<void> initialize() async {
if (_apiKey.isEmpty) {
throw StateError(
'GEMINI_API_KEY is not set. '
'Run with --dart-define=GEMINI_API_KEY=your_key',
);
}
_ai = Genkit(plugins: [googleAI(apiKey: _apiKey)]);
}
@override
Stream<String> generateResponseStream(String prompt) async* {
final ai = _ai;
if (ai == null) throw StateError('CloudAIService not initialized');
final stream = ai.generateStream(
model: googleAI.gemini('gemini-3.7-flash'),
prompt: prompt,
);
await for (final chunk in stream) {
if (chunk.text.isNotEmpty) yield chunk.text;
}
}
@override
Future<void> dispose() async {
_ai = null;
}
}
Replace the echo stub:
import '../services/cloud_ai_service.dart';
// in _ChatScreenState:
late final CloudAIService _service;
// in initState:
_service = CloudAIService();
_initService();
// in _initService:
await _service.initialize();
// in _sendMessage:
await for (final chunk in _service.generateResponseStream(text)) {
buffer.write(chunk);
// ... the throttled setState loop, then _scrollToBottom()
}
flutter run --dart-define=GEMINI_API_KEY=your_key_here
Type "Tell me about Paris" — Gemini streams a response token by token.
What happened? Genkit(plugins: [googleAI(...)]) registered Gemini as a model provider. ai.generateStream(model: googleAI.gemini('gemini-3.7-flash'), ...) streams the response. The Genkit instance is the single point of contact for all AI operations.
This is the only step with platform configuration in it, and it is less than you would expect on any of the six. Read the subsection for the platform you are running on and skip the others. Getting Started covers each of them at length in its Step 2; what follows is what this app needs.
Android — one line, because both the model download and the Gemini call are ordinary HTTPS requests (android/app/src/main/AndroidManifest.xml):
<uses-permission android:name="android.permission.INTERNET" />
...and the API floor in android/app/build.gradle.kts, because the on-device half of the app loads libLiteRtLm.so:
defaultConfig {
// ...
// libLiteRtLm.so needs API 30+ Bionic (pthread_cond_clockwait,
// sem_clockwait). Below 30 the app installs and then fails at the first
// model load with a dlopen error.
minSdk = 30
iOS — a deployment target of 15.0 or newer, and three memory entitlements in ios/Runner/Runner.entitlements:
<key>com.apple.developer.kernel.extended-virtual-addressing</key>
<true/>
<key>com.apple.developer.kernel.increased-memory-limit</key>
<true/>
<key>com.apple.developer.kernel.increased-debugging-memory-limit</key>
<true/>
Point the Runner target at that file in Xcode's Signing & Capabilities editor. The keys lift the per-process memory ceiling iOS imposes: half a gigabyte of weights plus a KV cache is comfortably over the default jetsam limit on an older iPhone, and the kill that follows has no Dart-visible error — the app simply disappears.
macOS — two entitlements and one build phase. The entitlements go in both macos/Runner/DebugProfile.entitlements and macos/Runner/Release.entitlements, and every step app from this one on already carries them:
<key>com.apple.security.cs.disable-library-validation</key>
<true/>
<key>com.apple.security.network.client</key>
<true/>
network.client is what lets a sandboxed macOS app reach Gemini and Hugging Face at all. disable-library-validation does nothing yet: it matters once Hardened Runtime is on, which notarizing the app for distribution requires. The build phase below signs LiteRT-LM and its companion libraries ad hoc, and library validation refuses code not signed by Apple or by your own team. The iOS keys above are deliberately not here: they are iOS entitlements. On macOS a build with no signing team fails outright with "Runner" has entitlements that require signing with a development certificate, and a team-signed build silently drops them. macOS support is Apple Silicon only.
The build phase is the part unique to macOS. Every step app from this one on ships a macos/Podfile whose post_install block stages the runtime's companion libraries into the built .app. A macOS build that succeeds proves nothing here — the app compiles, links, signs and launches without the staging too, and the failure arrives at the first model load. One trap, measured: with Swift Package Manager on and no other CocoaPods plugin in the app, Flutter prints Removing CocoaPods integration, the post_install block never runs, and nothing is staged. Either turn SPM off with flutter config --no-enable-swift-package-manager, or keep one CocoaPods plugin in the app.
Windows — nothing in the app, and x86_64 only: there is no Windows arm64 build of the runtime. The machine needs the Microsoft Visual C++ Redistributable (2019 or newer), which the DirectX shader compiler behind the GPU backend links against.
Linux — nothing in the app either. glibc 2.34 or newer, which means Ubuntu 22.04+, Debian 12+ or RHEL 9+; building a Flutter Linux app at all also wants clang cmake ninja-build libgtk-3-dev lld, and flutter doctor names whichever of those you are missing.
Web — one script tag. The on-device arm loads the runtime from a CDN, and that ES module assigns no window globals — module scripts are deferred, so Dart would reach the engine before the constructor exists. web/index.html publishes a promise instead, and Dart awaits it. Every step app from this one on carries it in :
<script type="module">
window.litertLmReady = (async () => {
const m = await import('https://cdn.jsdelivr.net/npm/@litert-lm/[email protected]/+esm');
window.Engine = m.Engine;
return m.Engine;
})();
</script>
The web arm is an early preview: WebGPU, and text only. That matters for one policy in particular — an image on Smart still routes to the cloud, which is the only branch that declares vision anywhere.
Add genkit_flutter_gemma and flutter_gemma:
# Step 3: On-device AI (LiteRT-LM engine)
genkit_flutter_gemma: ^0.6.0
flutter_gemma: ^1.8.1
# flutter_gemma 1.x registers no engine by default — opt into LiteRT-LM
# (.litertlm inference) here.
flutter_gemma_litertlm: ^1.6.3
Run flutter pub get.
Go to huggingface.co, sign in, and create a read-access token at Settings → Access Tokens.
Create lib/services/local_ai_service.dart:
import 'package:flutter_gemma/flutter_gemma.dart';
import 'package:flutter_gemma_litertlm/flutter_gemma_litertlm.dart';
import 'package:genkit/genkit.dart';
import 'package:genkit_flutter_gemma/genkit_flutter_gemma.dart';
import 'ai_service.dart';
// The on-device LLM installs straight from Hugging Face by repo + file.
const String _hfRepo = 'litert-community/Gemma3-1B-IT';
const String _hfModelFile =
'Gemma3-1B-IT_multi-prefill-seq_q4_ekv4096.litertlm';
const String _embeddingModelUrl =
'https://huggingface.co/litert-community/embeddinggemma-300m/resolve/main/embeddinggemma-300M_seq256_mixed-precision.tflite';
const String _tokenizerUrl =
'https://huggingface.co/litert-community/embeddinggemma-300m/resolve/main/sentencepiece.model';
// Pass at build time: flutter run --dart-define=HF_TOKEN=hf_xxx
const String _hfToken = String.fromEnvironment('HF_TOKEN');
const String _modelName = 'gemma-3-1b-it';
const String _embedderName = 'embedding-gemma-300m';
class LocalAIService implements AIService {
Genkit? _ai;
bool _isInitialized = false;
bool get isInitialized => _isInitialized;
// Shared Genkit instance exposed for RagService to use for embeddings.
Genkit get ai {
final ai = _ai;
if (ai == null) throw StateError('LocalAIService not initialized');
return ai;
}
String get embedderName => _embedderName;
@override
Future<void> initialize({void Function(int)? onProgress}) async {
if (_isInitialized) return;
// flutter_gemma 1.x registers no engine by default — opt into LiteRT-LM.
await FlutterGemma.initialize(inferenceEngines: [LiteRtLmEngine()]);
// Download the .litertlm model (skipped if already installed).
await FlutterGemma.installModel(
modelType: ModelType.gemmaIt,
fileType: ModelFileType.litertlm,
)
.fromHuggingFace(
_hfRepo,
file: _hfModelFile,
token: _hfToken.isEmpty ? null : _hfToken,
)
.withProgress((p) => onProgress?.call(p)) // p is int 0..100
.install();
await FlutterGemma.installEmbedder()
.modelFromNetwork(
_embeddingModelUrl,
token: _hfToken.isNotEmpty ? _hfToken : null,
)
.tokenizerFromNetwork(
_tokenizerUrl,
token: _hfToken.isNotEmpty ? _hfToken : null,
)
.install();
// One Genkit instance for both inference and embeddings.
_ai = Genkit(
plugins: [
GenkitFlutterGemmaPlugin(
models: [
FlutterGemmaModelConfig(
name: _modelName,
modelType: ModelType.gemmaIt,
fileType: ModelFileType.litertlm,
),
],
embedders: [FlutterGemmaEmbedderConfig(name: _embedderName)],
),
],
);
_isInitialized = true;
}
@override
Stream<String> generateResponseStream(String prompt) async* {
final stream = ai.generateStream(
model: flutterGemma.model(_modelName),
prompt: prompt,
);
await for (final chunk in stream) {
if (chunk.text.isNotEmpty) yield chunk.text;
}
}
@override
Future<void> dispose() async {
_ai = null;
_isInitialized = false;
}
}
Four things in that file belong to Step 5 rather than to this one — the two embedding URLs, the installEmbedder() call, the ai getter and embedderName. They ship here so that the RAG step is a new file and not a second edit of this one; ignore them until then.
flutter run --dart-define=GEMINI_API_KEY=your_key --dart-define=HF_TOKEN=hf_xxx
The first run downloads ~550 MB. Subsequent runs use the cached model.
Key insight: Notice that generateResponseStream looks identical to CloudAIService — only the model: parameter changes. Genkit decouples what model to use from how to call it.
// Cloud:
ai.generateStream(model: googleAI.gemini('gemini-3.7-flash'), prompt: prompt)
// Local:
ai.generateStream(model: flutterGemma.model('gemma-3-1b-it'), prompt: prompt)
Same API. Different backends.
Coming up: In Step 4 we retire CloudAIService and LocalAIService as separate classes. A single AiEngine builds one Genkit with both plugins registered, and genkit_hybrid composes the two resolved models into one routable Model — the app still calls ai.generateStream(model: ..., ...), it just gets that one Model from AiEngine instead of picking a service.
Add genkit_hybrid — the routing layer. Everything else (the cloud + on-device plugins and flutter_gemma_litertlm) is already in place from Steps 2–3:
# Hybrid on-device ↔ cloud routing
genkit_hybrid: ^0.2.1
Run flutter pub get.
rm lib/services/ai_service.dart lib/services/cloud_ai_service.dart \
lib/services/local_ai_service.dart
They're replaced by one AiEngine that owns a single Genkit instance for both plugins.
Create lib/services/ai_engine.dart:
import 'package:flutter/foundation.dart' show debugPrint, visibleForTesting;
import 'package:flutter_gemma/flutter_gemma.dart';
import 'package:flutter_gemma_litertlm/flutter_gemma_litertlm.dart';
import 'package:genkit/genkit.dart';
import 'package:genkit/plugin.dart' show GenkitPlugin;
import 'package:genkit_flutter_gemma/genkit_flutter_gemma.dart';
import 'package:genkit_google_genai/genkit_google_genai.dart';
import 'package:genkit_hybrid/genkit_hybrid.dart';
// Prod installs the on-device LLM straight from Hugging Face by repo + file
// (the plugin applies the configured token to gated huggingface.co URLs).
const _hfRepo = 'litert-community/Gemma3-1B-IT';
const _hfModelFile = 'Gemma3-1B-IT_multi-prefill-seq_q4_ekv4096.litertlm';
const _embeddingModelUrl =
'https://huggingface.co/litert-community/embeddinggemma-300m/resolve/main/embeddinggemma-300M_seq256_mixed-precision.tflite';
const _tokenizerUrl =
'https://huggingface.co/litert-community/embeddinggemma-300m/resolve/main/sentencepiece.model';
// Pass at build time: --dart-define=HF_TOKEN=hf_xxx --dart-define=GEMINI_API_KEY=AIza...
const _hfToken = String.fromEnvironment('HF_TOKEN');
const _geminiApiKey = String.fromEnvironment('GEMINI_API_KEY');
const kLocalModel = 'gemma-3-1b-it';
const kCloudModel = 'gemini-3.7-flash';
const kEmbedder = 'embedding-gemma-300m';
/// Context window for the on-device branch, in tokens. `maxTokens` is the
/// WHOLE window (input + output) and genkit_flutter_gemma defaults it to 1024.
/// RagService's take(3) of ~600-token city guides alone is ~1.7k tokens —
/// measured on device: "Input token ids are too long ... 1713 >= 1024". The
/// bundled Gemma-3-1B `.litertlm` is built for 4096 (`ekv4096`), so use it.
const kOnDeviceContextTokens = 4096;
/// The five routing policies the chat exposes. Each maps to one genkit_hybrid
/// construct (see [modelFor] / [strategyFor]), and carries the UI facts about
/// itself — its label, which branches it needs, whether it is text-only — so
/// the screen renders the dropdown from `PolicyMode.values` instead of
/// restating all five modes by hand.
enum PolicyMode {
cloud('Cloud', needsLocal: false),
local('Local', needsCloud: false, textOnly: true),
smart('Smart (image-aware)'),
cascade('Cascade (escalate on quality)', textOnly: true),
budget('Budget (cost-gated)');
const PolicyMode(
this.label, {
this.needsCloud = true,
this.needsLocal = true,
this.textOnly = false,
});
/// Dropdown text.
final String label;
/// Which branches the composite for this mode requires.
final bool needsCloud;
final bool needsLocal;
/// True when the primary route starts on the text-only on-device model, so
/// an attached image cannot be handled (the UI blocks send with a hint).
final bool textOnly;
bool availableWith({required bool cloud, required bool local}) =>
(!needsCloud || cloud) && (!needsLocal || local);
}
/// Owns a single Genkit instance with both plugins (cloud + on-device),
/// resolves the two base models, and composes them via genkit_hybrid per the
/// selected [PolicyMode]. Replaces the old Cloud/Local/HybridAIService trio.
class AiEngine {
Genkit? _ai;
Model? _local;
Model? _cloud;
/// One composite [Model] per policy, built AND registered once by
/// [_registerPolicyModels] — genkit reduces `generate(model: ...)` to the
/// model's name and looks it up in the registry, so a freshly built,
/// never-registered composite would fail every call with NOT_FOUND.
final Map<PolicyMode, Model> _models = {};
bool cloudReady = false;
bool localReady = false;
// CostStrategy demo signal: the app counts cloud calls against a small cap.
int cloudCallsSpent = 0;
int budgetCap = 3;
bool get budgetAvailable => cloudCallsSpent < budgetCap;
AiEngine();
/// Test seam: skips [FlutterGemma.initialize]/`installModel` (real I/O that
/// can't run in a unit test) and takes already-resolved branch models
/// directly, then runs the same build+register path [initialize] uses — so
/// a test driving [modelFor] through `ai.generate` here exercises the real
/// registration wiring, not just [strategyFor]. [local] is nullable —
/// mirrors [cloud] — so a cloud-only engine (the symmetric mirror of the
/// cloud-absent case) is constructible too.
@visibleForTesting
AiEngine.forTest({required Genkit ai, Model? local, Model? cloud}) {
_ai = ai;
_local = local == null ? null : _withContextBudget(local);
_cloud = cloud;
localReady = local != null;
cloudReady = cloud != null;
_registerPolicyModels();
}
Genkit get ai {
final ai = _ai;
if (ai == null) throw StateError('AiEngine not initialized');
return ai;
}
String get embedderName => kEmbedder;
Future<void> initialize({
void Function(int progress)? onProgress,
// Test seam: install the LLM from a pre-staged local file instead of
// downloading it — avoids a flaky ~500MB on-device download on CI / FTL.
String? localModelPath,
// Test seam: skip the embedder download when RAG isn't exercised.
bool downloadEmbedder = true,
}) async {
// Declarative plugin config — always includes the on-device plugin (its
// models/embedders are looked up by name later, independent of whether
// the install below actually succeeds).
final plugins = <GenkitPlugin>[
if (_geminiApiKey.isNotEmpty) googleAI(apiKey: _geminiApiKey),
GenkitFlutterGemmaPlugin(
models: [
FlutterGemmaModelConfig(
name: kLocalModel,
modelType: ModelType.gemmaIt,
fileType: ModelFileType.litertlm,
),
],
embedders: downloadEmbedder
? [FlutterGemmaEmbedderConfig(name: kEmbedder)]
: const [],
),
];
// Build Genkit BEFORE any on-device engine registration/install so `_ai`
// (and `_resolve`, and the `ai` getter) are always available afterward —
// the plugin list above is purely declarative (no I/O, no dependency on
// FlutterGemma.initialize() having run), so cloud resolution below needs
// no on-device engine and must not be taken down by a failure
// registering/installing it.
_ai = Genkit(plugins: plugins);
// CLOUD: needs no install, so its readiness never depends on the local
// LLM or the (optional) embedder below.
if (_geminiApiKey.isNotEmpty) {
try {
_cloud = await _resolve(googleAI.gemini(kCloudModel));
cloudReady = true;
} catch (e) {
debugPrint('⚠️ AiEngine: CLOUD backend unavailable — $e');
cloudReady = false;
}
}
// LOCAL: register the on-device engine, then install + resolve the LLM.
// flutter_gemma 1.x registers no engines by default; that registration
// now lives inside this try/catch (not before Genkit is built) so an
// engine-init failure only suppresses localReady, never cloud.
try {
// Opt into LiteRT-LM (.litertlm inference) + its LiteRT embedding
// backend.
await FlutterGemma.initialize(
inferenceEngines: [LiteRtLmEngine()],
embeddingBackends: [LiteRtEmbeddingBackend()],
);
// fileType MUST be litertlm to match the LiteRT-LM engine registered
// above.
final llm = FlutterGemma.installModel(
modelType: ModelType.gemmaIt,
fileType: ModelFileType.litertlm,
);
if (localModelPath != null) {
await llm.fromFile(localModelPath).install();
} else {
await llm
.fromHuggingFace(
_hfRepo,
file: _hfModelFile,
token: _hfToken.isEmpty ? null : _hfToken,
)
.withProgress((p) => onProgress?.call(p)) // p is int 0..100
.install();
}
_local = _withContextBudget(
await _resolve(flutterGemma.model(kLocalModel)),
);
localReady = true;
} catch (e) {
debugPrint('⚠️ AiEngine: on-device backend unavailable — $e');
localReady = false;
}
// EMBEDDER (OPTIONAL): RAG-only, never blocks chat — a failure here must
// not flip localReady or rethrow.
if (downloadEmbedder && localReady) {
try {
await FlutterGemma.installEmbedder()
.modelFromNetwork(
_embeddingModelUrl,
token: _hfToken.isEmpty ? null : _hfToken,
)
.tokenizerFromNetwork(
_tokenizerUrl,
token: _hfToken.isEmpty ? null : _hfToken,
)
.install();
} catch (e) {
debugPrint('⚠️ AiEngine: embedder install failed, RAG disabled — $e');
}
}
_registerPolicyModels();
}
// A plugin model is registered by name; genkit's `Model` is an `Action`, so
// look the concrete model up from the registry and cast.
Future<Model> _resolve(ModelRef ref) async {
final action = await ai.registry.lookupAction(ActionType.model, ref.name);
if (action == null) {
throw StateError('model "${ref.name}" is not registered');
}
return action as Model;
}
/// Wraps the on-device branch so every request carries the context budget
/// unless the caller set one. Copies the request rather than mutating it:
/// genkit_hybrid hands the SAME ModelRequest to the next branch when
/// cascade escalates, so an in-place write would leak a Gemma-only
/// maxTokens into the Gemini call. The metadata copy is required too —
/// genkit's Model constructor writes into the map it is handed.
Model _withContextBudget(Model inner) => Model(
name: '${inner.name}/ctx',
metadata: {...inner.metadata},
fn: (request, context) {
if (request == null || request.config?['maxTokens'] != null) {
return inner.fn(request, context);
}
final budgeted = ModelRequest.fromJson({
...request.toJson(),
'config': {...?request.config, 'maxTokens': kOnDeviceContextTokens},
});
return inner.fn(budgeted, context);
},
);
Map<String, Model> get _branches => {kOnDevice: ?_local, kCloud: ?_cloud};
/// Builds AND registers one composite [Model] per [PolicyMode] whose
/// required branches are available, populating [_models]. A mode that needs
/// `kCloud` (every mode but `local`) is skipped when there's no API key, so
/// a missing cloud branch degrades to a clear [modelFor] error instead of
/// crashing here on a half-built `cascadeModel` (its `order` validates
/// eagerly against `branches`, unlike `hybridModel`).
void _registerPolicyModels() {
for (final mode in PolicyMode.values) {
if (!mode.availableWith(cloud: _cloud != null, local: _local != null)) {
continue;
}
final model = _buildModel(mode);
ai.registry.register(model);
_models[mode] = model;
}
}
/// The composable Genkit `Model` for [mode]. Cascade is a `cascadeModel`;
/// every other mode is `hybridModel(strategy: strategyFor(mode))`. Called
/// once per mode by [_registerPolicyModels] — not a per-request factory.
Model _buildModel(PolicyMode mode) {
if (mode == PolicyMode.cascade) {
return cascadeModel(
branches: _branches,
order: const [kOnDevice, kCloud],
// DEMO PROXY — not a real quality signal. A production cascade
// escalates on model *confidence*; the best training-free signal is
// the reply's average token log-probability. We can't use it here:
// LiteRT-LM gives `accept` only decoded text (no per-token
// probabilities), and asking a ~1B model to self-rate confidence is
// unreliable — small models are confidently wrong. So we escalate on a
// crude "too short to be a real answer" check. See the codelab's
// "A real cascade signal" note.
accept: (r) => r.text.trim().length > 20,
name: 'cascade',
);
}
return hybridModel(
branches: _branches,
strategy: strategyFor(mode),
name: mode.name,
);
}
/// The registered, resolvable `Model` for [mode] — built once by
/// [_registerPolicyModels] during [initialize] (or [AiEngine.forTest]).
Model modelFor(PolicyMode mode) {
final model = _models[mode];
if (model == null) {
throw StateError(
'No model registered for $mode — call initialize() first (or, if '
'this mode needs the cloud branch, make sure a cloud API key is set).',
);
}
return model;
}
/// Pure policy → RoutingStrategy mapping (no models needed), so the routing
/// decisions are unit-testable. [PolicyMode.cascade] has no strategy — it is
/// a Model, built in [_buildModel].
RoutingStrategy strategyFor(PolicyMode mode) {
switch (mode) {
case PolicyMode.cloud:
return PreRoutingStrategy((_) => kCloud);
case PolicyMode.local:
return PreRoutingStrategy((_) => kOnDevice);
case PolicyMode.smart:
// Image → cloud only (only it declares vision). Text → both qualify,
// cloud-first in `supports` insertion order, on-device as the tail.
// No WithFallback: CapabilityStrategy already yields the on-device
// tail for text, and for an image a forced on-device tail would hand
// the picture to a model that cannot see it. Offline + image should
// fail loudly, not silently degrade to text-only.
return CapabilityStrategy(
supports: {
kCloud: {ModelCapability.vision},
kOnDevice: <ModelCapability>{},
},
);
case PolicyMode.budget:
return CostStrategy(
budgetAvailable: () => budgetAvailable,
premium: kCloud,
cheap: kOnDevice,
);
case PolicyMode.cascade:
throw ArgumentError('cascade has no RoutingStrategy; use modelFor');
}
}
void dispose() {
_ai = null;
_local = null;
_cloud = null;
_models.clear();
}
}
kOnDevice and kCloud are branch-key constants exported by genkit_hybrid itself — reuse them instead of inventing your own strings so every strategy agrees on the same keys.
For Step 4 we only need PolicyMode.cloud and PolicyMode.local: strategyFor maps each straight to a PreRoutingStrategy that always returns one key. smart, cascade, and budget are covered in Step 4.5 — strategyFor(PolicyMode.cascade) deliberately throws, because cascade isn't a RoutingStrategy at all; _buildModel builds it as a cascadeModel directly.
Three things in that file run ahead of this step, and are there so you never have to go back and edit it. AiEngine.forTest and initialize's localModelPath are test seams — they let test/ai_engine_policy_test.dart drive the real registration path without a 550 MB download. _withContextBudget is Step 6's, and Step 6 explains it; until RAG inflates the prompt it changes nothing you can observe.
The punchline: hybridModel() (and cascadeModel()) return an ordinary Genkit Model. Nothing downstream needs to know routing happened — the result composes with everything a normal model composes with: streaming, a RAG-augmented prompt, images. AiEngine.modelFor(mode) is a drop-in replacement for googleAI.gemini(...) or flutterGemma.model(...).
But an ordinary
Model
still has to be registered. genkit resolves generate(model: ...) (and generateStream) by reducing it to its .name and looking that name up in ai.registry — a Model you built but never registered fails every call with NOT_FOUND. That's why initialize() calls _registerPolicyModels() right after resolving _local/_cloud: it builds and registers all five policy composites once, up front, and modelFor just returns the already-registered instance from _models. Building a fresh hybridModel/cascadeModel inside modelFor itself — without registering it — is the one thing to avoid here.
In chat_screen.dart, replace the strategy toggle with a DropdownButton over all five PolicyMode values. There's nothing to hand-write per mode: the enum already knows its own label and its own prerequisites, so the item list is a loop and a new policy shows up in the picker the moment you add it to the enum.
Padding(
padding: const EdgeInsets.symmetric(horizontal: 8, vertical: 4),
child: DropdownButton<PolicyMode>(
value: _policy,
isExpanded: true,
items: [
for (final mode in PolicyMode.values)
DropdownMenuItem(
value: mode,
enabled: mode.availableWith(
cloud: _engine.cloudReady,
local: _engine.localReady,
),
child: Text(mode.label),
),
],
onChanged: (m) {
if (m != null) setState(() => _policy = m);
},
),
),
availableWith is what keeps a half-ready app honest: with no API key only Local is selectable, and until the on-device model finishes downloading only Cloud is.
_sendMessage() no longer picks between two services — it always calls the same Genkit, and lets AiEngine.modelFor(_policy) decide who answers:
final userMessage = Message(
role: Role.user,
content: [TextPart(text: text)],
);
// ...
// Captured before the call: genkit_hybrid doesn't report which branch
// actually ran, so this is a best-effort demo counter, not an exact
// count of cloud calls — see the accounting comment below.
final wasBudgetAvailable = _engine.budgetAvailable;
final stream = _engine.ai.generateStream(
model: _engine.modelFor(_policy),
messages: [userMessage],
);
await for (final chunk in stream) {
buffer.write(chunk.text);
// ... the throttled setState loop, unchanged from Step 2/3
}
// ...
// Best-effort demo counter for CostStrategy: genkit_hybrid exposes no
// "which branch ran" signal, so a Budget call that transiently fell
// back to on-device still counts here as spent; Budget stops climbing
// once the cap is hit either way.
if (_policy == PolicyMode.cloud) {
_engine.cloudCallsSpent++;
} else if (_policy == PolicyMode.budget && wasBudgetAvailable) {
_engine.cloudCallsSpent++;
}
text goes straight to the model here. In Step 6 RAG rewrites it first, and the local is renamed prompt to say so — the call itself does not change.
strategyFor treats them correctly, but nothing exercises their interesting behavior (an image, a bad on-device answer, a spent budget) until then.Three of the five PolicyMode values only get interesting once the app can send more than plain text, and once there's a signal to route on besides "the user picked cloud or local." This step adds image input, wires it through CapabilityStrategy, and walks through what smart, cascade, and budget actually decide.
# Image input (multimodal)
image_picker: ^1.2.3
Run flutter pub get.
In chat_screen.dart, add the picker and its state:
import 'dart:convert';
import 'dart:typed_data';
// ...
import 'package:image_picker/image_picker.dart';
// in _ChatScreenState:
final _picker = ImagePicker();
Uint8List? _attachedImage;
String? _attachedMime;
// ...
Future<void> _attachImage() async {
final XFile? picked = await _picker.pickImage(source: ImageSource.gallery);
if (picked == null) return;
final bytes = await picked.readAsBytes();
if (!mounted) return;
setState(() {
_attachedImage = bytes;
_attachedMime = picked.mimeType ?? 'image/jpeg';
});
}
Add an IconButton(icon: const Icon(Icons.image_outlined)) next to the send button that calls _attachImage, and a small thumbnail preview (Image.memory
_attachedImage != null._sendMessage() now builds a content list instead of a single TextPart:
final content = <Part>[TextPart(text: text)];
if (_attachedImage != null) {
final mime = _attachedMime ?? 'image/jpeg';
final dataUri = 'data:$mime;base64,${base64Encode(_attachedImage!)}';
// contentType MUST be set: the on-device plugin drops media without an
// image/* contentType, and CapabilityStrategy reads it to detect vision.
content.add(
MediaPart(
media: Media(contentType: mime, url: dataUri),
),
);
}
final userMessage = Message(role: Role.user, content: content);
contentType is the load-bearing detail: CapabilityStrategy (below) only recognizes a MediaPart as vision when its Media.contentType starts with image/ — an image_picker file with no MIME type falls back to 'image/jpeg' so it's never silently dropped.
Back in AiEngine.strategyFor, the three remaining cases:
case PolicyMode.smart:
// Image → cloud only (only it declares vision). Text → both qualify,
// cloud-first in `supports` insertion order, on-device as the tail.
// No WithFallback: CapabilityStrategy already yields the on-device
// tail for text, and for an image a forced on-device tail would hand
// the picture to a model that cannot see it. Offline + image should
// fail loudly, not silently degrade to text-only.
return CapabilityStrategy(
supports: {
kCloud: {ModelCapability.vision},
kOnDevice: <ModelCapability>{},
},
);
case PolicyMode.budget:
return CostStrategy(
budgetAvailable: () => budgetAvailable,
premium: kCloud,
cheap: kOnDevice,
);
CapabilityStrategy inspects the outgoing ModelRequest for media parts and keeps the branches that declare what the request needs. A text-only request needs nothing, so both branches qualify and it returns [kCloud, kOnDevice] (cloud-first, in supports' insertion order) — the on-device tail is already there, for free, and hybridModel uses it when the cloud call throws. An image request requires vision, which only kCloud declares, so it returns [kCloud] and nothing else: there is no second branch to fall to, by design._buildModel, not strategyFor: cascadeModel(branches: _branches, order: [kOnDevice, kCloud], accept: (r) => r.text.trim().length > 20). It tries the on-device model first; if the response passes accept (here, "longer than 20 characters") it's returned as-is, otherwise it escalates to cloud. cascadeModel is non-streaming internally — even through ai.generateStream, a cascade-routed request buffers the whole winning response and emits it as a single chunk.CostStrategy returns [premium, cheap] while budgetAvailable() is true, [cheap] once it isn't. AiEngine owns the budget itself: cloudCallsSpent is a plain int the app increments after every cloud/budget call (see the _sendMessage snippet above) and compares against budgetCap (3 by default). genkit_hybrid has no billing SDK — it only ever sees the resulting bool.A real cascade signal. That accept predicate is a deliberately crude demo proxy — "long enough" is a poor stand-in for "good enough" (a correct "Paris." escalates needlessly; 25 characters of nonsense passes). A production cascade escalates on model confidence, and the strongest training-free signal is the reply's average token log-probability — how surprised the model was by its own tokens; it decisively beats other zero-shot signals and holds up out-of-distribution. We fall back to a length check for one honest reason: the on-device runtime (LiteRT-LM) returns only decoded text, so accept never sees per-token probabilities. And resist the obvious shortcut of asking the model to score its own confidence — a ~1B model is badly calibrated and confidently wrong, which makes verbalized self-confidence a worse signal than the crude length check. (See Zero-Shot Confidence for Small LLMs and Do Small LMs Know When They're Wrong?.)
Smart + image on a cloud outage fails, and that's the point: with an attached image, CapabilityStrategy returns [kCloud] — the only branch that declares vision — and nothing behind it. Offline, that call throws and the error reaches the chat as Error: .... An earlier version of this app wrapped Smart in WithFallback(..., fallbackOrder: [kOnDevice]), which appended the on-device model as a tail for both shapes of request; the picture then went to a model that can't see it and got answered from the question's text alone — a confident, plausible reply about a photo nobody read. A wrong answer nobody flags is worse than a visible failure, so the wrapper is gone. Text is unaffected: CapabilityStrategy puts kOnDevice behind kCloud on its own, so an offline text question still lands on the phone.
Budget shares its counter with Cloud: cloudCallsSpent is one counter, not one per policy — a Cloud-mode send also spends the Budget allowance. Demo Cloud before Budget and the allowance may already be partly (or fully) spent by the time you switch.
An image can't reach a policy whose primary route is the text-only on-device model — which is exactly the textOnly flag each PolicyMode already carries, so there's no helper to write and no second list of modes to keep in sync. _sendMessage() checks it before doing anything else:
if (_attachedImage != null && _policy.textOnly) {
ScaffoldMessenger.of(context).showSnackBar(
const SnackBar(
content: Text(
"The on-device model can't see images — switch to Smart or Cloud.",
),
),
);
return;
}
local always starts on-device; cascade's first hop is always on-device too — both are declared textOnly: true and blocked. smart and cloud send an image to kCloud and nowhere else, so they're let through. budget is let through as well, but that's the one optimistic case: once cloudCallsSpent hits budgetCap, CostStrategy routes to [kOnDevice] only — textOnly is a fact about the policy, not about its current spend, so an image sent on a spent budget still isn't guaranteed a vision-capable branch.
CapabilityStrategy routes the vision request straight to kCloud).hybridModel moves on to kOnDevice, the tail CapabilityStrategy returned for a text request (slower, but it answers). Now attach an image and send that instead: this one fails with Error: ..., because a vision request has no on-device tail to move on to.cloudCallsSpent counts every completed Cloud- or Budget-mode call cumulatively for the whole session — including the Cloud-mode message from step 1 — so it may already be close to budgetCap (3) by the time you get here. Once it reaches the cap, routing flips to on-device only (CostStrategy sees budgetAvailable == false); a hot restart resets the counter to 0 if you want to watch it flip from a clean count.flutter test test/ai_engine_policy_test.dart exercises the same strategyFor routes and PolicyMode facts as a fast, deviceless unit test — useful to rerun after touching routing logic instead of redoing the whole runbook by hand.
An embedding turns text into a vector of numbers that captures semantic meaning. Similar texts have similar vectors. EmbeddingGemma 300M runs entirely on-device.
This already runs inside AiEngine.initialize(), in the optional block after the LLM install — a failure there disables RAG and never touches the chat:
await FlutterGemma.installEmbedder()
.modelFromNetwork(
_embeddingModelUrl,
token: _hfToken.isEmpty ? null : _hfToken,
)
.tokenizerFromNetwork(
_tokenizerUrl,
token: _hfToken.isEmpty ? null : _hfToken,
)
.install();
There's no second Genkit to build — AiEngine already declared the embedder back in Step 4, right next to the on-device model in the sameGenkitFlutterGemmaPlugin:
GenkitFlutterGemmaPlugin(
models: [
FlutterGemmaModelConfig(
name: kLocalModel,
modelType: ModelType.gemmaIt,
fileType: ModelFileType.litertlm,
),
],
embedders: downloadEmbedder
? [FlutterGemmaEmbedderConfig(name: kEmbedder)]
: const [],
),
So the one AiEngine Genkit already exposes the embedder — you just call ai.embed(...) on it (next).
final embeddings = await _ai.embed(
embedder: flutterGemma.embedder(_embedderName),
document: DocumentData(content: [TextPart(text: content)]),
);
embeddings.first.embedding is the List — 768 numbers for EmbeddingGemma 300M.
Create lib/services/rag_service.dart. In initialize(), loop over 10 city JSON files and embed each one. Store the List vectors in memory.
for (final city in _cityFiles) {
onStatus?.call('Embedding $city...');
final jsonString = await rootBundle.loadString(
'assets/tourist_data/$city.json',
);
final data = jsonDecode(jsonString) as Map<String, dynamic>;
final name = data['name'] as String? ?? city;
final content = _buildContent(data);
final embeddings = await _ai.embed(
embedder: flutterGemma.embedder(_embedderName),
document: DocumentData(content: [TextPart(text: content)]),
);
_store.add(
_VectorDocument(
id: city,
content: content,
city: name,
embedding: embeddings.first.embedding,
),
);
}
Key insight: ai.embed(embedder: ...) is model-agnostic. Replace flutterGemma.embedder(...) with any other registered embedder — the rest of the code stays the same.
RagService doesn't manage its own Genkit instance or model installation — it takes both from AiEngine, the same way _sendMessage does:
final rag = RagService(
ai: _engine.ai,
embedderName: _engine.embedderName,
);
await rag.initialize(
onStatus: (s) {
if (mounted) setState(() => _statusMessage = s);
},
);
_ragService = rag;
_ragReady = true;
When the user sends a query, embed it and find the closest city documents using cosine similarity. The two retrieval numbers get names — they're the knobs you'll actually reach for, and one of them decides how much text the on-device model has to swallow:
/// Minimum cosine similarity threshold for RAG retrieval results.
const double kMinSimilarity = 0.5;
/// Maximum number of documents to retrieve from the vector store.
const int kTopK = 3;
// ...
Future<RagResult> searchAndBuildContext(String query) async {
if (!_isInitialized) throw StateError('RagService not initialized');
final queryEmbeddings = await _ai.embed(
embedder: flutterGemma.embedder(_embedderName),
document: DocumentData(content: [TextPart(text: query)]),
);
final queryVector = queryEmbeddings.first.embedding;
final scored =
_store
.map(
(doc) => (doc: doc, score: _cosine(queryVector, doc.embedding)),
)
.where((r) => r.score >= kMinSimilarity)
.toList()
..sort((a, b) => b.score.compareTo(a.score));
final topK = scored.take(kTopK).toList();
if (topK.isEmpty) {
return RagResult(
augmentedPrompt: query,
retrievedContext: '',
sources: [],
);
}
final context = topK.map((r) => r.doc.content).join('\n\n');
final sources = topK
.map((r) => '${r.doc.city} (${(r.score * 100).toStringAsFixed(0)}%)')
.toList();
final augmentedPrompt =
'Based on the following travel information:\n\n$context\n\n'
'Answer the question: $query';
return RagResult(
augmentedPrompt: augmentedPrompt,
retrievedContext: context,
sources: sources,
);
}
_cosine throws an ArgumentError when the two vectors' lengths disagree rather than asserting it — an assert is compiled out of a release build, which is precisely where a mismatched embedder would otherwise return silent nonsense.
kTopK is the first knob in this codelab that can break a working app. Retrieval doesn't just improve the prompt, it inflates it — and the on-device branch is answering inside a fixed window.
maxTokens
is the whole context window, not the reply length. It is the KV-cache budget: everything the model reads plus everything it writes. genkit_flutter_gemma defaults it to 1024, which is fine for the chat we've had so far and not fine the moment RAG lands. Three city guides at ~600 tokens each is ~1.7k of prompt before the model has said a word, so the first RAG question on Local dies during prefill with Input token ids are too long ... 1713 >= 1024 — a native error, not a bad answer. The fix isn't a smaller kTopK: the bundled Gemma 3 1B was built for a 4096-token window, which is what the ekv4096 in its file name means. Give the on-device branch the window it already has.
Back in AiEngine, wrap the resolved on-device model so every request it receives carries that budget:
/// Context window for the on-device branch, in tokens. `maxTokens` is the
/// WHOLE window (input + output) and genkit_flutter_gemma defaults it to 1024.
/// RagService's take(3) of ~600-token city guides alone is ~1.7k tokens —
/// measured on device: "Input token ids are too long ... 1713 >= 1024". The
/// bundled Gemma-3-1B `.litertlm` is built for 4096 (`ekv4096`), so use it.
const kOnDeviceContextTokens = 4096;
// ... in initialize(), where the on-device model is resolved:
_local = _withContextBudget(
await _resolve(flutterGemma.model(kLocalModel)),
);
// ...
/// Wraps the on-device [inner] model so every request reaching it carries a
/// context window big enough for the RAG prompt. genkit_flutter_gemma reads
/// `maxTokens` ONLY from the per-request `request.config` (defaulting to
/// 1024) — registration-time [FlutterGemmaModelConfig] has no options field
/// — so the budget has to ride along with each request. genkit_hybrid calls
/// a branch as `branch.fn(request, context)`, so forwarding the same
/// `context` leaves streaming and fallback untouched. Only the on-device
/// branch is wrapped: Gemini's config has no `maxTokens` key. An explicit
/// request `maxTokens` wins. The request is COPIED, never mutated — cascade
/// hands the very same object to the cloud branch next. [inner]'s metadata
/// is forwarded as a COPY too: genkit's `Model` constructor writes into the
/// map it is handed, so passing `inner.metadata` itself would rewrite the
/// wrapped model's own metadata.
Model _withContextBudget(Model inner) => Model(
name: '${inner.name}/ctx',
metadata: {...inner.metadata},
fn: (request, context) {
if (request == null || request.config?['maxTokens'] != null) {
return inner.fn(request, context);
}
final budgeted = ModelRequest.fromJson({
...request.toJson(),
'config': {...?request.config, 'maxTokens': kOnDeviceContextTokens},
});
return inner.fn(budgeted, context);
},
);
The cloud branch is untouched — Gemini's window is orders of magnitude larger, and the wrapper only ever sees requests already routed on-device.
In chat_screen.dart:
Switch in the AppBar to toggle RAG_sendMessage(), if RAG is enabled call searchAndBuildContext(text) before generatingragResult.sources in a banner below the AppBarTry these queries:
mode.availableWith(...) in the pickerTextField.onSubmitted alike, on the same condition — nothing in flight, and at least one branch readyCapability | Technology |
Cloud inference |
|
On-device inference |
|
Hybrid routing |
|
Multimodal input |
|
On-device embeddings |
|
RAG pipeline | Genkit |
The old approach needed two completely different APIs — Firebase AI Logic for cloud and raw flutter_gemma calls for local. With Genkit:
// Both use the same API — only model: changes
ai.generateStream(model: googleAI.gemini('gemini-3.7-flash'), prompt: prompt)
ai.generateStream(model: flutterGemma.model('gemma-3-1b-it'), prompt: prompt)
ai.embed(embedder: flutterGemma.embedder('embedding-gemma-300m'), document: ...)
// genkit_hybrid composes both into one routable model — still the same call:
ai.generateStream(model: engine.modelFor(policy), messages: [userMessage])
Routing, fallback, observability, and tool calling all work the same way regardless of which model backend you use.
drift so you don't re-embed on every cold startgemini-3.7-flash for gemini-3.7-pro for complex queries, or add a second on-device model for specialized tasksdefineFlow to add observability, retries, and structured output