ZeticMLangeLLMModel
API reference for running LLM inference in Flutter.
This page reflects zetic_mlange 1.13.1.
ZeticMLangeLLMModel loads an on-device LLM and supports text generation, token streaming, function calling, image response for LFM-VL models, and KV state persistence.
Import
import 'package:zetic_mlange/zetic_mlange.dart';create
static Future<ZeticMLangeLLMModel> create({
required String personalKey,
required String name,
int? version,
LLMModelMode modelMode = LLMModelMode.runAuto,
ModelCacheHandlingPolicy cacheHandlingPolicy =
ModelCacheHandlingPolicy.removeOverlapping,
LLMInitOption? initOption,
LLMKVCacheCleanupPolicy kvCacheCleanupPolicy =
LLMKVCacheCleanupPolicy.cleanUpOnFull,
MlangeProgressCallback? onDownload,
})| Parameter | Type | Default | Description |
|---|---|---|---|
personalKey | String | - | Personal key for accessing the model. |
name | String | - | Model name in account_name/project_name format. |
version | int? | null | Model version. null loads the latest version. |
modelMode | LLMModelMode | runAuto | Backend selection strategy. |
cacheHandlingPolicy | ModelCacheHandlingPolicy | removeOverlapping | Managed artifact cache cleanup policy. |
initOption | LLMInitOption? | null | LLM initialization options. |
kvCacheCleanupPolicy | LLMKVCacheCleanupPolicy | cleanUpOnFull | Convenience default used when initOption is omitted. |
onDownload | MlangeProgressCallback? | null | Download progress callback from 0.0 to 1.0. |
final model = await ZeticMLangeLLMModel.create(
personalKey: personalKey,
name: 'account_name/project_name',
initOption: const LLMInitOption(nCtx: 4096),
);Text Generation
run
Starts generation for a prompt. Read generated tokens with waitForNextToken(). Throws MlangeException if generation cannot start.
void run(String text)model.run('Explain on-device AI in one paragraph.');Reading Generated Tokens
Waits for the next generated token.
LLMNextTokenResult waitForNextToken()while (true) {
final next = model.waitForNextToken();
if (next.isFinished) break;
append(next.token);
}Media Sessions
Use capabilities() to check the loaded model before sending media or requesting speech. Support varies by model and platform, and individual capabilities do not guarantee that every combination is supported. Text-only models can use run or runWithTools.
final caps = await model.capabilities();
if (caps.text) {
await for (final text in model.respondStream(userText: 'Explain on-device AI.')) {
append(text);
}
}LLMCapabilities reports text, image, audio, video, speechOutput, mediaInFollowupTurns, multiTurn, maxMediaPerTurn, and speechSampleRate. Consume a response stream before starting another operation. Cancel its subscription or call await model.cancel() to stop a response. await model.resetSession() cancels active work and clears the conversation; await model.close() releases the model.
Image and audio input
final caps = await model.capabilities();
if (caps.image) {
final image = LLMImage(rgb: rgbBytes, width: width, height: height);
await for (final text in model.respondStream(
userText: 'What is in this image?',
media: [image],
)) {
append(text);
}
}
if (caps.audio) {
final audio = LLMAudio(samples: monoPcm16kHz);
await for (final text in model.respondStream(
userText: 'Transcribe this speech.',
media: [audio],
)) {
append(text);
}
}Import dart:typed_data for Uint8List RGB bytes and Float32List samples. Images use packed RGB with exactly width * height * 3 bytes. Audio is nonempty mono float PCM at 16 kHz with finite samples in [-1, 1].
Video input
The app decodes and samples video into ZeticMLangeLLMImage RGB frames; the SDK does not accept a video file. Supply source frame indices, actual presentation times in seconds, and metadata FPS. Frames must have matching dimensions and ordered indices and times. Preserve actual timestamps for variable-frame-rate video.
final video = ZeticMLangeLLMVideo(
frames: frames,
sourceFrameIndices: sourceFrameIndices,
presentationTimes: presentationTimes,
metadataFps: metadataFps,
);
if (caps.video) {
await for (final text in model.respondStream(
userText: 'Describe this video.',
video: video,
)) {
append(text);
}
}The optimized iOS LFM3 path samples at 2 fps with at least four frames, and accepts at most ten frames or 192 MiB of RGB data; context and output budgets can reduce the usable count. Current Android session backends do not accept video. Check video on the loaded model. Some models accept separate audio with video; it is an independent input, not a synchronized soundtrack. Image plus video is not supported on this iOS path.
Speech output
if (caps.speechOutput) {
await for (final event in model.respondWithSpeech(
userText: 'Say hello.',
reply: LLMSpeechReply.textAndSpeech,
)) {
if (event is LLMTextEvent) {
append(event.text);
} else if (event is LLMSpeechEvent) {
playPcm(event.samples, caps.speechSampleRate);
}
}
}Use LLMSpeechReply.speech for speech only or textAndSpeech for both event types. Speech events contain float PCM at caps.speechSampleRate; this output rate is independent of the 16 kHz input rate.
Vision-Language Response
Use respond(...) with an LFM-VL-capable model.
Future<String> respond({
String systemPrompt = '',
required String userText,
required ZeticMLangeLLMImage image,
})final image = ZeticMLangeLLMImage(
rgb: rgbBytes,
width: width,
height: height,
);
final response = await model.respond(
systemPrompt: 'Answer briefly.',
userText: 'What is in this image?',
image: image,
);Function Calling
String? functionCallingSystemPrompt
typedef LLMToolExecutor =
Future<LLMToolResult> Function(LLMToolCall call)
void registerTool(LLMToolSpec spec, LLMToolExecutor executor)
bool unregisterTool(String name)
void clearTools()
List<LLMToolSpec> registeredTools()
Stream<String> runWithTools(String text)model.registerTool(
const LLMToolSpec(
name: 'lookup',
description: 'Look up local app data.',
parametersJson: '{"type":"object","properties":{"query":{"type":"string"}}}',
),
(call) async => const LLMToolResult(content: '{"result":"Found"}'),
);
await for (final token in model.runWithTools(
'Use lookup to answer the question.',
)) {
append(token);
}runWithTools returns a Stream<String>. Consume it to completion before starting another model operation. If an executor fails, the SDK returns a JSON error tool result to the model rather than emitting the executor failure through the stream.
Cancelling the stream ends the SDK request and ignores a later executor completion. It does not cancel a Future already running in your app.
KV State Persistence
void saveKVState(String path)
void loadKVState(String path)
void resetKVState()Use these APIs to persist or reset the current LLM state for resume flows.
Non-OK native statuses throw MlangeKVException. Calling these operations on a local LFM-VL model throws MlangeException.
Background Downloads
Schedule an LLM artifact download without initializing the LLM runtime. Keep the returned handle to observe or stop the job after the app moves to the background.
static Future<BackgroundDownloadHandle> downloadInBackground({
required String personalKey,
required String name,
int? version,
LLMModelMode modelMode = LLMModelMode.runAuto,
ModelCacheHandlingPolicy cacheHandlingPolicy =
ModelCacheHandlingPolicy.keepExisting,
})
static Future<BackgroundDownloadStatus> getBackgroundDownloadStatus(
BackgroundDownloadHandle handle,
)
static Future<BackgroundDownloadStopResult> stopBackgroundDownload(
BackgroundDownloadHandle handle,
)
static Future<ModelRemovalResult> removeDownloadedModel({
required String name,
int? version,
})final handle = await ZeticMLangeLLMModel.downloadInBackground(
personalKey: personalKey,
name: 'account_name/project_name',
);
final status = await ZeticMLangeLLMModel.getBackgroundDownloadStatus(handle);
final stopResult = await ZeticMLangeLLMModel.stopBackgroundDownload(handle);
final removal = await ZeticMLangeLLMModel.removeDownloadedModel(
name: 'account_name/project_name',
);On iOS, await downloadInBackground before the host app enters the background and forward background URL-session events from AppDelegate. See Flutter advanced configuration. Stopping preserves an already installed model; if removal.isInUse is true, close active LLM instances and retry removal.
Lifecycle
bool get isClosed
void cleanUp()
Future<void> resetSession()
Future<void> close()Call cleanUp() or resetSession() before starting a fresh conversation. Call close() when the model is no longer needed.