ZeticMLangeLLMModel

This page reflects zetic_mlange 1.11.1.

ZeticMLangeLLMModel loads an on-device LLM and supports text generation, token streaming, function calling, image response for LFM-VL models, and KV state persistence.

Import

import 'package:zetic_mlange/zetic_mlange.dart';

`create`

static Future<ZeticMLangeLLMModel> create({
  required String personalKey,
  required String name,
  int? version,
  LLMModelMode modelMode = LLMModelMode.runAuto,
  ModelCacheHandlingPolicy cacheHandlingPolicy =
      ModelCacheHandlingPolicy.removeOverlapping,
  LLMInitOption? initOption,
  LLMKVCacheCleanupPolicy kvCacheCleanupPolicy =
      LLMKVCacheCleanupPolicy.cleanUpOnFull,
  MlangeProgressCallback? onDownload,
})

Parameter	Type	Default	Description
`personalKey`	`String`	-	Personal key for accessing the model.
`name`	`String`	-	Model name in `account_name/project_name` format.
`version`	`int?`	`null`	Model version. `null` loads the latest version.
`modelMode`	`LLMModelMode`	`runAuto`	Backend selection strategy.
`cacheHandlingPolicy`	`ModelCacheHandlingPolicy`	`removeOverlapping`	Managed artifact cache cleanup policy.
`initOption`	`LLMInitOption?`	`null`	LLM initialization options.
`kvCacheCleanupPolicy`	`LLMKVCacheCleanupPolicy`	`cleanUpOnFull`	Convenience default used when `initOption` is omitted.
`onDownload`	`MlangeProgressCallback?`	`null`	Download progress callback from `0.0` to `1.0`.

final model = await ZeticMLangeLLMModel.create(
  personalKey: personalKey,
  name: 'account_name/project_name',
  initOption: const LLMInitOption(nCtx: 4096),
);

Text Generation

`run`

Starts generation for a prompt. Read generated tokens with waitForNextToken(). Throws MlangeException if generation cannot start.

void run(String text)

model.run('Explain on-device AI in one paragraph.');

Reading Generated Tokens

Waits for the next generated token.

LLMNextTokenResult waitForNextToken()

while (true) {
  final next = model.waitForNextToken();
  if (next.isFinished) break;
  append(next.token);
}

Vision-Language Response

Use respond(...) with an LFM-VL-capable model.

Future<String> respond({
  String systemPrompt = '',
  required String userText,
  required ZeticMLangeLLMImage image,
})

final image = ZeticMLangeLLMImage(
  rgb: rgbBytes,
  width: width,
  height: height,
);

final response = await model.respond(
  systemPrompt: 'Answer briefly.',
  userText: 'What is in this image?',
  image: image,
);

Function Calling

String? functionCallingSystemPrompt

typedef LLMToolExecutor =
    Future<LLMToolResult> Function(LLMToolCall call)

void registerTool(LLMToolSpec spec, LLMToolExecutor executor)
bool unregisterTool(String name)
void clearTools()
List<LLMToolSpec> registeredTools()
Stream<String> runWithTools(String text)

model.registerTool(
  const LLMToolSpec(
    name: 'lookup',
    description: 'Look up local app data.',
    parametersJson: '{"type":"object","properties":{"query":{"type":"string"}}}',
  ),
  (call) async => const LLMToolResult(content: '{"result":"Found"}'),
);

await for (final token in model.runWithTools(
  'Use lookup to answer the question.',
)) {
  append(token);
}

runWithTools returns a Stream<String>. Consume it to completion before starting another model operation. If an executor fails, the SDK returns a JSON error tool result to the model rather than emitting the executor failure through the stream.

Cancelling the stream ends the SDK request and ignores a later executor completion. It does not cancel a Future already running in your app.

KV State Persistence

void saveKVState(String path)
void loadKVState(String path)
void resetKVState()

Use these APIs to persist or reset the current LLM state for resume flows. Non-OK native statuses throw MlangeKVException. Calling these operations on a local LFM-VL model throws MlangeException.

Background Downloads

Schedule an LLM artifact download without initializing the LLM runtime. Keep the returned handle to observe or stop the job after the app moves to the background.

static Future<BackgroundDownloadHandle> downloadInBackground({
  required String personalKey,
  required String name,
  int? version,
  LLMModelMode modelMode = LLMModelMode.runAuto,
  ModelCacheHandlingPolicy cacheHandlingPolicy =
      ModelCacheHandlingPolicy.keepExisting,
})

static Future<BackgroundDownloadStatus> getBackgroundDownloadStatus(
  BackgroundDownloadHandle handle,
)

static Future<BackgroundDownloadStopResult> stopBackgroundDownload(
  BackgroundDownloadHandle handle,
)

static Future<ModelRemovalResult> removeDownloadedModel({
  required String name,
  int? version,
})

final handle = await ZeticMLangeLLMModel.downloadInBackground(
  personalKey: personalKey,
  name: 'account_name/project_name',
);

final status = await ZeticMLangeLLMModel.getBackgroundDownloadStatus(handle);
final stopResult = await ZeticMLangeLLMModel.stopBackgroundDownload(handle);
final removal = await ZeticMLangeLLMModel.removeDownloadedModel(
  name: 'account_name/project_name',
);

On iOS, await downloadInBackground before the host app enters the background and forward background URL-session events from AppDelegate. See Flutter advanced configuration. Stopping preserves an already installed model; if removal.isInUse is true, close active LLM instances and retry removal.

Lifecycle

bool get isClosed
void cleanUp()
Future<void> resetSession()
Future<void> close()

Call cleanUp() or resetSession() before starting a fresh conversation. Call close() when the model is no longer needed.

ZeticMLangeLLMModel

On this page