Melange
LLM Inference

LLM Inference Overview

Run large language models on-device with ZETIC Melange.

The Android and iOS examples target the planned ZeticMLange 1.13.0 release. The Flutter package has its own version; consult its platform-specific API reference.

ZeticMLangeLLMModel runs text generation on-device and exposes token streaming through waitForNextToken(). The public LLM surface also includes function calling, composition-based RAG, vision-language image response, and KV state persistence on Android, iOS, and Flutter.

Load A Model

Use a model name from the Melange Dashboard or a supported public Hugging Face model name.

val model = ZeticMLangeLLMModel(
    context = context,
    personalKey = PERSONAL_KEY,
    name = MODEL_NAME,
    modelMode = LLMModelMode.RUN_AUTO,
    initOption = LLMInitOption(nCtx = 2048),
)
let model = try await ZeticMLangeLLMModel(
  personalKey: PERSONAL_KEY,
  name: MODEL_NAME,
  modelMode: .RUN_AUTO,
  initOption: LLMInitOption(nCtx: 2048)
)
final model = await ZeticMLangeLLMModel.create(
  personalKey: personalKey,
  name: modelName,
  modelMode: LLMModelMode.runAuto,
  initOption: const LLMInitOption(nCtx: 2048),
);

Generate Text

For models that support the token API, run(...) starts generation. Read tokens with waitForNextToken() until the stream is finished. On Android and iOS, session-backed models use respond(...) instead; see the platform-specific API reference for the selected model.

model.run("What is on-device AI?")

val output = StringBuilder()
while (true) {
    val next = model.waitForNextToken()
    if (next.isFinal || next.token.isEmpty()) break
    output.append(next.token)
}
try model.run("What is on-device AI?")

var output = ""
while true {
  let next = model.waitForNextToken()
  if next.isFinished { break }
  output.append(next.token)
}
model.run('What is on-device AI?');

final output = StringBuffer();
while (true) {
  final next = model.waitForNextToken();
  if (next.isFinished) break;
  output.write(next.token);
}

Selection Controls

Use modelMode to select the automatic strategy.

OptionPurpose
modelModeSelects automatic strategy: auto, speed, or accuracy.
initOption.nCtxRequests the context size. The runtime may normalize it.
cacheHandlingPolicyControls downloaded model artifact cleanup on disk.

Capabilities

Quick Start Templates

Next Steps

On this page