LLM Inference Overview
Run large language models on-device with ZETIC Melange.
Examples on this page reflect ZeticMLange Android 1.10.0, ZeticMLange iOS 1.10.0, and zetic_mlange 1.10.0.
ZeticMLangeLLMModel runs text generation on-device and exposes token streaming through waitForNextToken(). The public LLM surface also includes function calling, composition-based RAG, vision-language image response, and KV state persistence on native Android/iOS.
Load A Model
Use a model name from the Melange Dashboard or a supported public Hugging Face model name.
val model = ZeticMLangeLLMModel(
context = context,
personalKey = PERSONAL_KEY,
name = MODEL_NAME,
modelMode = LLMModelMode.RUN_AUTO,
initOption = LLMInitOption(nCtx = 4096),
)let model = try await ZeticMLangeLLMModel(
personalKey: PERSONAL_KEY,
name: MODEL_NAME,
modelMode: .RUN_AUTO,
initOption: LLMInitOption(nCtx: 4096)
)final model = await ZeticMLangeLLMModel.create(
personalKey: personalKey,
name: modelName,
modelMode: LLMModelMode.runAuto,
initOption: const LLMInitOption(nCtx: 4096),
);Generate Text
run(...) starts generation. Read tokens with waitForNextToken() until the stream is finished.
model.run("What is on-device AI?")
val output = StringBuilder()
while (true) {
val next = model.waitForNextToken()
if (next.isFinal || next.token.isEmpty()) break
output.append(next.token)
}try model.run("What is on-device AI?")
var output = ""
while true {
let next = model.waitForNextToken()
if next.isFinished { break }
output.append(next.token)
}model.run('What is on-device AI?');
final output = StringBuffer();
while (true) {
final next = model.waitForNextToken();
if (next.isFinished) break;
output.write(next.token);
}Selection Controls
Use modelMode to select the automatic strategy.
| Option | Purpose |
|---|---|
modelMode | Selects automatic strategy: auto, speed, or accuracy. |
initOption.nCtx | Requests the context size. The runtime may normalize it. |
cacheHandlingPolicy | Controls downloaded model artifact cleanup on disk. |
1.10.0 Capabilities
Function Calling
Connect model output to app-defined tools.
RAG
Ground generation with retrieved local context.
Vision-Language
Ask image + text questions with LFM-VL-capable models.