ZeticMLangeLLMModel
API reference for running LLM inference on Android.
This page describes the API planned for ZeticMLange Android 1.13.0.
ZeticMLangeLLMModel loads an on-device LLM from the Melange registry. Available generation, tools, media input, and speech output features depend on the loaded package and backend.
Import
import com.zeticai.mlange.core.model.llm.ZeticMLangeLLMModelConstructor
ZeticMLangeLLMModel(
context: Context,
personalKey: String,
name: String,
version: Int? = null,
modelMode: LLMModelMode = LLMModelMode.RUN_AUTO,
cacheHandlingPolicy: ModelCacheHandlingPolicy = ModelCacheHandlingPolicy.REMOVE_OVERLAPPING,
initOption: LLMInitOption = LLMInitOption(),
onDownload: ((Float) -> Unit)? = null,
)| Parameter | Type | Default | Description |
|---|---|---|---|
context | Context | - | Android context used for cache and file access. |
personalKey | String | - | Personal key for accessing the model. |
name | String | - | Model name in account_name/project_name format. |
version | Int? | null | Model version. null loads the latest version. |
modelMode | LLMModelMode | RUN_AUTO | Backend selection strategy. |
cacheHandlingPolicy | ModelCacheHandlingPolicy | REMOVE_OVERLAPPING | Managed artifact cache cleanup policy. |
initOption | LLMInitOption | LLMInitOption() | LLM initialization options. |
onDownload | ((Float) -> Unit)? | null | Download progress callback from 0.0 to 1.0. |
val model = ZeticMLangeLLMModel(
context = context,
personalKey = PERSONAL_KEY,
name = "account_name/project_name",
initOption = LLMInitOption(nCtx = 2048),
)This value is an example, not a guarantee that every model supports the same context size. The context includes input and generated tokens. Model loading can perform network and file I/O: call it off the UI thread.
Text Generation
run(text)
Starts generation for a prompt. Read generated tokens with waitForNextToken(). Throws ZeticMLangeException if generation cannot start.
@Throws(ZeticMLangeException::class)
fun run(text: String)model.run("Explain on-device AI in one paragraph.")waitForNextToken()
Waits for the next generated token.
fun waitForNextToken(): LLMNextTokenResultwhile (true) {
val next = model.waitForNextToken()
if (next.isFinal || next.token.isEmpty()) break
append(next.token)
}Media Sessions
Media-capable packages use the following session API. Inspect model.capabilities before sending a modality or requesting speech. A text-only package reports LLMCapabilities.NONE; use its run, generate, or runWithTools API instead. Calling respond on it fails with NOT_IMPLEMENTED.
fun respond(
systemPrompt: String = "",
userText: String,
media: List<LLMMedia> = emptyList(),
): Flow<String>
fun respondWithSpeech(
systemPrompt: String = "",
userText: String,
media: List<LLMMedia> = emptyList(),
reply: LLMSpeechReply = LLMSpeechReply.SPEECH,
): Flow<LLMResponseEvent>Both flows are cold: a turn starts when collected. Turns are serialized. Collection cancellation or model.cancel() stops submitted turns; reset and close wait for active native work. A turn without text or media fails. Native response errors use ZeticMLangeLLMResponseException and preserve a status such as INVALID_INPUT, INVALID_STATE, or NOT_IMPLEMENTED. After INVALID_STATE, reset the session and resend the media required for the new turn.
Image input
import com.zeticai.mlange.core.model.llm.LLMMedia
val image = LLMMedia.Image.fromBitmap(bitmap)
model.respond(userText = "What is in this image?", media = listOf(image))
.collect { piece -> append(piece) }Alternatively construct LLMMedia.Image(rgb, width, height) with tightly packed RGB bytes. The array length must equal width * height * 3; the input is copied.
Audio input and speech output
import com.zeticai.mlange.core.model.llm.LLMMedia
import com.zeticai.mlange.core.model.llm.LLMResponseEvent
import com.zeticai.mlange.core.model.llm.LLMSpeechReply
val audio = LLMMedia.Audio(samples = monoPcm16kHz)
model.respondWithSpeech(
userText = "Respond to the recording.",
media = listOf(audio),
reply = LLMSpeechReply.TEXT_AND_SPEECH,
).collect { event ->
when (event) {
is LLMResponseEvent.Text -> append(event.piece)
is LLMResponseEvent.Speech -> playPcm(event.pcm, event.sampleRate)
}
}Input PCM is nonempty mono FloatArray at 16 kHz with finite samples in [-1, 1]. LLMMedia.Audio.from(...) averages interleaved channels and resamples; fromPcm16(...) accepts signed 16-bit samples. Output is float PCM; use the sample rate carried by each speech event, which may differ from the input rate. Speech requires capabilities.speechOutput.
Video and capability limits
LLMMedia.Video represents frames sampled by the app, source frame indices, presentation times, and metadata FPS. It does not decode a video file. Current Android session backends do not accept video: capabilities.video is false and sending video returns NOT_IMPLEMENTED. Image, audio, multiple inputs, and speech support also depend on the loaded backend; check the corresponding capabilities rather than assuming that support on another platform implies support on Android.
LLMInitOption.nCtx controls the context window where supported. kvCacheCleanupPolicy controls behavior when the conversation fills the window. With CLEAN_UP_ON_FULL, a response can end early when the context fills; some no-room turns are reset and retried once. With DO_NOT_CLEAN_UP, the failing conversation is kept and media turns may require resetSession() before continuing.
Function Calling
var functionCallingSystemPrompt: String?
fun registerTool(spec: LLMToolSpec, executor: LLMToolExecutor)
fun unregisterTool(name: String): Boolean
fun clearTools()
fun registeredTools(): List<LLMToolSpec>
fun runWithTools(text: String): Flow<String>After registering a tool, call runWithTools(...) and collect its Flow. run(...) is only available when no tools are registered.
Tool execution is not supported by media sessions.
model.registerTool(
LLMToolSpec(
name = "lookup",
description = "Look up local app data.",
parametersJson = """{"type":"object","properties":{"query":{"type":"string"}}}""",
),
) { call ->
LLMToolResult(content = """{"result":"Found"}""")
}
model.runWithTools("Use lookup to answer the question.").collect { token ->
append(token)
}KV State Persistence
fun saveKVState(path: String)
fun loadKVState(path: String)
fun resetKVState()Use these APIs to persist or reset the current LLM state for resume flows.
Background Downloads
Schedule a durable LLM artifact download without initializing the LLM runtime. Keep the returned handle to observe or stop the job after the app moves to the background.
fun downloadInBackground(
context: Context,
personalKey: String,
name: String,
version: Int? = null,
modelMode: LLMModelMode = LLMModelMode.RUN_AUTO,
cacheHandlingPolicy: ModelCacheHandlingPolicy = ModelCacheHandlingPolicy.KEEP_EXISTING,
): BackgroundDownloadHandle
fun getBackgroundDownloadStatus(
context: Context,
handle: BackgroundDownloadHandle,
): BackgroundDownloadStatus
fun stopBackgroundDownload(
context: Context,
handle: BackgroundDownloadHandle,
): BackgroundDownloadStopResult
fun removeDownloadedModel(
context: Context,
name: String,
version: Int? = null,
): ModelRemovalResultval handle = ZeticMLangeLLMModel.downloadInBackground(
context = context,
personalKey = PERSONAL_KEY,
name = "account_name/project_name",
)
val status = ZeticMLangeLLMModel.getBackgroundDownloadStatus(context, handle)
val stopResult = ZeticMLangeLLMModel.stopBackgroundDownload(context, handle)
val removal = ZeticMLangeLLMModel.removeDownloadedModel(
context = context,
name = "account_name/project_name",
)Stopping preserves an already installed model. Close active LLM instances and retry removal when removal.isInUse is true. See Cache Management for states and result values.
Lifecycle
val isClosed: Boolean
fun cleanUp()
fun resetSession()
fun close()Call cleanUp() or resetSession() before starting a fresh conversation. Call close() when the model is no longer needed.