Melange
API ReferenceAndroid

ZeticMLangeLLMModel

API reference for running LLM inference on Android.

This page describes the API planned for ZeticMLange Android 1.13.0.

ZeticMLangeLLMModel loads an on-device LLM from the Melange registry. Available generation, tools, media input, and speech output features depend on the loaded package and backend.

Import

import com.zeticai.mlange.core.model.llm.ZeticMLangeLLMModel

Constructor

ZeticMLangeLLMModel(
    context: Context,
    personalKey: String,
    name: String,
    version: Int? = null,
    modelMode: LLMModelMode = LLMModelMode.RUN_AUTO,
    cacheHandlingPolicy: ModelCacheHandlingPolicy = ModelCacheHandlingPolicy.REMOVE_OVERLAPPING,
    initOption: LLMInitOption = LLMInitOption(),
    onDownload: ((Float) -> Unit)? = null,
)
ParameterTypeDefaultDescription
contextContext-Android context used for cache and file access.
personalKeyString-Personal key for accessing the model.
nameString-Model name in account_name/project_name format.
versionInt?nullModel version. null loads the latest version.
modelModeLLMModelModeRUN_AUTOBackend selection strategy.
cacheHandlingPolicyModelCacheHandlingPolicyREMOVE_OVERLAPPINGManaged artifact cache cleanup policy.
initOptionLLMInitOptionLLMInitOption()LLM initialization options.
onDownload((Float) -> Unit)?nullDownload progress callback from 0.0 to 1.0.
val model = ZeticMLangeLLMModel(
    context = context,
    personalKey = PERSONAL_KEY,
    name = "account_name/project_name",
    initOption = LLMInitOption(nCtx = 2048),
)

This value is an example, not a guarantee that every model supports the same context size. The context includes input and generated tokens. Model loading can perform network and file I/O: call it off the UI thread.

Text Generation

run(text)

Starts generation for a prompt. Read generated tokens with waitForNextToken(). Throws ZeticMLangeException if generation cannot start.

@Throws(ZeticMLangeException::class)
fun run(text: String)
model.run("Explain on-device AI in one paragraph.")

waitForNextToken()

Waits for the next generated token.

fun waitForNextToken(): LLMNextTokenResult
while (true) {
    val next = model.waitForNextToken()
    if (next.isFinal || next.token.isEmpty()) break
    append(next.token)
}

Media Sessions

Media-capable packages use the following session API. Inspect model.capabilities before sending a modality or requesting speech. A text-only package reports LLMCapabilities.NONE; use its run, generate, or runWithTools API instead. Calling respond on it fails with NOT_IMPLEMENTED.

fun respond(
    systemPrompt: String = "",
    userText: String,
    media: List<LLMMedia> = emptyList(),
): Flow<String>

fun respondWithSpeech(
    systemPrompt: String = "",
    userText: String,
    media: List<LLMMedia> = emptyList(),
    reply: LLMSpeechReply = LLMSpeechReply.SPEECH,
): Flow<LLMResponseEvent>

Both flows are cold: a turn starts when collected. Turns are serialized. Collection cancellation or model.cancel() stops submitted turns; reset and close wait for active native work. A turn without text or media fails. Native response errors use ZeticMLangeLLMResponseException and preserve a status such as INVALID_INPUT, INVALID_STATE, or NOT_IMPLEMENTED. After INVALID_STATE, reset the session and resend the media required for the new turn.

Image input

import com.zeticai.mlange.core.model.llm.LLMMedia

val image = LLMMedia.Image.fromBitmap(bitmap)
model.respond(userText = "What is in this image?", media = listOf(image))
    .collect { piece -> append(piece) }

Alternatively construct LLMMedia.Image(rgb, width, height) with tightly packed RGB bytes. The array length must equal width * height * 3; the input is copied.

Audio input and speech output

import com.zeticai.mlange.core.model.llm.LLMMedia
import com.zeticai.mlange.core.model.llm.LLMResponseEvent
import com.zeticai.mlange.core.model.llm.LLMSpeechReply

val audio = LLMMedia.Audio(samples = monoPcm16kHz)
model.respondWithSpeech(
    userText = "Respond to the recording.",
    media = listOf(audio),
    reply = LLMSpeechReply.TEXT_AND_SPEECH,
).collect { event ->
    when (event) {
        is LLMResponseEvent.Text -> append(event.piece)
        is LLMResponseEvent.Speech -> playPcm(event.pcm, event.sampleRate)
    }
}

Input PCM is nonempty mono FloatArray at 16 kHz with finite samples in [-1, 1]. LLMMedia.Audio.from(...) averages interleaved channels and resamples; fromPcm16(...) accepts signed 16-bit samples. Output is float PCM; use the sample rate carried by each speech event, which may differ from the input rate. Speech requires capabilities.speechOutput.

Video and capability limits

LLMMedia.Video represents frames sampled by the app, source frame indices, presentation times, and metadata FPS. It does not decode a video file. Current Android session backends do not accept video: capabilities.video is false and sending video returns NOT_IMPLEMENTED. Image, audio, multiple inputs, and speech support also depend on the loaded backend; check the corresponding capabilities rather than assuming that support on another platform implies support on Android.

LLMInitOption.nCtx controls the context window where supported. kvCacheCleanupPolicy controls behavior when the conversation fills the window. With CLEAN_UP_ON_FULL, a response can end early when the context fills; some no-room turns are reset and retried once. With DO_NOT_CLEAN_UP, the failing conversation is kept and media turns may require resetSession() before continuing.

Function Calling

var functionCallingSystemPrompt: String?

fun registerTool(spec: LLMToolSpec, executor: LLMToolExecutor)
fun unregisterTool(name: String): Boolean
fun clearTools()
fun registeredTools(): List<LLMToolSpec>
fun runWithTools(text: String): Flow<String>

After registering a tool, call runWithTools(...) and collect its Flow. run(...) is only available when no tools are registered.

Tool execution is not supported by media sessions.

model.registerTool(
    LLMToolSpec(
        name = "lookup",
        description = "Look up local app data.",
        parametersJson = """{"type":"object","properties":{"query":{"type":"string"}}}""",
    ),
) { call ->
    LLMToolResult(content = """{"result":"Found"}""")
}

model.runWithTools("Use lookup to answer the question.").collect { token ->
    append(token)
}

KV State Persistence

fun saveKVState(path: String)
fun loadKVState(path: String)
fun resetKVState()

Use these APIs to persist or reset the current LLM state for resume flows.

Background Downloads

Schedule a durable LLM artifact download without initializing the LLM runtime. Keep the returned handle to observe or stop the job after the app moves to the background.

fun downloadInBackground(
    context: Context,
    personalKey: String,
    name: String,
    version: Int? = null,
    modelMode: LLMModelMode = LLMModelMode.RUN_AUTO,
    cacheHandlingPolicy: ModelCacheHandlingPolicy = ModelCacheHandlingPolicy.KEEP_EXISTING,
): BackgroundDownloadHandle

fun getBackgroundDownloadStatus(
    context: Context,
    handle: BackgroundDownloadHandle,
): BackgroundDownloadStatus

fun stopBackgroundDownload(
    context: Context,
    handle: BackgroundDownloadHandle,
): BackgroundDownloadStopResult

fun removeDownloadedModel(
    context: Context,
    name: String,
    version: Int? = null,
): ModelRemovalResult
val handle = ZeticMLangeLLMModel.downloadInBackground(
    context = context,
    personalKey = PERSONAL_KEY,
    name = "account_name/project_name",
)

val status = ZeticMLangeLLMModel.getBackgroundDownloadStatus(context, handle)
val stopResult = ZeticMLangeLLMModel.stopBackgroundDownload(context, handle)
val removal = ZeticMLangeLLMModel.removeDownloadedModel(
    context = context,
    name = "account_name/project_name",
)

Stopping preserves an already installed model. Close active LLM instances and retry removal when removal.isInUse is true. See Cache Management for states and result values.

Lifecycle

val isClosed: Boolean
fun cleanUp()
fun resetSession()
fun close()

Call cleanUp() or resetSession() before starting a fresh conversation. Call close() when the model is no longer needed.

On this page