ZeticMLangeLLMModel
API reference for running LLM inference on iOS.
This page describes the API planned for ZeticMLange iOS 1.13.0.
ZeticMLangeLLMModel loads a model selected by Melange for the device. Depending on the selected model and backend, it can generate text, process image, audio, or video inputs, stream speech, call tools, or persist KV state. Check the loaded model's capabilities and handle unsupported operations.
Import
import ZeticMLangeInitializer
public init(
personalKey: String,
name: String,
version: Int? = nil,
modelMode: LLMModelMode = .RUN_AUTO,
cacheHandlingPolicy: ModelCacheHandlingPolicy = .REMOVE_OVERLAPPING,
initOption: LLMInitOption = LLMInitOption(),
onDownload: ((Float) -> Void)? = nil
) async throws| Parameter | Type | Default | Description |
|---|---|---|---|
personalKey | String | - | Personal key for accessing the model. |
name | String | - | Model name in account_name/project_name format. |
version | Int? | nil | Model version. nil loads the default version. |
modelMode | LLMModelMode | .RUN_AUTO | Backend selection strategy. |
cacheHandlingPolicy | ModelCacheHandlingPolicy | .REMOVE_OVERLAPPING | Managed artifact cache cleanup policy. |
initOption | LLMInitOption | LLMInitOption() | LLM initialization options. |
onDownload | ((Float) -> Void)? | nil | Download progress callback from 0.0 to 1.0. |
let model = try await ZeticMLangeLLMModel(
personalKey: PERSONAL_KEY,
name: "account_name/project_name",
initOption: LLMInitOption(nCtx: 2048)
)The SDK selects and downloads the model using the supplied name; apps do not need to download a model file themselves. Model initialization can involve network, file and model-loading work, so avoid blocking the main thread. The nCtx above is an example; the supported context size depends on the selected model. Prompt, media, conversation history and generated tokens share the available context.
Text Generation
The available generation API depends on the loaded backend. For a session-capable model, use respond. Models that expose token-based generation use run and waitForNextToken below.
run(_:)
Starts generation for a prompt.
public func run(_ text: String) throwstry model.run("Explain on-device AI in one paragraph.")Reading Generated Tokens
Waits for the next generated token.
public func waitForNextToken() -> LLMNextTokenResultwhile true {
let next = model.waitForNextToken()
if next.isFinished { break }
append(next.token)
}Session Response and Media
public var capabilities: LLMCapabilities { get }
public func respond(
systemPrompt: String = "",
userText: String,
media: [LLMMedia] = []
) throws -> LLMResponseStream<String>
public func respondWithSpeech(
systemPrompt: String = "",
userText: String,
media: [LLMMedia] = [],
reply: LLMSpeechReply = .speech
) throws -> LLMResponseStream<LLMResponseEvent>LLMCapabilities reports text, image, audio, video, speechOutput, mediaInFollowupTurns, multiTurn, maxMediaPerTurn, and speechSampleRate. Check these values on the loaded model before using a modality; support can differ between models and backends. Capabilities do not guarantee that every combination of individually supported media types is supported.
respond streams text fragments and respondWithSpeech streams events containing text or owned PCM samples. Both return a cold LLMResponseStream: a turn is submitted when iteration starts. Keep the model alive through iteration. Turns are serialized; task cancellation, breaking the loop, or cancel() stops delivery. cancel() also stops already-submitted queued turns. Iterate a new response for each turn.
if model.capabilities.text {
for try await piece in try model.respond(userText: "Explain on-device AI.") {
print(piece, terminator: "")
}
}An input with neither text nor media is invalid. The context window includes prompt, media, conversation and output. LLMInitOption.kvCacheCleanupPolicy controls behavior when it fills: .CLEAN_UP_ON_FULL can end an answer early and clear conversation state, while .DO_NOT_CLEAN_UP preserves state and reports an error. If the response reports invalidState, call resetSession() and resend the required media. Errors carry LLMResponseStatus and a message. Other native failures do not automatically reset the session.
Image input
LLMImage accepts a packed, unpadded RGB byte array (width * height * 3) or a CGImage. Apply any image orientation before constructing it; the CGImage initializer does not apply EXIF or UIImage orientation.
let image = try LLMImage(cgImage: cgImage)
if model.capabilities.image {
for try await piece in try model.respond(
userText: "What is in this image?",
media: [.image(image)]
) {
print(piece, terminator: "")
}
}Audio input
LLMAudio(samples:) accepts nonempty, finite mono float32 PCM samples in [-1, 1] at 16 kHz. LLMAudio(samples:sampleRate:channels:) accepts interleaved samples, averages channels and resamples to 16 kHz. The input rate is independent of the speech output sample rate.
let audio = try LLMAudio(samples: pcm16kHz)
if model.capabilities.audio {
for try await piece in try model.respond(
userText: "Transcribe this speech.",
media: [.audio(audio)]
) {
print(piece, terminator: "")
}
}Video input
The app must decode and sample the video into RGB LLMImage frames; the SDK API does not accept a video file URL. Provide the source frame index and actual presentation time in seconds for each frame, as well as the source metadata FPS. Frames must have matching dimensions and ordered indices and presentation times; preserve actual timestamps for variable-frame-rate video.
let video = try LLMVideo(
frames: sampledFrames,
sourceFrameIndices: frameIndices,
presentationTimes: presentationSeconds,
metadataFPS: sourceFPS
)
if model.capabilities.video {
for try await piece in try model.respond(
userText: "Describe this video.", media: [.video(video)]
) {
print(piece, terminator: "")
}
}For the optimized iOS LFM3 video path, sample at 2 fps with at least four frames. It rejects more than ten frames or more than 192 MiB of combined RGB frame data, and context and output budgets can reduce the usable frame count further. Unsupported backends report video == false. Some models can accept video and a separate audio input; that audio is an independent span, not a synchronized soundtrack. Image plus video input is not supported on this path. Check the model's capabilities and supported combinations.
Streaming speech output
if model.capabilities.speechOutput {
for try await event in try model.respondWithSpeech(
userText: "Say hello.", reply: .textAndSpeech
) {
switch event {
case .text(let text):
print(text, terminator: "")
case .speech(let pcm, let sampleRate):
// Play or store this owned mono Float PCM chunk at sampleRate.
_ = (pcm, sampleRate)
}
}
}Use .speech for speech-only output or .textAndSpeech for text and speech events. Read the sample rate from each speech event; capabilities.speechSampleRate describes the model's speech output rate. Audio input and speech output use different sample-rate contracts.
Session Lifecycle
public func cancel()
public func resetSession() throwsresetSession() clears the current conversation. close() stops and waits for submitted session work before releasing the model; resetSession() also stops and waits before resetting. Perform blocking reset and close work away from the main thread. Follow the loaded model's mediaInFollowupTurns and multiTurn capabilities when sending further turns; some media models require the media to be provided again after reset.
Function Calling
public var functionCallingSystemPrompt: String?
public func registerTool(_ spec: LLMToolSpec, executor: @escaping LLMToolExecutor) throws
public func unregisterTool(name: String) throws -> Bool
public func clearTools() throws
public func registeredToolSpecs() throws -> [LLMToolSpec]try model.registerTool(
LLMToolSpec(
name: "lookup",
description: "Look up local app data.",
parametersJson: #"{"type":"object","properties":{"query":{"type":"string"}}}"#
)
) { call in
LLMToolResult(content: #"{"result":"Found"}"#)
}
for try await token in try model.runWithTools("Use lookup to answer the question.") {
append(token)
}runWithTools(_:) returns an AsyncThrowingStream. Consume it to completion before starting another model operation.
KV State Persistence
public func saveKVState(path: String) throws
public func loadKVState(path: String) throws
public func resetKVState() throwsUse these APIs to persist or reset the current LLM state for resume flows.
Background Downloads
Schedule an LLM artifact download without initializing the LLM runtime. Await scheduling while the app is foregrounded so iOS can register the background URL-session task before the app moves to the background.
static func downloadInBackground(
personalKey: String,
name: String,
version: Int? = nil,
modelMode: LLMModelMode = .RUN_AUTO,
cacheHandlingPolicy: ModelCacheHandlingPolicy = .KEEP_EXISTING
) async throws -> BackgroundDownloadHandle
static func getBackgroundDownloadStatus(
_ handle: BackgroundDownloadHandle
) async throws -> BackgroundDownloadStatus
static func stopBackgroundDownload(
_ handle: BackgroundDownloadHandle
) async throws -> BackgroundDownloadStopResult
static func removeDownloadedModel(
name: String,
version: Int? = nil
) async throws -> ModelRemovalResultlet handle = try await ZeticMLangeLLMModel.downloadInBackground(
personalKey: PERSONAL_KEY,
name: "account_name/project_name"
)
let status = try await ZeticMLangeLLMModel.getBackgroundDownloadStatus(handle)
let stopResult = try await ZeticMLangeLLMModel.stopBackgroundDownload(handle)
let removal = try await ZeticMLangeLLMModel.removeDownloadedModel(
name: "account_name/project_name"
)Configure the iOS host app to forward background URL-session events through ZeticMLangeModel.handleBackgroundURLSessionEvents(...); that forwarding API is on the general-model class and supports LLM downloads as well. See iOS advanced configuration. Stopping preserves an already installed model; if removal.isInUse is true, close active LLM instances and retry removal.
Lifecycle
public var isClosed: Bool { get }
public func cleanUp() throws
public func close()
public func forceDeinit()Call cleanUp() before starting a fresh conversation. Call close() when the model is no longer needed.