LLM Inference
LLM Inference Modes
Configure LLM inference modes for speed or accuracy with ZETIC Melange.
LLMModelMode controls the automatic selection strategy used by ZeticMLangeLLMModel.
Available Modes
| Mode | Purpose |
|---|---|
RUN_AUTO / runAuto | Default strategy. Lets the SDK select the best available runtime and quantization for the device. |
RUN_SPEED / runSpeed | Prioritizes lower latency. |
RUN_ACCURACY / runAccuracy | Prioritizes better accuracy when multiple candidates are available. |
API Usage
val modelSpeed = ZeticMLangeLLMModel(
context = context,
personalKey = PERSONAL_KEY,
name = MODEL_NAME,
modelMode = LLMModelMode.RUN_SPEED,
)let modelSpeed = try await ZeticMLangeLLMModel(
personalKey: PERSONAL_KEY,
name: MODEL_NAME,
modelMode: .RUN_SPEED
)
final modelSpeed = await ZeticMLangeLLMModel.create(
personalKey: personalKey,
name: modelName,
modelMode: LLMModelMode.runSpeed,
);