LLM Inference
Retrieval-Augmented Generation
Use retrieval and local context with ZeticMLange LLMs.
Retrieval-augmented generation (RAG) lets your app retrieve relevant text chunks and pass them into an on-device LLM as grounded context.
This page reflects ZeticMLange 1.9.1. RAG APIs are available on Android, iOS, and Flutter through a composition-based RagPipeline shape.
Concepts
| Concept | Purpose |
|---|---|
| Retriever | App-provided component that returns relevant chunks for a query. |
| Retrieved chunk | Text plus optional score, source, and metadata. |
| RAG pipeline | Combines retrieval, prompt assembly, and token streaming. |
| Local RAG pipeline | Flutter, Android, and iOS helper for on-device chunking, embedding, and retrieval. |
Android
Android exposes RagPipeline, Retriever, RetrievedChunk, and LocalRagPipeline.
class MyRetriever : Retriever {
override suspend fun retrieve(query: String, topK: Int): List<RetrievedChunk> {
return listOf(
RetrievedChunk(
text = "ZeticMLange runs optimized models on-device.",
score = 0.94f,
source = "docs",
),
)
}
}
val pipeline = RagPipeline(
retriever = MyRetriever(),
llm = model,
profile = profile,
)
pipeline.respond("What does ZeticMLange do?").collect { token ->
append(token)
}iOS
iOS exposes RagPipeline, Retriever, RetrievedChunk, and LocalRagPipeline.
final class MyRetriever: Retriever {
func retrieve(query: String, topK: Int) async throws -> [RetrievedChunk] {
[
RetrievedChunk(
text: "ZeticMLange runs optimized models on-device.",
score: 0.94,
source: "docs"
)
]
}
}
let pipeline = RagPipeline(
retriever: MyRetriever(),
llm: model,
profile: profile
)
for try await token in pipeline.respond(query: "What does ZeticMLange do?") {
append(token)
}For fully local retrieval:
let localRag = try LocalRagPipeline.create(
embedderGgufPath: embedderPath,
profile: profile
)
try await localRag.indexDocs([
(text: "ZeticMLange runs optimized models on-device.", source: "docs")
])Flutter
Flutter exposes RagPipeline, RagRetriever, RetrievedChunk, and LocalRagPipeline.
final retriever = MyRetriever();
final rag = RagPipeline(
retriever: retriever,
llm: model,
profile: const RagProfile.qwen25(),
);
await for (final token in rag.respond(query: 'What does ZeticMLange do?')) {
append(token);
}MyRetriever can be any app-owned class that implements RagRetriever:
final class MyRetriever implements RagRetriever {
@override
Future<List<RetrievedChunk>> retrieve(String query, {required int topK}) async {
return const [
RetrievedChunk(
text: 'ZeticMLange runs optimized models on-device.',
score: 0.94,
source: 'docs',
),
];
}
}For a local retrieval pipeline:
final localRag = await LocalRagPipeline.create(
profile: const RagProfile.qwen25(backboneGgufPath: backbonePath),
embedderGgufPath: embedderPath,
);
await localRag.indexDocs([
const RagDocument(
text: 'ZeticMLange runs optimized models on-device.',
source: 'docs',
),
]);
final rag = RagPipeline(
retriever: localRag,
llm: model,
profile: const RagProfile.qwen25(backboneGgufPath: backbonePath),
);
await for (final token in rag.respond(query: 'What does ZeticMLange do?')) {
append(token);
}