Face Landmark Detection

Build an on-device face landmark detection application using a two-model pipeline with ZETIC Melange. This tutorial demonstrates how to chain Face Detection and Face Landmark models together for accurate facial landmark extraction on Android and iOS.

We provide Face Landmark demo application source code for both Android and iOS.

What You Will Build

A real-time face landmark detection application that first detects faces, then extracts detailed facial landmarks from each detected face region. This two-step pipeline ensures accurate landmark placement by feeding properly cropped face images to the landmark model.

Prerequisites

A ZETIC Melange account with a Personal Key (sign up at melange.zetic.ai)
Python 3.8+ with tf2onnx installed
The Face Detection and Face Landmark TFLite models
Android Studio or Xcode for mobile deployment

What is Face Landmark?

The Face Landmark model in Google's MediaPipe is a highly efficient machine learning model used for real-time face detection and landmark extraction.

Official documentation: Face Landmarker - Google AI

Model Pipelining

For accurate use of the face landmark model, it is necessary to pass an image of the correct facial area to the model. To accomplish this, we construct a pipeline with the Face Detection model:

Face Detection: Use the Face Detection model to accurately detect face regions in the image. Extract that part of the original image using the detected face region information.
Face Landmark: Input the extracted face image into the Face Landmark model to analyze facial landmarks.

Step 1: Convert the Models to ONNX

Prepare both models from GitHub and convert them to ONNX format.

Face Detection model:

pip install tf2onnx
python -m tf2onnx.convert --tflite face_detection_short_range.tflite --output face_detection_short_range.onnx --opset 13

Face Landmark model:

python -m tf2onnx.convert --tflite face_landmark.tflite --output face_landmark.onnx --opset 13

Step 2: Generate Melange Models

Upload both models and their inputs via the Melange Dashboard or use the CLI:

zetic gen -p $PROJECT_NAME -i input.npy face_detection_short_range.onnx
zetic gen -p $PROJECT_NAME -i input.npy face_landmark.onnx

Step 3: Implement ZeticMLangeModel

We prepared model keys for the demo app: face_detection and face_landmark. You can use these model keys to try the Melange Application.

For detailed application setup, please follow the Android Integration Guide guide.

val faceLandmarkModel = ZeticMLangeModel(this, PERSONAL_KEY, "face_landmark")

faceLandmarkModel.run(inputs)

val outputs = faceLandmarkModel.outputBuffers

For detailed application setup, please follow the iOS Integration Guide guide.

let faceLandmarkModel = try ZeticMLangeModel(personalKey: PERSONAL_KEY, name: "face_landmark")

try faceLandmarkModel.run(inputs)

let outputs = faceLandmarkModel.getOutputDataArray()

Step 4: Use the Face Landmark Wrapper

We provide a Face Landmark feature extractor as an Android and iOS module.

The Face Landmark feature extractor extension will be released as an open-source repository soon.

// (0) Initialize Face Landmark wrapper
val feature = FaceLandmarkWrapper()

// (1) Preprocess bitmap and get processed float array
val inputs = feature.preprocess(bitmap)

// ... run model ...

// (2) Postprocess to bitmap
val resultBitmap = feature.postprocess(outputs)

import ZeticMLange

// (0) Initialize Face Landmark wrapper
let feature = FaceLandmarkWrapper()

// (1) Preprocess UIImage and get processed float array
let inputs = feature.preprocess(image)

// ... run model ...

// (2) Postprocess to UIImage
let resultBitmap = feature.postprocess(&outputs)

Complete Face Landmark Pipeline Implementation

The complete implementation requires pipelining two models: Face Detection followed by Face Landmark.

Step 1: Face Detection

// (0) Initialize face detection model
val faceDetectionModel = ZeticMLangeModel(this, PERSONAL_KEY, "face_detection")
val faceDetection = FaceDetectionWrapper()

// (1) Preprocess image
val faceDetectionInputs = faceDetection.preprocess(bitmap)

// (2) Run face detection model
faceDetectionModel.run(faceDetectionInputs)
val faceDetectionOutputs = faceDetectionModel.outputBuffers

// (3) Postprocess to get face regions
val faceDetectionPostprocessed = faceDetection.postprocess(faceDetectionOutputs)

Step 2: Face Landmark

// (0) Initialize face landmark model
val faceLandmarkModel = ZeticMLangeModel(this, PERSONAL_KEY, "face_landmark")
val faceLandmark = FaceLandmarkWrapper()

// (1) Preprocess with detected face regions
val faceLandmarkInputs = faceLandmark.preprocess(bitmap, faceDetectionPostprocessed)

// (2) Run face landmark model
faceLandmarkModel.run(faceLandmarkInputs)
val faceLandmarkOutputs = faceLandmarkModel.outputBuffers

// (3) Postprocess to get landmarks
val faceLandmarkPostprocessed = faceLandmark.postprocess(faceLandmarkOutputs)

Step 1: Face Detection

// (0) Initialize face detection model
let faceDetectionModel = try ZeticMLangeModel(personalKey: PERSONAL_KEY, name: "face_detection")
let faceDetection = FaceDetectionWrapper()

// (1) Preprocess image
let faceDetectionInputs = faceDetection.preprocess(bitmap)

// (2) Run face detection model
try faceDetectionModel.run(faceDetectionInputs)
let faceDetectionOutputs = faceDetectionModel.getOutputDataArray()

// (3) Postprocess to get face regions
let faceDetectionPostprocessed = faceDetection.postprocess(&faceDetectionOutputs)

Step 2: Face Landmark

// (0) Initialize face landmark model
let faceLandmarkModel = try ZeticMLangeModel(personalKey: PERSONAL_KEY, name: "face_landmark")
let faceLandmark = FaceLandmarkWrapper()

// (1) Preprocess with detected face regions
let faceLandmarkInputs = faceLandmark.preprocess(bitmap, faceDetectionPostprocessed)

// (2) Run face landmark model
try faceLandmarkModel.run(faceLandmarkInputs)
let faceLandmarkOutputs = faceLandmarkModel.getOutputDataArray()

// (3) Postprocess to get landmarks
let faceLandmarkPostprocessed = faceLandmark.postprocess(&faceLandmarkOutputs)

Conclusion

With ZETIC Melange, building multi-model pipelines for on-device AI is straightforward. The Face Detection to Face Landmark pipeline demonstrates how you can chain models together for accurate, real-time facial analysis with NPU acceleration.

We are continually adding new models to our examples and HuggingFace page.

Stay tuned and contact us to collaborate on exciting projects!

Face Landmark Detection

On this page