Introduction

Face Landmark is an advanced deep learning model designed to detect and classify human emotions from facial landmarks in images or video streams. In this blog post, we'll explore how to implement Face Emotion Recognition on various mobile devices using ZETIC.MLange, a powerful framework for on-device AI applications. After this post you can make your own on-device face emotion recognition app utilizing Mobile NPUs.

What is Face Landmark?

The Face Landmark model in Google’s MediaPipe is a highly efficient machine learning model used for real-time face detection and landmark extraction.

What is ZETIC.MLange?: Bringing AI to Mobile devices

ZETIC.MLange is a On-device AI framework that enables developers to deploy complex AI models, like Face Landmark, on mobile devices with target hardware utilizations. It leverages on-device NPU (Neural Processing Unit) capabilities for efficient inference.

Github Repository

We provide Face Emotion Recognition demo application source code for both Android and iOS. repository

Model pipelining

For accurate use of the face landmark model, it is necessary to pass an image of the correct facial area to the model. To accomplish this, we construct a pipeline with the Face Detection Model.

1. Face Detection: we use the Face Detection Model to accurately detect the face regions in the image. Using the information from the detected face region, we extract that part of the original image.

2. Face Landmark: Input the extracted face image into the Face Landmark model to analyze facial landmarks.

Implementation Guide

0. Prerequisites

Prepare the model and input sample of Face Landmark and Face Detection from hugging face.

1. Face Detection model

$ pip install tf2onnx
$ python -m tf2onnx.convert --tflite face_detection_short_range.tflite --output face_detection_short_range.onnx --opset 13

2. Face Landmark model

$ pip install tf2onnx
$ python -m tf2onnx.convert --tflite face_landmark.tflite --output face_landmark.onnx --opset 13

3. ZETIC.MLange module file

• Android: Link

• iOS: Link

Step 1. Generate ZETIC.MLange Model Key

Generate MLange Model Key with mlange_gen

# (1) Get mlange_gen
$ wget <https://github.com/zetic-ai/ZETIC_MLange_document/raw/main/bin/mlange_gen> && chmod 755mlange_gen

# (2) Run mlange_gen for two models
#    - Face detection model
$ ./mlange_gen -m face_detection_short_range.onnx -i input.npy

#    - Face emotion recognition model
$ ./mlange_gen -m face_landmark.pt -i

Expected output

Step 2. Implement ZeticMLangeModel with your model key

Anroid (Kotlin):
For the detailed application setup, please follow deploy to Android Studio page

val faceLandmarkModel = ZeticMLangeModel(this, 'face_landmark')

faceLandmarkModel.run(inputs)

val outputs = faceLandmarkModel.outputBuffers

iOS (Swift):
For the detailed application setup, please follow deploy to XCode page

let faceLandmarkModel = ZeticMLangeModel('face_landmark')

faceLandmarkModel.run(inputs)

let outputs = faceLandmarkModel.getOutputDataArray()

Step 3. Prepare Face Emotion Recognition image feature extractor for Android and iOS

Android (Kotlin)

// (0) Initialize ZeticMLangeFeatureFaceLandmark
val feature = ZeticMLangeFeatureFaceLandmark()

// (1) Preprocess bitmap and get processed float array
val inputs = feature.preprocess(bitmap)

...

// (2) Postprocess to bitmap
val resultBitmap = feature.postprocess(outputs

iOS (Swift)

import ZeticMLange

// (0) Initialize ZeticMLangeFeatureFaceLandmark
let feature = ZeticMLangeFeatureFaceLandmark()

// (1) Preprocess UIImage and get processed float array
let inputs = feature.preprocess(image)

...

// (2) Postprocess to UIImage
let resultBitmap = feature.postprocess(&outputs)

Step 4. Putting It All Together

Android (Kotlin)

1. Face Detection Model

// (0) Initialization Models
val faceDetectionModel = ZeticMLangeModel(this, 'face_detection')

// (1) Initialization Feature
val faceDetectionFeature = ZeticMLangeFeatureFaceDetection()

// (2) Preprocess Image
val faceDetectionInputs = faceDetectionFeature.preprocess(bitmap)

// (3) Process Model
faceDetectionModel.run(faceDetectionInputs)
val faceDetectionOutputs = faceDetectionModel.outputBuffers

// (4) Postprocess model run result
val faceDetectionPostprocessed = faceDetectionFeature.postprocess(faceDetectionOutputs

2. Face Landmark Model: Pass the result of face detection model as a input.

// (0) Initialization Models
val faceLandmarkModel = ZeticMLangeModel(this, 'face_landmark')

// (1) Initialization Feature
val faceLandmarkFeature = ZeticMLangeFeatureFaceLandmark()

// (2) Preprocess Image
val faceLandmarkInputs = faceLandmarkFeature.preprocess(bitmap, faceDetectionPostprocessed)

// (3) Process Model
faceLandmarkModel.run(faceLandmarkInputs)
val faceLandmarkOutputs = faceLandmarkModel.outputBuffers

// (4) Postprocess model run result
val faceLandmarkPostprocessed = faceLandmarkFeature.postprocess(faceLandmarkOutputs

iOS (Swift)

1. Face Detection Model

// (0) Initialization Models
let faceDetectionModel = ZeticMLangeModel('face_detection')

// (1) Initialization Feature
let faceDetectionFeature = ZeticMLangeFeatureFaceDetection()

// (2) Preprocess Image
let faceDetectionInputs = faceDetectionFeature.preprocess(bitmap)

// (3) Process Model
faceDetectionModel.run(faceDetectionInputs)
let faceDetectionOutputs = faceDetectionModel.getOutputDataArray()

// (4) Postprocess model run result
let faceDetectionPostprocessed = faceDetectionFeature.postprocess(faceDetectionOutputs)

2. Face Landmark Model: Pass the result of face detection model as a input.

// (0) Initialization Models
let faceLandmarkModel = ZeticMLangeModel(this, 'face_landmark')

// (1) Initialization Feature
let faceLandmarkFeature = ZeticMLangeFeatureFaceLandmark()

// (2) Preprocess Image
let faceLandmarkInputs = faceLandmarkFeature.preprocess(bitmap, faceDetectionPostprocessed)

// (3) Process Model
faceLandmarkModel.run(faceLandmarkInputs)
let faceLandmarkOutputs = faceLandmarkModel.getOutputDataArray()

// (4) Postprocess model run result
let faceLandmarkPostprocessed = faceLandmarkFeature.postprocess(faceLandmarkOutputs)

Conclusion: Face Landmark and On-Device AI - Innovation at the Edge and Limitless Potential

Face landmark detection, empowered by On-Device AI, represents a key innovation at the intersection of computer vision and edge computing. By leveraging the capabilities of neural processing units (NPUs) embedded within mobile and edge devices, we enable fast, efficient, and accurate facial analysis in real time, without the need for cloud resources. This technology promises to significantly enhance a wide array of applications, from augmented reality (AR) and virtual reality (VR) experiences to biometric authentication, and personalized content delivery.

The primary benefit of On-Device AI lies in its ability to process and analyze data locally, ensuring rapid response times and preserving user privacy by avoiding the need for cloud data transfers. This decentralized approach not only minimizes latency but also reduces bandwidth and operational costs, making advanced face landmark detection more accessible and scalable across various industries.

Do you have more questions? We welcome your thoughts and inquiries!

• For More Information: If you need further details, please don't hesitate to reach out through ZETIC.ai's Contact Us.

• Join Our Community: Want to share ideas with other developers? Join our Discord community and feel free to leave your comments!

Your participation can help shape the future of on-device AI. We look forward to meeting you in the exciting world of AI!