Technical Requirements for On-device AI

On-device AI has the advantage of not incurring server costs, not being dependent on the network, and not having to worry about data leakage because model inference is performed inside the device.

However, the fact that model inference is performed inside the device means that a lot of optimization must be done. High-performance models must be optimized, limited computing resources must be utilized to the maximum, and overhead must be reduced.

Several techniques are used to achieve this. The most important part is to write core logic using C++, which can bring great benefits in terms of memory management and speed. In addition, most AI model runtimes are written in C++, which has the advantage of being easy to utilize.

ZETIC.MLange uses a technology called JNI to connect C++ and Java (Kotlin) codes to perform on-device AI model inference on the Android OS.

What is JNI(Java Native Interface)?

Java Native Interface (JNI) serves as a bridge between Java and native code (C/C++). It defines a comprehensive framework for interaction between these different programming environments:

• JNI establishes function calling conventions between the Java VM (Dalvik/ART) and native code

• It enables creation and manipulation of Java objects from native code

• Handles data conversion (Marshalling & Unmarshalling) between Java and Native environments

• Provides native thread management capabilities

• Utilizes the JNIEnv structure as the primary interface for native operations

How Can I Use It?

1. Build Native Libraries with CMake

CMake is a powerful build system generator that offers several advantages:

• Automatic generation of build files

• Support for multiple platforms including Makefile, Visual Studio Solution, and Xcode

• Seamless integration with Gradle

You can generate library file (.so) with add_library keyword.

When working with Android, it's crucial to build for all supported ABIs: arm64-v8a, armeabi-v7a, x86, x86_64

2. Link Native Libraries with NDK (Native Development Kit)

The Android NDK (Native Development Kit) is essential for native code development on Android. It provides:

• A comprehensive toolkit for using C/C++ code on Android

• Tools for building and executing native code

• JNI integration for Java/Kotlin interoperability

• Independent memory management from Java's garbage collection

Load Library

Libraries linked using NDK can be referenced at runtime using the System.loadLibrary() function.

3. Native Bridging with JNI

In Kotlin, if you use the external keyword and follow the naming conventions below in C++, JNI will finally bridge the code between Kotlin and C++.

extern "C" JNIEXPORT [return type] JNICALL Java_[package_name]_[class_name]_[method_name](JNIEnv*, jobject, [parameters

Architecture Proposal

Our recommended architecture implements a carefully designed layered approach that separates different concerns: Domain layer interfaces with the application logic, Wrapper layer provides abstraction and type safety, JNI layer handles the Java-Native communication, and Native layer contains the core C++ implementation. This layered architecture: Domain ↔ Wrapper ↔ JNI ↔ Native not only ensures a clean separation of responsibilities but also promotes maintainable, testable code that can evolve independently at each layer. The structure makes it easier to debug issues, modify implementations, and extend functionality without affecting other layers.

ZETIC.MLange's JNI Usage

In our implementation, we've developed a sophisticated approach to handling multiple machine learning frameworks and creating a unified interface. Our primary focus areas are:

• Building comprehensive support for multiple ML frameworks including ONNX, QNN, and TFLite, ensuring seamless integration and optimal performance across different model formats and runtime environments

• Creating a robust and flexible unified interface through the abstract ZeticMLangeTargetModel class, which provides a consistent API layer while encapsulating the complexity of different ML framework implementations

Pro Tips: More Optimization

To achieve optimal performance, consider these advanced techniques:

• Choose between NewObject and NewGlobalRef based on object lifecycle requirements

• Implement object pooling to reduce object creation overhead

• Minimize marshalling operations to reduce performance overhead

By following these guidelines and best practices, you can effectively utilize JNI in your Android applications while maintaining high performance and code quality.