Android + OpenCV: Part 7 —Face Detection in Native Library

The native code of face detection is almost finished. I will compare the C++ code and Java code with their performance at the end of this part. In the last section, I have done the grayscale conversion. At present, I need to finish the code for downsizing the image and paint the rectangles.

Here is the menu:

📛 1. Replace OpenCV 3.4.10 to 4.3.0
😄2. Permission of CameraBridgeViewBase
🌞3. C++: faceDetection()
😙4. FaceDetector: “scaleFactor”
😵5. FaceDetector: “detectMultiScale”
👴6. MainActivity & Test
😱7. Shocking Performance Result

📛1. Replace OpenCV 3.4.10 to 4.3.0

I want to continue with version 3.4.10, but the detectMultiScale() of the CascadeClassifier is not working. If you have the same problem, I suggest that you either downgrade or replace it with the newest, such version 4.3.0. It’s similar to part 4.

1st. The setting.gradle,

def opencvsdk = '{Your Path}/OpenCV-android-sdk-430'
include ':OpenCV430'
project(':OpenCV430').projectDir = new File(opencvsdk + '/sdk')

2nd. The build.gradle(Module: app): def opencvsdk in all externalNativeBuild{}s and sourceSets

android {
    compileSdkVersion 29
    buildToolsVersion "29.0.3"

    defaultConfig {
        ...
        externalNativeBuild {
            def opencvsdk = '{Your Path}/OpenCV-android-sdk-430/sdk'
            cmake {
                // Macro constants for the C compiler.
                cFlags "-D__STDC_FORMAT_MACROS"
                // Sets optional flags for the C++ compiler.
                cppFlags "-std=c++14 -frtti -fexceptions"
                // Passes optional arguments to CMake.
                arguments "-DANDROID_TOOLCHAIN=clang",
                        "-DANDROID_STL=c++_shared",
                        "-DANDROID_PLATFORM=android-21",
                        "-DOpenCV_DIR=" + opencvsdk + "/native/jni"
            }
        }
    }

    ...
    externalNativeBuild {
        cmake {
            path file('src/main/jni/CMakeLists.txt')
        }
    }
    sourceSets {
        def opencvsdk = '{Your Path}/OpenCV-android-sdk-430/sdk'
        main {
            jni {
                srcDirs 'src/main/jni', opencvsdk
            }
        }
    }
    ...
}

dependencies {
    ...
    // include OpenCV SDK
    implementation project(':OpenCV430')
}

3rd. The path in Android.mk and CMakelists.txt.

Android.mk:
CVROOT := '{Your Path}/OpenCV-android-sdk-430/sdk/native/jni'
CMakeLists.txt:
...
# Configure path to OpenCV include directories
include_directories( ${OpenCV_DIR}/native/jni/include )
...
set_target_properties(
        # Specifies the target library.
        cv_core-lib
        # Specifies the parameter you want to define.
        PROPERTIES IMPORTED_LOCATION
        # Provides the path to the library you want to import.
        ${OpenCV_DIR}/native/libs/${ANDROID_ABI}/libopencv_java4.so )

😄2. Permission of CameraBridgeViewBase

After you replace the code in deliverAndDrawFrame() of CameraBridgeViewBase.java (You’d better back it up first.) from Mike Heavers, you may have noticed the CameraBridgeViewBase has added permission control, called setCameraPermissionGranted(), to turn the camera ON/OFF. So you have to turn it ON in MainActivity.

Turn on OpenCV & camera:

private fun checkOpenCV(context: Context) {
    if (OpenCVLoader.initDebug()) {
        //shortMsg(context, OPENCV_SUCCESSFUL)
        lgd("OpenCV started...")
        viewFinder?.let {
            viewFinder.setCameraPermissionGranted()
            viewFinder.enableView()
            lgd("CameraView turned ON...")
        }
    } else { lge(OPENCV_PROBLEM) }
}

onCreate():

// Request camera permissions
if (allPermissionsGranted()) {
    checkOpenCV(this)
} else {
    ActivityCompat.requestPermissions(
        this,
        REQUIRED_PERMISSIONS,
        REQUEST_CODE_PERMISSIONS
    )
}

And onResume():

override fun onResume() {
    super.onResume()
    checkOpenCV(this)
}

😉Isn’t it great? You have more control.

🌞3. C++: faceDetection()

Let’s create a new native function, faceDetection(). In OpenCvNativeCall.kt,

external fun faceDetection(
    matAddrRgba:Long,
    height: Int,
    rotation: Int,
    modelPath: String
): Boolean;

The arguments are the address of the color image, the height of downsized grayscale, the rotation of the screen, and the path of the Haar cascade model file. Right-click the “faceDetection” to create a new function in the Cpp file, opencv_jni.cpp.

extern "C"
JNIEXPORT jboolean JNICALL
Java_com_homan_huang_opencvcamerademo_OpenCvNativeCall_faceDetection(
        JNIEnv *env, jobject thiz,
        jlong mat_addr_rgba,
        jint height,
        jint rotation,
        jstring model_path ) {
}

We need to copy the same function to the header. In addition, I add a global var, faceDetector.

#include "opencv_jni.h"

// global var
CascadeClassifier faceDetector;

Let’s continue with the function:

extern "C"
JNIEXPORT jboolean JNICALL
Java_com_homan_huang_opencvcamerademo_OpenCvNativeCall_faceDetection( ... ) {
    Mat& mRgb = *(Mat*) mat_addr_rgba;
    Mat mGr = Mat();
    int mHeight = (int) height;
    int mRotation = (int) rotation;

    if (!color2Gray(mRgb, mGr)) {
        lge("Grayscale conversion: failed!!!");
    }

They are conversions from Java to C. We often need to work on at the beginning of each NDK function.

😙4. FaceDetector: “scaleFactor”

In CascadeClassifier, the “detectMultiScale” function helps us to detect the faces and insert the coordinates into an array. We also can set scaleFactor to downsize the image, which I have calculated in the past part.

extern "C"
JNIEXPORT jboolean JNICALL
Java_com_homan_huang_opencvcamerademo_OpenCvNativeCall_faceDetection( ... ) {
    ...
    // downsize ratio of the grayscale image
    Size src = Size(mGr.size().width, mGr.size().height);
    double ratio = getRatio(src, mHeight);

    

    return (jboolean) true;
}

Let’s get the ratio in C++.

// Ratio between true size and grayscale size
double getRatio(Size src, int newSize) {
    short int w = static_cast<short>(src.width);
    short int h = static_cast<short>(src.height);
    short int heightMin = 320;

    if (newSize < heightMin) {
        lgd("Input size is too small! Set to 320 px.");
    } else {
        heightMin = static_cast<short>(newSize);
    }

    float ratio;
    if (w > h) {
        if (w < heightMin) return 0.0;
        ratio = (float) heightMin / w;
    } else {
        if (h < heightMin) return 0.0;
        ratio = (float) heightMin / h;
    }

    return ratio;
}

It’s a little different from the one at Java. I set the smaller image to 0.0 because the scale factor works like this:

# 1.1 = downsize 10% of the original
# 1.5 = downsize 50% of the original

The 0.0 means that I don’t downsize at all. At later line,

double scale = 1.0+ratio;

1 + 0 = 1, it’ll be no change.

😵5. FaceDetector: “detectMultiScale”

Now, let’s create a face detector.

extern "C"
JNIEXPORT jboolean JNICALL
Java_com_homan_huang_opencvcamerademo_OpenCvNativeCall_faceDetection( ... ) {
    ...
    // convert model_path to string
    const char *value = env->GetStringUTFChars(model_path, 0);
    String modelPath = (String) value;

    // load face detector
    faceDetector = CascadeClassifier();
    if (!faceDetector.load(modelPath)) {
        String error = "Error loading the model: "+modelPath;
        lge(error.c_str());
        return (jboolean) false;
    }

    // draw faces
    drawFaceRectangle(
            mRgb,
            mGr,
            modelPath,
            ratio,
            mRotation);

    return (jboolean) true;
}

Next, right-click the “drawFaceRectangle” to create the header & function.

void drawFaceRectangle(
        Mat &rgba,
        Mat &gray,
        String path,
        double ratio,
        int rotation) {
}

In this function, we need to understand the arguments of the detectMultiScale().

faceDetector.detectMultiScale(
        gray, // image
        faces, // array
        scale, // scale
        neighbor, // min neighbors
        0, // flags,
        Size(30, 30), // min size: 30x30
        Size((int)scrW, (int)scrH) // max size: screen size
);

They are easy to understand. You have to set the minimum and maximum face size. So before the detector, I need to set some variables and rotate the grayscale image for detection.

// width and height of frame
float scrW = (float)rgba.size().width;
float scrH = (float)rgba.size().height;

vector<cv::Rect> faces;
double scale = 1.0+ratio;
int neighbor = 3;

// fix orientation so it can be detected
rotateGray(gray, rotation);

Right-click the “rotateGray” to add a new header and function.

// Rotate the grayscale image by rotation
void rotateGray(Mat &src, int rotation) {
    switch (rotation) {
        case 0:
            rotate(src, src, ROTATE_90_CLOCKWISE);
            flip(src, src, 1);
            break;
        case 90:
            break;
        case 180:
            rotate(src, src, ROTATE_90_COUNTERCLOCKWISE);
            break;
        case 270:
            flip(src, src, 0);
            break;
        default:
            string msg = "Error: wrong rotation data -- " +
                         to_string(rotation);
            lge(msg.c_str());
            break;
    }
}

Next, we can draw each rectangle in the array.

void drawFaceRectangle(
        Mat &rgba,
        Mat &gray,
        String path,
        double ratio,
        int rotation) {
    ...
    faceDetector.detectMultiScale( ... );

    for (int i=0; i<faces.size(); i++) {
        Rect rect = faces[i];

        float x = (float)rect.x;
        float y = (float)rect.y;        
        float rw = (float)rect.width;
        float rh = (float)rect.height;
        
        float w = x + rw;
        float h = y + rh;
        float yFix, hFix;

        // draw rectangle
        switch (rotation) {
            case 0:
                rectFace(rgba, y, x, h, w, RED);
                drawDot(rgba, y, x, GREEN);
                break;

            case 90:
                rectFace(rgba, x, y, w, h, RED);
                drawDot(rgba, x, y, GREEN);
                break;

            case 180:
                // fix height
                yFix = scrW - y;
                hFix = yFix - rh;
                rectFace(rgba, yFix, x, hFix, w, YELLOW);
                drawDot(rgba, yFix, x, BLUE);
                break;

            case 270:
                // fix height
                yFix = scrH - y;
                hFix = yFix - rh;
                rectFace(rgba, x, yFix, w, hFix, YELLOW);
                drawDot(rgba, x, yFix, BLUE);
                break;

            default:
                string msg = "Error: wrong rotation data -- " +
                             to_string(rotation);
                lge(msg.c_str());
                break;
        }
    }
}

👴6. MainActivity & Test

Let’s add the call native call in MainActivity.kt.

override fun onCameraFrame(inputFrame: CvCameraViewFrame?): Mat {
    imageMat = inputFrame!!.rgba()

    val javaEnabled = false;

    if (!javaEnabled) {
        // C++
        if (OpenCvNativeCall().faceDetection(
                imageMat.nativeObjAddr,
                480,
                screenRotation,
                faceModel!!.absolutePath)) {
            longMsg(this, "Failed to load Face Detector!!!")
        }
    } else {...}
    return imageMat
}

I add a longMsg call in companion object{},

companion object {
    ...
    fun longMsg(context: Context, s: String) =
        Toast.makeText(context, s, Toast.LENGTH_LONG).show()

It’s ready. Let’s run.

😻Here are some captures from AI faces.

😱7. Shocking Performance Result

This is how I test the performance between Kotlin and C++:

1. Record the time elapse of each frame.
2. Calculate the average time from the record.

Simple, isn’t it?

// Performance variables
var frameStart: Long = 0
var frameEnd: Long = 0
var pCounter: Int = 0
val frameRecord = LongArray(1000)
val javaEnabled = false

override fun onCameraFrame(inputFrame: CvCameraViewFrame?): Mat {

I set my app to finish at the 1000th frame.

override fun onCameraFrame(inputFrame: CvCameraViewFrame?): Mat {
    if (pCounter == 1000) {
        evalPerfomance()
        finish()
    }
    // start time
    frameStart = System.currentTimeMillis()

    ...

    postText("$pCounter", 100.0, 100.0, YELLOW)

    // end time
    frameEnd = System.currentTimeMillis()
    frameRecord[pCounter] = frameEnd - frameStart
    pCounter++

    return imageMat
}
/*
    Print text on MAT:
        mText: title
        x: x-coor of location
        y: y-coor of location
        color: RGB
*/
fun postText(mText: String, x: Double, y:Double, color: Scalar) {
    Imgproc.putText(
        imageMat, // image
        mText, // text
        Point(x, y), // location
        FONT_HERSHEY_PLAIN, // font face
        1.5, //font scale
        color, // RGB
        1 //thickness
    )
}

When javaEnabled = true, it will run the Kotlin code only.

Here is the average function.

fun evalPerfomance() {
    val avg = frameRecord.average()
    lgd("Java Enable? $javaEnabled ... Runtime average = $avg ms")
}

After that, I run the code on my phone and face a group of AI faces image. Here is the result.

7204-7204 D/MYLOG MainActivity: OpenCV started...
7204-7204 I/MYLOG MainActivity: OpenCV Loaded Successfully!
7204-7204 D/MYLOG MainActivity: OpenCV started...
7204-7204 D/MYLOG MainActivity: CameraView turned ON...
7204-7204 D/MYLOG MainActivity:  System Ui Visibility Change
7204-7239 D/MYLOG MainActivity: Java Enable? true ... Runtime average = 39.043 ms
7510-7510 D/MYLOG MainActivity: OpenCV started...
7510-7510 I/MYLOG MainActivity: OpenCV Loaded Successfully!
7510-7510 D/MYLOG MainActivity: OpenCV started...
7510-7510 D/MYLOG MainActivity: CameraView turned ON...
7510-7510 D/MYLOG MainActivity:  System Ui Visibility Change
7510-7547 D/MYLOG MainActivity: Java Enable? false ... Runtime average = 88.269 ms

What?! The Java library ran 2X faster than the NDK code. This result shows Java version has passed 66.66 ms vs 15 fps default setting by Android SDK. But C++ version has failed to pass the line. We need to tune it to reach 66.66ms or lower.

Before: Part 6 — OpenCV Native Library

Next: Part 8 — DIY SDK + Contrib Modules on Windows