Android + OpenCV: Part 4— Fix Orientations of Face Detection

7 min readApr 20, 2020

There are four orientations of a cell phone. 90 Degree only is not good enough for common users, right? He doesn’t like it. She doesn't like it. I don’t think you will like it. So I need to rotate the image so the camera can detect the faces. This is the menu of this part. I will remove OpenCV Manager, SDK junks and fix all of the orientations.

1. Remove OpenCV Manager
2. Remove Elder OpenCV Modules
3. Drawing Functions: Rectangle & Dot
4. Fix Zero
5. Fix 180
6. Fix 270

💀1. Remove OpenCV Manager

This is April 18, 2020. I downloaded OpenCV 3.4.10 from the Internet. Let’s install it and test the new SDK.

Sb: What?! I just changed 3.4.3 to 3.4.9.
Homan: Yeah! OpenCV updates its version quite quick recently.

We DO NOT need to redo the steps of Part I, because I will teach you to import the whole SDK.

In fact, the OpenCV Manager is not good for commercial usage. No customer will like to install an extra app somewhere else. So let’s remove the manager.

First, you need to open the settings.gradle to add SDK path.

def opencvsdk = '../OpenCV-Android-Lib/OpenCV-android-sdk-3410'
include ':OpenCV34X'
project(':OpenCV34X').projectDir = new File(opencvsdk + '/sdk')

Second, please open the build.gradle(Module: app) to add SDK.

dependencies {
    ...

    // include OpenCV Manager and SDK
    implementation project(':OpenCV34X')
}

Sync.

Third, let’s remove BaseLoaderCallback in MainActivity.kt.

class MainActivity : AppCompatActivity(), CameraBridgeViewBase.CvCameraViewListener2 {
...
    // openCV callback
    //lateinit var cvBaseLoaderCallback: BaseLoaderCallback


    override fun onCreate(savedInstanceState: Bundle?) {
        ...

        /*
        cvBaseLoaderCallback = object : BaseLoaderCallback(this) {...}
        */
				
	callFaceDetector()
    }
...
    private fun callFaceDetector() {
        try {
            lgi(OPENCV_SUCCESSFUL)

            loadFaceLib()

            if (faceDetector!!.empty()) {
                faceDetector = null
            } else {
                faceDir.delete()
            }
            viewFinder.enableView()
        } catch (e: IOException) {
            lge(OPENCV_FAIL)
            shortMsg(this@MainActivity, OPENCV_FAIL)
            e.printStackTrace()
        }
    }
	...
    override fun onResume() {
        super.onResume()
        //OpenCVLoader.initAsync(... cvBaseLoaderCallback)
        checkOpenCV(this)
        viewFinder?.let { viewFinder.enableView() }
    }
    ...
    private fun checkOpenCV(context: Context) {
        if (OpenCVLoader.initDebug()) {
            shortMsg(context, OPENCV_SUCCESSFUL)
            lgd("OpenCV started...")
        } else {
            lge(OPENCV_PROBLEM)
        }
    }
    ...
}

That’s it. Let’s uninstall the OpenCV Manager on your phone. Next, you need to fix the camera view again in Part 2; compile the app and run again. We must make sure everything is working fine.

💀2. Remove Elder OpenCV Modules

After we installed OpenCV34X, you will found many junks nearby.

We shall get rid of them.
File -> Project Structure… -> Modules -> Use “ — ” to remove useless modules: OpenCV343 & OpenCV349

Luckily, you shall see them gone in Android. However, if you switch to the Project view, they are still there. Let’s right-click and delete them.

Also, you need to remove all OpenCV library files in JNI folder, such as open_cv_jni made in Part 1. If the app is running fine. Commit to VCS with different content.

👍3. Drawing Functions: Rectangle &. Dot

I need to create a rectangle easier, so I code my version of the rectangle.

/* Draw rectangle of the face:
    x: x-coor of upper corner
    y: y-coor of upper corner
    w: x-coor of opposite corner
    h: y-coor of opposite corner
    color: RGB
 */
fun rectFace(x: Double, y: Double, w: Double, h: Double, color:Scalar) {
    Imgproc.rectangle(
        imageMat, // image
        Point(x, y), // upper corner
        Point(w, h),  // opposite corner
        color  // RGB
    )
}

Also, I need to pinpoint the location, so I add a dot on the corner of the rectangle.

/*
    Draw a dot:
        x: x-coor of center
        y: y-coor of center
        color: RGB
 */
fun drawDot(x: Double, y:Double, color:Scalar) {
    Imgproc.circle(
        imageMat, // image
        Point(x, y),  // center
        4, // radius
        color, // RGB
        -1, // thickness: -1 = filled in
        8 // line type
    )
}

BTW, I add some color code for easy input.

companion object {
    ...
    // RGB
    private val YELLOW = Scalar(255.0, 255.0, 0.0)
    private val BLUE = Scalar(0.0, 0.0, 255.0)
    private val RED = Scalar(255.0, 0.0, 0.0)
    private val GREEN = Scalar(0.0, 255.0, 0.0)
}

Now, let’s paint some faces.

😃4. Fix Zero

The first orientation I chose is Zero Degree. Let’s flip it after downsize.

fun get480Image(src: Mat): Mat {
    val imageSize = Size(
        src.width().toDouble(), 
        src.height().toDouble())
    imageRatio = ratioTo480(imageSize)

    // Downsize image
    val dst = Mat()
    val dstSize = Size(
        imageSize.width*imageRatio ,
        imageSize.height*imageRatio)
    resize(src, dst, dstSize)

    // Check rotation
    when (screenRotation) {
        0-> {
            rotate(dst, dst, ROTATE_90_CLOCKWISE)
        }
    }

    return dst
}

Now, I need to draw the rectangle and run the app.

fun drawFaceRectangle() {
    val faceRects = MatOfRect()
    faceDetector!!.detectMultiScale(
        grayMat,
        faceRects)

    val scrW = imageMat.width().toDouble()
    val scrH = imageMat.height().toDouble()

    for (rect in faceRects.toArray()) {
        var x = rect.x.toDouble()
        var y = rect.y.toDouble()
        var w = 0.0
        var h = 0.0
        var rw = rect.width.toDouble() // rectangle width
        var rh = rect.height.toDouble() // rectangle height

        if (imageRatio.equals(1.0)) {
            w = x + rw
            h = y + rh
        } else {
            x /= imageRatio
            y /= imageRatio
            rw /= imageRatio
            rh /= imageRatio
            w = x + rw
            h = y + rh
        }

        when (screenRotation) {
            90-> {
                rectFace(x, y, w, h, RED)
                drawDot(x, y, GREEN)
            }
            0-> {
                rectFace(y, x, h, w, RED)
                drawDot(y, x, GREEN)
            }
        }
    }
}

Let’s shoot some fake faces.

The rectangles are in mirror positions. So I need to flip.

fun get480Image(src: Mat): Mat {
    ...
    // Check rotation
    when (screenRotation) {
        0-> {
            rotate(dst, dst, ROTATE_90_CLOCKWISE)
            //mirror 
            flip(dst, dst, 1)
        }

Run the app to check the correction.

Perfect, it’s an easy fix.

😐5. Fix 180

The next orientation is 180 Degree. I rotate the image to the opposite direction of Zero Degree.

when (screenRotation) {
    0-> {
        rotate(dst, dst, ROTATE_90_CLOCKWISE)
        //mirror 
        flip(dst, dst, 1)
    }
    180-> {
        rotate(dst, dst, ROTATE_90_COUNTERCLOCKWISE)
    }
}

And I draw with the same code like Zero does.

when (screenRotation) {
    90-> {
        rectFace(x, y, w, h, RED)
        drawDot(x, y, GREEN)
    }
    0-> {
        rectFace(y, x, h, w, RED)
        drawDot(y, x, GREEN)
    }
    180-> {
        rectFace(y, x, h, w, RED)
        drawDot(y, x, GREEN)
    }
}

Run the app. Result:

It’s upside down. Next, I have tried to flip in a different direction after rotation. You can try on the option of flip code: 1, 0, -1.

rotate(dst, dst, ROTATE_90_COUNTERCLOCKWISE)
flip(dst, dst, 1)

They don’t work at all. This is odd. So I need to manually change the rectangle position without the flip. These two variables can help me to get to the right spot.

val scrW = imageMat.width().toDouble()
val scrH = imageMat.height().toDouble()

They are image width and height. According to the coordinates, I draw an extra rectangle to distinct the old one.

180-> {
    lgd("x: $x -- y: $y :: sW: $scrW, sH: $scrH")

    rectFace(y, x, h, w, RED)
    drawDot(y, x, GREEN)

    // fix height
    val yFix = scrH - y
    val hFix = yFix - rh

    rectFace(yFix, x, hFix, w, YELLOW)
    drawDot(yFix, x, BLUE) 
}

Run the app.

That’s working. 😍 Yellow rectangles are in the right positions.

😅 5. Fix 270

Let’s continue. The 270 Degree is the opposite direction of 90 Degree. So I flip.

// Check rotation
when (screenRotation) {
    0-> {
        rotate(dst, dst, ROTATE_90_CLOCKWISE)
        flip(dst, dst, 1)
    }
    180-> {
        rotate(dst, dst, ROTATE_90_COUNTERCLOCKWISE)
    }
    270-> {
        flip(dst, dst, 0)
    }
}

The drawing is as same as 90 Degree.

when (screenRotation) {
    90-> {
        rectFace(x, y, w, h, RED)
        drawDot(x, y, GREEN)
    }
    ...
    270-> {
        rectFace(x, y, w, h, RED)
        drawDot(x, y, GREEN)
    }
}

This doesn’t work. Each rectangle is running in the opposite direction. When I tilt-up, the rectangle tilts down. Next, I have tried to flip -1, double flip, and rotate 180. 😎: You can try these experiments. It’s fun to play them all. 😢: None of them is working. Again, I have to manually change the position after flip = 0, the first one I tried.

when (screenRotation) {
    ...
    270-> {
        rectFace(x, y, w, h, RED)
        drawDot(x, y, GREEN)

        // fix height
        val yFix = scrH - y
        val hFix = yFix - rh

        rectFace(x, yFix, w, hFix, YELLOW)
        drawDot(x, yFix, BLUE)
    }
}

The result: