Android + OpenCV: Part 10 — Beginning of Face Landmark Detection
If you are following my work to Part 8, you shall have your SDK ready. In this part, I will implement Face API to draw the face objects, such as eyes, mouth, and nose.
_ _ _ _ _ _ _ _ — == Menu == — _ _ _ _ _ _ _ _
📸1. Fix Grayscale in Camera View
😄2. FaceMark: Load Library
💠3. Orientation: Fix Rect List
🙂4. 68 Points of Face Landmark
🙈5. Prepare the View for Facemark
🍳6. Fitting the Facemark (Only the Nose)
🆗7. Warning of Non-90 Rotation
📸1. Fix Grayscale in JavaCamera2View
This is a face recognition app. So let’s modify the JavaCamera2View source code. This is the benefit of open-source software. You do DIY your part. I don’t like to rotate and flip the grayscale image in the MainActivity. That doesn’t make sense. So I change JavaCamera2View directly. And JavaCamera2View and JavaCameraView have extended to CameraBridgeViewBase. So you need to make a change for three files.
1️⃣. 👀 CameraBridgeViewBase.
I will search with a word, gray, in this file.
public static final int RGBA = 1;
public static final int GRAY = 2;...
protected class CvCameraViewListenerAdapter implements CvCameraViewListener2 {
...
public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
...
switch (mPreviewFormat) {
...
case GRAY:
result = mOldStyleListener.onCameraFrame(
inputFrame.gray());
break;
...
};...
}...
};public interface CvCameraViewFrame {
...
/**
* This method returns single channel gray scale Mat with frame
*/
public Mat gray();
};
There’re three locations. I want a new name: mGray. So the change will be,
public static final int RGBA = 1;
public static final int GRAY = 2;
public static final int MGRAY = 3; // my modified gray
...
protected class CvCameraViewListenerAdapter implements CvCameraViewListener2 {
...
public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
...
switch (mPreviewFormat) {
...
case GRAY:...
case MGRAY: // my style of grayscale
result = mOldStyleListener.
onCameraFrame(inputFrame.mGray());
break;
...
};...
}...
};public interface CvCameraViewFrame {
...
public Mat gray();
/**
* This method returns single channel gray scale Mat with frame
*/
public Mat mGray(); // my gray function
};
2️⃣. JavaCameraView.
By the change of interface, I need to add a mGray() into JavaCameraFrame{} of JavaCameraView.
private class JavaCameraFrame implements CvCameraViewFrame {
@Override
public Mat gray() {
return mYuvFrameData.submat(0, mHeight, 0, mWidth);
}
@Override // a copy of gray()
public Mat mGray() {
return mYuvFrameData.submat(0, mHeight, 0, mWidth);
}I won’t bother to add more code because I don’t use this class. You can copy and paste for gray().
3️⃣. JavaCamera2View
The gray() is also in JavaCameraFrame{}.
private class JavaCamera2Frame implements CvCameraViewFrame {
@Override
public Mat gray() {...}...
public JavaCamera2Frame(Image image, int rotation) {
...
mGray = new Mat();
}
public void release() {
...
mGray.release();.
}
...
private Mat mGray;
};Well, it has a mGray already. So I use mGr instead.
private class JavaCamera2Frame implements CvCameraViewFrame {
@Override
public Mat gray() {...}
// Face detection grayscale image
@Override
public Mat mGray() {
Image.Plane[] planes = mImage.getPlanes();
int w = mImage.getWidth();
int h = mImage.getHeight();
assert(planes[0].getPixelStride() == 1);
ByteBuffer y_plane = planes[0].getBuffer();
int y_plane_step = planes[0].getRowStride();
mGr = new Mat(h, w, CvType.CV_8UC1, y_plane, y_plane_step);
switch (mRotation) {
case 0:
Core.rotate(mGr, mGr,
Core.ROTATE_90_CLOCKWISE);
break;
case 180:
Core.rotate(mGr, mGr,
Core.ROTATE_90_COUNTERCLOCKWISE);
break;
case 270:
Core.flip(mGr, mGr, 0);
break;
}
return mGr;
}...
public JavaCamera2Frame(Image image, int rotation) {
...
mGray = new Mat();
mGr = new Mat();
mRotation = rotation;
}
public void release() {
...
mGray.release();
mGr.release();
}
...
private Mat mGray;
private Mat mGr;
private int mRotation;
};I add a rotation variable to auto-shift the gray orientation.
private void createCameraPreviewSession() {
...
mImageReader.setOnImageAvailableListener(
new ImageReader.OnImageAvailableListener() {
@Override
public void onImageAvailable(ImageReader reader) {
...
int rotation = JavaCamera2View.
super.getOrientation();
JavaCamera2Frame tempFrame = new JavaCamera2Frame(
image, rotation);
...
}
}, mBackgroundHandler);
...
}The rotation data is from its super class, CameraBridgeViewBase, if you have added the rotation meter in Part 9. Now, you can call inputFrame.mGray().
😄2. FaceMark: Load Library
The Face API has compiled for your project if you have completed Part 8 to build your own SDK. It has a Facemark class to detect the landmark objects on the face.
To use Facemark, we can download a custom model or train your own model. Let’s work with the custom model at first, lbfmodel.yaml. This a text style YAML data set like JSON. Please download and rename lbfmodel.yaml.txt to lbfmodel.yaml. Next, you need to save the file into raw folder. I need some variables for the new raw file.
companion object {
...
// Face model
private const val FACE_DIR = "facelib"
private const val FACE_MODEL =
"haarcascade_frontalface_alt.xml"
private const val FACE_MODEL_ID =
R.raw.haarcascade_frontalface_alt
private const val FACEMARK_MODEL =
"lbfmodel.yaml"
private const val FACEMARK_MODEL_ID =
R.raw.lbfmodel
private const val byteSize = 4096 // buffer size
...
}Now, let’s load it to the private folder.
var faceModel: File? = null // face detector model
var fmModel: File? = null // facemark modelprivate fun loadFaceLib() {
try {
val cascadeInputStream =
resources.openRawResource(FACE_MODEL_D_ID)
val fmInputStream =
resources.openRawResource(FACEMARK_MODEL_ID)
// create a private directory
faceDir = getDir(FACE_DIR, Context.MODE_PRIVATE)
// create a model file
faceModel = File(faceDir, FACE_MODEL)
fmModel = File(faceDir, FACEMARK_MODEL)
if (!faceModel!!.exists()) { // copy model
fileCopy(faceModel!!, cascadeInputStream)
assert(faceModel!!.exists() == true ) {
"Out of space! Failed to create $FACE_MODEL"
}
lgi("Done created: $FACE_MODEL")
}
if (!fmModel!!.exists()) { // copy model
fileCopy(fmModel!!, fmInputStream)
assert(fmModel!!.exists() == true ) {
"Out of space! Failed to create $FACEMARK_MODEL"
}
lgi("Done created: $FACEMARK_MODEL")
}
// initial face detector
faceDetector =
CascadeClassifier(faceModel!!.absolutePath)
assert(faceDetector != null) {
"Failed to load Cascade Lib!" }
lgd("Cascade libraries load successful.")
// initial face mark detector
fmLbfDectector = Face.createFacemarkLBF()
fmLbfDectector.loadModel(fmModel!!.absolutePath)
lgd("Facemark-LBF libraries load successful.")
} catch (e: IOException) {
lge("Error loading cascade face model...$e")
}
}
This is the filecopy(),
private fun fileCopy(mFile: File, mIs: InputStream) {
// Output stream
val mOs = FileOutputStream(mFile)
val buffer = ByteArray(byteSize)
var byteRead = mIs.read(buffer)
while (byteRead != -1) {
mOs.write(buffer, 0, byteRead)
byteRead = mIs.read(buffer)
}
mIs.close()
mOs.close()
}Please check your Logcat to confirm you have loaded the model file properly. Commit your code and move on.
💠3. Orientation: Fix Rect List
Here is my re-organized code:
I add a class to contain Color and Name of Color.
/**
* Object Color:
* mColor: RGB Scalar
* name: name of color
*/
class ObjColor(c:Scalar, n:String) {
var mColor: Scalar = c
var name: String = n
override fun toString(): String {
return name
}
}So the companion object of MainActivity,
// RGB
val YELLOW = ObjColor(Scalar(255.0, 255.0, 0.0), "Yellow")
val BLUE = ObjColor(Scalar(0.0, 0.0, 255.0), "Blue")
val RED = ObjColor(Scalar(255.0, 0.0, 0.0), "Red")
val GREEN = ObjColor(Scalar(0.0, 255.0, 0.0), "Green")
val PINK = ObjColor(Scalar(255.0, 25.0, 255.0), "Pink")
val BLACK = ObjColor(Scalar(0.0, 0.0, 0.0), "Black")
val WHITE = ObjColor(Scalar(242.0, 243.0, 244.0), "White")The onCameraFrame(),
override fun onCameraFrame(inputFrame: CvCameraViewFrame?): Mat {
if (pCounter == 1000) {
evalPerfomance()
finish()
}
// start time
frameStart = System.currentTimeMillis()
val grayHeight = 480if (!javaEnabled) {...
} else {
imageMat = inputFrame!!.rgba()
// mat file for face detection
grayMat = inputFrame.mGray()
// detect face rectangle
getFaces( grayHeight, screenRotation )
Imgproc.putText(
imageMat,
"Java: $javaEnabled ... "+ pCounter.toString(),
Point(100.0, 100.0),
FONT_HERSHEY_PLAIN,
1.0,
YELLOW.mColor
)
// end time
frameEnd = System.currentTimeMillis()
if (pCounter < 1000) {
frameRecord[pCounter] = frameEnd - frameStart
}
pCounter++
return imageMat
}
}
The getFaces(),
/*
Draw face rectangles from faceRects
*/
fun getFaces(
grayHeight: Int,
screenRotation: Int
) {
val faceRects = MatOfRect()
// image width & height
val scrW = grayMat.width().toDouble()
val scrH = grayMat.height().toDouble()
val scale = 1.5
val neighbor = 3
// Detect faces
faceDetector!!.detectMultiScale(
grayMat, // image
faceRects, // array
scale, // scale
neighbor, // min neighbors
0, // flags,
Size(80.0, 80.0), // min size
Size(scrW, scrH) // max size
)
// Draw face rectangles
drawRects(faceRects)
// Initial landmarks
val landmarks = ArrayList<MatOfPoint2f>()
// Get landmarks
fmLbfDectector.fit(imageMat, faceRects, landmarks)
// draw landmarks
for (i in 0 until landmarks.size) {
val mLandmark = landmarks[i]
for (j in 0 until mLandmark.rows()) {
val dp = mLandmark[j, 0]
Imgproc.circle(
imageMat, // image
Point(dp[0], dp[1]), // location
2, // radius
GREEN.mColor, // color
-1 // thickness
)
}
}
}The drawRects(),
/**
* Draw rectangles from face array
*/
fun drawRects(faceRects: MatOfRect) {
for (rect in faceRects.toArray()) {
val x = rect.x.toDouble()
val y = rect.y.toDouble()
val rw = rect.width.toDouble() // rectangle width
val rh = rect.height.toDouble() // rectangle height
val w = x + rw
val h = y + rh
when (screenRotation) {
0 -> paintFace(x, y, w, h, RED)
90 -> paintFace(x, y, w, h, PINK)
180 -> paintFace(x, y, w, h, YELLOW)
270 -> paintFace(x, y, w, h, GREEN)
}
}
}No more getRatio(), I will use 1.5 as scale.
And paintFace(),
/**
* Paint one face with rectangle
*/
fun paintFace(
x: Double, //(x, y) coordinate
y: Double,
w: Double, //(w, h) coordinate
h: Double,
color: ObjColor
) {
Imgproc.rectangle(
imageMat, // image
Point(x, y), // upper corner
Point(w, h), // opposite corner
color.mColor, // RGB
2 // line thickness
)
// dot
Imgproc.circle(
imageMat, // image
Point(x, y), // center
4, // radius
BLUE.mColor, // RGB
-1, // thickness: -1 = filled in
8 // line type
)
}If you execute the above code, you can get an error about RIO out of range of non-90 Degree screen.
// Get landmarks
fmLbfDectector.fit(imageMat, faceRects, landmarks)You know, I rotate and flip image in camera view before the face is detectable. So the faceRects will cover rectangles out of the screen. I need to re-program the Rect-list for faceRects.
...
// Re-organize Rects
if (faceRects.toArray().isNotEmpty() && screenRotation != 90) {
val mTranList = TransformRects(
faceRects.toList(), scrW, scrH, screenRotation)
faceRects.fromList( mTranList.newRects )
}
// Draw face rectangles
drawRects(faceRects)
// Initial landmarks
val landmarks = ArrayList<MatOfPoint2f>()
// Get landmarks
fmLbfDectector.fit(imageMat, faceRects, landmarks)Here is added TransformRects.kt class,
/**
* Transform Rectangles by screen orientation
* newRects: new list of rectangles
* mH: height of image
* rotation: screen orientation
*/
class TransformRects(rectList: MutableList<Rect>,
mW: Double,
mH: Double,
rotation: Int) {
val newRects = mutableListOf<Rect>()
init {
for (rect in rectList) {
val x = rect.x
val y = rect.y
val rw = rect.width // rectangle width
val rh = rect.height // rectangle height
when (rotation) {
0 -> {
val xFix = mW - x - rw;
newRects.add(Rect(y, xFix.toInt(), rh, rw))
}
//90: no change
180 -> {
// fix height
val yFix = mH.toInt() - y - rh
newRects.add(Rect(yFix, x, rh, rw))
}
270 -> {
// fix height
val yFix = mH.toInt() - y - rh
newRects.add(Rect(x, yFix, rw, rh))
}
}
}
}
}Now, let’s test the rectangles with all the rotations ONLY. The face landmarks may not be correct. Commit before you start to fix landmarks.
🙂4. 68 Points of Face Landmark
There’re 68 points of a facial landmark from the iBUG 300-W dataset:
1, 🔱Jaw: 1 to 17, vs array[0..16]
2, 🌛Left eyebrow: 18 to 22, vs array[17..21]
3, 🌜Right eyebrow: 23 to 27, vs array[22..26]
4, 👃🏻Nose: 28 to 36, vs array[27..35]
5, 🐍Left eye: 37 to 42, vs array[36..41]
6, 💁🏻Right eye: 43 to 48, vs array[42..47]
7, 💋Mouth: 49 to 68, vs array[48..67]
🙈5. Prepare the Fix of Facemark
We knew the orientation of 90-degree is the default screen for the detector. All we need to fix is the rotation of 0, 180, and 270. I will mark the x-axis and y-axis the rectangle of each face to visualize for x and y location.
// Draw face rectangles
drawRects(faceRects)
// Initial landmarks
val landmarks = ArrayList<MatOfPoint2f>()
// Get landmarks
fmLbfDectector.fit(imageMat, faceRects, landmarks)// Face rectangle list
val rectList = faceRects.toList()// draw landmarks
for (i in 0 until landmarks.size) {
val mLandmark = landmarks[i]
val mRect = rectList[i]
val mX = mRect.x
val mY = mRect.y
val mX2 = mX + mRect.width
val mY2 = mY + mRect.height
//lgi ( "cRect: $mX:$mY, $mX:$mY2, $mX2:$mY, $mX2:$mY2")
// rect: x axis
Imgproc.line(
imageMat, // image
Point(mX.toDouble(), mY.toDouble()), // point a
Point(mX2.toDouble(), mY.toDouble()), // point b
WHITE.mColor, // color
4 // thickness
)
// rect: y axis
Imgproc.line(
imageMat, // image
Point(mX.toDouble(), mY.toDouble()), // point a
Point(mX.toDouble(), mY2.toDouble()), // point b
BLUE.mColor, // color
4 // thickness
)
...
The white line is X-axis. The blue line is Y-axis. The mX, mX2, mY, and mY2 are the vertices of a face rectangle.
🍳6. Fitting the Facemark(Only the Nose)
Let’s just paint the nose section to have a clear view.
// draw landmarks
for (i in 0 until landmarks.size) {
...
for (j in 27 until 36) {
val dp = mLandmark[j, 0]
val x = dp[0]
val y = dp[1]
var dot: Point = Point()
var dot2: Point = Point()
when (screenRotation) {
0 -> {
dot = Point(x, y)
Imgproc.circle(
imageMat, // image
dot, // location
1, // radius
YELLOW.mColor, // color
-1 // thickness
)
}
90-> {
dot = Point(x, y)
Imgproc.circle(
imageMat, // image
dot, // location
1, // radius
YELLOW.mColor, // color
-1 // thickness
)
}
180 -> {
dot = Point(x, y)
Imgproc.circle(
imageMat, // image
dot, // location
1, // radius
YELLOW.mColor, // color
-1 // thickness
)
}
270 -> {
dot = Point(x, y)
Imgproc.circle(
imageMat, // image
dot, // location
1, // radius
YELLOW.mColor, // color
-1 // thickness
)
}
}
}
}✔️Rotation: 90
Working fine!
✔️Rotation: 0
Let’s shoot a single face.
According to the image, the new X & Y coordinates will be
val x2 = mX + ( y - mY )
val y2 = mY + ( x - mX )Now, let’s paint the new nose.
dot2 = Point(x2, y2)
Imgproc.circle(
imageMat, // image
dot2, // location
1, // radius
PINK.mColor, // color
-1 // thickness
)Run.
No good. It’s failed to catch the noses.
✔️Rotation: 180
The 180 rotation has similar to 0 rotation.
val x2 = mX2 - ( y - mY )
val y2 = mY + ( x - mX )Let’s check the result.
It’s same mess. I cannot use 180 rotation.
✔️Rotation: 270
I hope 270 is working, too.
val y2 = mY2 - y + mY
dot2 = Point(x, y2)The result is:
It’s still not good for face recognition.
🆗7. Warning of Non-90 Rotation
So far, by using OpenCV library, we can only use 90-degree rotation to process face recognition. It has a lot of errors from other orientations. If you want to use it for commercial usage, I suggest that you need to post a warning on the rotation.
val mOrientationEventListener = object : OrientationEventListener(this) {
@RequiresApi(Build.VERSION_CODES.M)
override fun onOrientationChanged(orientation: Int) {
// Monitors orientation values to determine the target rotation value
when (orientation) {
in 45..134 -> {
screenRotation = 270
val msg = screenRotation.toString()+ BAD_ANGLE
rotation_tv.rotation = 0F
rotation_tv.text = msg
}
in 135..224 -> {
screenRotation = 180
val msg = screenRotation.toString()+ BAD_ANGLE
rotation_tv.rotation = 180.0F
rotation_tv.text = msg
}
in 225..314 -> {
screenRotation = 90
rotation_tv.rotation = 90.0F
rotation_tv.text = screenRotation.toString()
}
else -> {
screenRotation = 0
val msg = screenRotation.toString()+ BAD_ANGLE
rotation_tv.rotation = 0F
rotation_tv.text = msg
}
}
}
}...
companion object {
...
private const val BAD_ANGLE = " Bad Angle!"
}
Display as:
Before: Part 9 — Display Custom Meter
