Android + OpenCV : Part 2 — CameraView

5 min readApr 18, 2020

In March 2020, the OpenCV updated Android pack is version 3.4.9. You can download and install 3.4.9 or the newest version on OpenCV.org by redoing the steps of Part 1. In Part 2, I will insert the camera view, and listener to capture the image frame. Here is the menu:

1. Install OpenCV Manager
2. UI Code
3. Listener: CvCameraViewListener2
4. Android Permissions
5. CameraView: viewfinder
7. OpenCV Manager: BaseLoaderCallBack
8. Handle Camera Frame
9. First View on the Camera
10. Fix the View
11. Test Again

👔 1. Install OpenCV Manager

If you want to use all the functions of OpenCV, you can not ignore OpenCV Manger. But you have to install it from separate app. If you don't want it installed, please check on Part 4. Now, let’s connect your phone and open the terminal in your project. Next, you need to go to
“OpenCV library directory”\apk.

>dir
 Volume in drive D is 512 Gigabyte
 Volume Serial Number is BE4E-1582


 Directory of D:\Android\OpenCV-Android-Lib\OpenCV-android-sdk-349\apk


04/05/2020  12:40 AM    <DIR>          .
04/05/2020  12:40 AM    <DIR>          ..
12/19/2019  10:38 AM         7,956,744 OpenCV_3.4.9_Manager_3.49_arm64-v8a.apk
12/19/2019  10:31 AM        11,953,673 OpenCV_3.4.9_Manager_3.49_armeabi-v7a.apk
12/19/2019  10:34 AM        11,883,427 OpenCV_3.4.9_Manager_3.49_armeabi.apk
12/19/2019  10:55 AM        15,910,690 OpenCV_3.4.9_Manager_3.49_mips.apk
12/19/2019  10:51 AM         8,518,751 OpenCV_3.4.9_Manager_3.49_mips64.apk
12/19/2019  10:47 AM        22,178,553 OpenCV_3.4.9_Manager_3.49_x86.apk
12/19/2019  10:43 AM        21,849,970 OpenCV_3.4.9_Manager_3.49_x86_64.apk
12/06/2019  06:53 PM               978 readme.txt
               8 File(s)    100,252,786 bytes
               2 Dir(s)  425,404,383,232 bytes free


D:\Android\OpenCV-Android-Lib\OpenCV-android-sdk-349\apk>

Next, you need to check CPU type of your phone. Here is mine: arm64-v8a.

>adb shell getprop ro.product.cpu.abi
arm64-v8a

Let’s find connected devices.

>adb devices
List of devices attached
ZY326LVCZQ      device

Please install the right type of manager APK.

>adb -s ZY326LVCZQ install OpenCV_3.4.9_Manager_3.49_arm64-v8a.apk
Performing Streamed Install
Success

Check your phone for installation:

😙2. UI Code

At activity_main.xml, please add JavaCamera2View.

<androidx.constraintlayout.widget.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context=".MainActivity">

    <org.opencv.android.JavaCamera2View
        android:id="@+id/cameraView"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        app:show_fps="true"
        app:layout_constraintBottom_toBottomOf="parent"
        app:layout_constraintLeft_toLeftOf="parent"
        app:layout_constraintRight_toRightOf="parent"
        app:layout_constraintTop_toTopOf="parent" />

    <TextView
        ... />

</androidx.constraintlayout.widget.ConstraintLayout>

id: cameraView
show_fps: true

👂 3. Listener: CvCameraViewListener2

At MainActivity.kt, let’s add the new listener.

class MainActivity : AppCompatActivity(), CameraBridgeViewBase.CvCameraViewListener2 {

Now, Alt+Enter to implement the new member functions.

I select them all.

override fun onCameraViewStarted(width: Int, height: Int) {
    TODO("Not yet implemented")
}

override fun onCameraViewStopped() {
    TODO("Not yet implemented")
}

override fun onCameraFrame(inputFrame: CameraBridgeViewBase.CvCameraViewFrame?): Mat {
    TODO("Not yet implemented")
}

Very clear, OpenCV runs as: start->frame->stop. You just insert the code one by one. Unlike Part 1, I move the checkOpenCV()

private fun checkOpenCV(context: Context) {

        if (OpenCVLoader.initDebug()) {
            shortMsg(context, OPENCV_SUCCESSFUL)
            lgi("OpenCV started...")
        } else {
            shortMsg(context, OPENCV_FAIL)
            lge(OPENCV_FAIL)
        }
}

out of companion object.

companion object {

        val TAG = "MYLOG " + MainActivity::class.java.simpleName
        fun lgd(s: String) = Log.d(TAG, s)
        fun lge(s: String) = Log.e(TAG, s)
        fun lgi(s: String) = Log.i(TAG, s)

        fun shortMsg(context: Context, s: String) =
            Toast.makeText(context, s, Toast.LENGTH_SHORT).show()

        // messages:
        private const val OPENCV_SUCCESSFUL = "OpenCV Loaded Successfully!"
        private const val OPENCV_FAIL = "Could not load OpenCV!!!"
        private const val OPENCV_PROBLEM = "There's a problem in OpenCV."
        private const val PERMISSION_NOT_GRANTED = "Permissions not granted by the user."

}

🔑4. Android Permissions

You need permission to use the camera. At AndroidManisfest.xml,

<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="com.homan.huang.opencvcamerademo">

    <uses-permission android:name="android.permission.CAMERA" />
    <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
    <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
    <uses-permission android:name="android.permission.RECORD_AUDIO" />
    <uses-permission android:name="android.permission.ACCESS_FINE_LOCATION" />

Back to MainActivity.kt,

// Permission vars:
private const val REQUEST_CODE_PERMISSIONS = 111
private val REQUIRED_PERMISSIONS = arrayOf(
    CAMERA,
    WRITE_EXTERNAL_STORAGE,
    READ_EXTERNAL_STORAGE,
    RECORD_AUDIO,
    ACCESS_FINE_LOCATION
)

class MainActivity : AppCompatActivity(), CameraBridgeViewBase.CvCameraViewListener2 {

    ...

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        window.clearFlags(FLAG_FORCE_NOT_FULLSCREEN)
        window.setFlags(FLAG_FULLSCREEN, FLAG_FULLSCREEN)
        window.addFlags(FLAG_KEEP_SCREEN_ON)
        setContentView(R.layout.activity_main)

        // Request camera permissions
        if (allPermissionsGranted()) {
            checkOpenCV(this)
        } else {
            ActivityCompat.requestPermissions(
                this,
                REQUIRED_PERMISSIONS,
                REQUEST_CODE_PERMISSIONS )
        }
        ...
    }

...
    /**
     * Process result from permission request dialog box, has the request
     * been granted? If yes, start Camera. Otherwise display a toast
     */
    override fun onRequestPermissionsResult(
        requestCode: Int, permissions: Array<String>, grantResults: IntArray
    ) {
        if (requestCode == REQUEST_CODE_PERMISSIONS) {
            if (allPermissionsGranted()) {
                checkOpenCV(this)
            } else {
                shortMsg(this, PERMISSION_NOT_GRANTED)
                finish()
            }
        }
    }

    /**
     * Check if all permission specified in the manifest have been granted
     */
    private fun allPermissionsGranted() = REQUIRED_PERMISSIONS.all {
        ContextCompat.checkSelfPermission(baseContext, it) == PackageManager.PERMISSION_GRANTED
    }

    ...
}

👀 5. CameraView: viewfinder

Let’s insert CameraView in MainActivity.kt.

// view
private val viewFinder by lazy { findViewById<JavaCameraView>(R.id.cameraView) }

Setup the camera view:

override fun onCreate(savedInstanceState: Bundle?) {
    ...

    viewFinder.visibility = SurfaceView.VISIBLE
    viewFinder.setCameraIndex(
        CameraCharacteristics.LENS_FACING_FRONT)
    viewFinder.setCvCameraViewListener(this)

You need to disable the view in onPause() and onDestroy().

override fun onPause() {
    super.onPause()
    viewFinder?.let { viewFinder.disableView() }
}

override fun onDestroy() {
    super.onDestroy()
    viewFinder?.let { viewFinder.disableView() }
}

👔7. OpenCV Manager: BaseLoaderCallBack

Next, we need a callback from OpenCV Manager to handle the OpenCV library. The BaseLoaderCallback helps us to find the library after OpenCVLoader fails to load the library. Please check this image.

Let’s insert the callback.

// openCV callback
lateinit var cvBaseLoaderCallback: BaseLoaderCallback

override fun onCreate(savedInstanceState: Bundle?) {
    ...

    cvBaseLoaderCallback = object : BaseLoaderCallback(this) {
        override fun onManagerConnected(status: Int) {

            when (status) {
                SUCCESS -> {
                    lgi(OPENCV_SUCCESSFUL)
                    shortMsg(this@MainActivity, OPENCV_SUCCESSFUL)
                    viewFinder.enableView()
                }

                else -> super.onManagerConnected(status)
            }
        }
    }

}

You need to resume the callback at onResume().

override fun onResume() {
    super.onResume()
    OpenCVLoader.initAsync(OpenCVLoader.OPENCV_VERSION, this, cvBaseLoaderCallback)
}

📷 8. Handle Camera Frame

The OpenCV stores the image in Mat type.

// image storage
lateinit var imageMat: Mat

Let’s insert the code in the listener’s member.

override fun onCameraViewStarted(width: Int, height: Int) {
    imageMat = Mat(width, height, CvType.CV_8UC4)
}

override fun onCameraViewStopped() {
    imageMat.release()
}

override fun onCameraFrame(inputFrame: CameraBridgeViewBase.CvCameraViewFrame?): Mat {
    imageMat = inputFrame!!.rgba()
    return imageMat
}

😎9. First View on the Camera

Let’s run.

Not so good, everything is looking sideways. How can we fix it?

🔨10. Fix the View

If you rotate the image in frame function, the app will run very slow. Thanks to Mike Heavers, we can fix the view in the module. For many years, I still don’t understand why OpenCV SDK doesn’t fix the problem. Please check his article and give him a clap. The fix is: openCV-portait-camera-android.java