Integrating Camera2 API on Android, feat. Kotlin
Today, we are going to look at what it takes to get a camera integrated using Android’s creatively named Camera2 API. Specifically, we are going to do the minimum to get a SurfaceView displaying a preview of our input source, and simultaneously set up an ImageReader to process individual frames from the same source.
As you know, when building something on the Android platform, there are a lot of unknowns to account for. There are so many different device types, and there is just no predicting what the services and hardware of that device will look like. That’s why we need to interface with APIs that resolve the values of those unknowns for us, and we need to safely handle all eventualities in our code. Camera2 makes all this simple for us.
So the first thing you might want to inquire when running your application on some device is, what kind of devices are available to me? Does the device have both a front facing as well as back facing camera? What resolutions does it support? We answer these questions using the CameraManager.
By the way, I’ve started a new, empty project, I’m going to start building from there within MainActivity. You don’t need any external dependencies to use this API.
Requesting Permission
Actually let’s backtrack. First we need to request permission to use the camera. Here’s a helper to help you do that:
/** Helper to ask camera permission. */
object CameraPermissionHelper {
private const val CAMERA_PERMISSION_CODE = 0
private const val CAMERA_PERMISSION = Manifest.permission.CAMERA
/** Check to see we have the necessary permissions for this app. */
fun hasCameraPermission(activity: Activity): Boolean {
return ContextCompat.checkSelfPermission(activity, CAMERA_PERMISSION) == PackageManager.PERMISSION_GRANTED
}
/** Check to see we have the necessary permissions for this app, and ask for them if we don't. */
fun requestCameraPermission(activity: Activity) {
ActivityCompat.requestPermissions(
activity, arrayOf(CAMERA_PERMISSION), CAMERA_PERMISSION_CODE)
}
/** Check to see if we need to show the rationale for this permission. */
fun shouldShowRequestPermissionRationale(activity: Activity): Boolean {
return ActivityCompat.shouldShowRequestPermissionRationale(activity, CAMERA_PERMISSION)
}
/** Launch Application Setting to grant permission. */
fun launchPermissionSettings(activity: Activity) {
val intent = Intent()
intent.action = Settings.ACTION_APPLICATION_DETAILS_SETTINGS
intent.data = Uri.fromParts("package", activity.packageName, null)
activity.startActivity(intent)
}
}Implement this callback:
override fun onRequestPermissionsResult(requestCode: Int, permissions: Array<out String>, grantResults: IntArray) {
if (!CameraPermissionHelper.hasCameraPermission(this)) {
Toast.makeText(this, "Camera permission is needed to run this application", Toast.LENGTH_LONG)
.show()
if (!CameraPermissionHelper.shouldShowRequestPermissionRationale(this)) {
// Permission denied with checking "Do not ask again".
CameraPermissionHelper.launchPermissionSettings(this)
}
finish()
}
recreate()
}Make sure you’ve added this line to the Manifest file:
<uses-permission android:name="android.permission.CAMERA" />Then call:
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
if (!CameraPermissionHelper.hasCameraPermission(this)) {
CameraPermissionHelper.requestCameraPermission(this)
return
}
}Accessing Devices with CameraManager
private fun startCameraSession() { val cameraManager = getSystemService(Context.CAMERA_SERVICE) as CameraManager if (cameraManager.cameraIdList.isEmpty()) {
// no cameras
return
} val firstCamera = cameraIdList[0] cameraManager.openCamera(firstCamera, object: CameraDevice.StateCallback() {
override fun onDisconnected(p0: CameraDevice) { }
override fun onError(p0: CameraDevice, p1: Int) { }
override fun onOpened(cameraDevice: CameraDevice) {
// use the camera
val cameraCharacteristics = cameraManager.getCameraCharacteristics(cameraDevice.id)
cameraCharacteristics[CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP]?.let { streamConfigurationMap ->
streamConfigurationMap.getOutputSizes(ImageFormat.YUV_420_888)?.let { yuvSizes ->
val previewSize = yuvSizes.last()
}
}
}
}, Handler { true })}
Here, we use the manager to grab the first camera in the list. We then call CameraManager.openCamera(), passing the id to gain access to that camera device. Once it opens successfully, we call CameraManager.getCameraCharacteristics to inquire about the specifics of that camera device. If you wanted to be more particular about which camera to use, you could call getCameraCharacteristics before opening the camera to check if it has the properties you require.
In this example, I retrieve the camera’s StreamConfigurationMap, which contains all the information about supported output formats for the device. Next I grab the YUV_420_888 format, if it exists, as that is what was needed for my project. You can go with some other format as needed. I don’t know much about that. I then grab the last size from that list, which will be the lowest resolution size as needed for this example.
Next, we need to do something crucially important which a newcomer might not expect. We need to check if the orientation of the Android device and the orientation of the data being output by the camera, are swapped! It is possible that the device is oriented portrait, but the camera is still outputting images which are oriented landscape according to the camera sensors. The Camera2 sample project provides a method for determining if this is the case:
private fun areDimensionsSwapped(displayRotation: Int, cameraCharacteristics: CameraCharacteristics): Boolean {
var swappedDimensions = false
when (displayRotation) {
Surface.ROTATION_0, Surface.ROTATION_180 -> {
if (cameraCharacteristics.get(CameraCharacteristics.SENSOR_ORIENTATION) == 90 || cameraCharacteristics.get(CameraCharacteristics.SENSOR_ORIENTATION) == 270) {
swappedDimensions = true
}
}
Surface.ROTATION_90, Surface.ROTATION_270 -> {
if (cameraCharacteristics.get(CameraCharacteristics.SENSOR_ORIENTATION) == 0 || cameraCharacteristics.get(CameraCharacteristics.SENSOR_ORIENTATION) == 180) {
swappedDimensions = true
}
}
else -> {
// invalid display rotation
}
}
return swappedDimensions
}Then back in startCameraSession:
val previewSize = yuvSizes.last()// cont.
val displayRotation = windowManager.defaultDisplay.rotationval swappedDimensions = areDimensionsSwapped(displayRotation, cameraCharacteristics)// swap width and height if needed
val rotatedPreviewWidth = if (swappedDimensions) previewSize.height else previewSize.width
val rotatedPreviewHeight = if (swappedDimensions) previewSize.width else previewSize.height
Setting up the SurfaceView
At this point, we are ready to integrate with a SurfaceView. To do this is actually quite simple. First we replace the autogenerated “Hello World!” TextView with a SurfaceView:
<?xml version="1.0" encoding="utf-8"?>
<android.support.constraint.ConstraintLayout
xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:tools="http://schemas.android.com/tools"
xmlns:app="http://schemas.android.com/apk/res-auto"
android:layout_width="match_parent"
android:layout_height="match_parent"
tools:context=".MainActivity">
<SurfaceView
android:id="@+id/surfaceView"
android:layout_width="match_parent"
android:layout_height="match_parent"
app:layout_constraintBottom_toBottomOf="parent"
app:layout_constraintEnd_toEndOf="parent"
app:layout_constraintStart_toStartOf="parent"
app:layout_constraintTop_toTopOf="parent" />
</android.support.constraint.ConstraintLayout>We already have the output size given by our camera device characteristics, so we just set the preview’s fixed size on the holder to be the output size:
surfaceView.holder.setFixedSize(rotatedPreviewWidth, rotatedPreviewHeight)Now we just need to wire up the input source, namely our camera device, and the surface. We do this with CameraCaptureSession.CaptureCallback():
val previewSurface = surfaceView.holder.surface
val captureCallback = object : CameraCaptureSession.StateCallback()
{
override fun onConfigureFailed(session: CameraCaptureSession) {}
override fun onConfigured(session: CameraCaptureSession) {
// session configured
val previewRequestBuilder = cameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW)
.apply {
addTarget(previewSurface)
} session.setRepeatingRequest(
previewRequestBuilder.build(),
object: CameraCaptureSession.CaptureCallback() {},
Handler { true }
)
}
}
cameraDevice.createCaptureSession(mutableListOf(previewSurface), captureCallback, Handler { true })
If you try to run this, everything will build fine. But you won’t see any preview appearing… that’s because we never called startCameraSession() yet! Actually it’s not quite that simple, because SurfaceView must prepare itself asynchronously, and it might not be ready even after the Activity is interactive.
So we need a callback:
val surfaceReadyCallback = object: SurfaceHolder.Callback {
override fun surfaceChanged(p0: SurfaceHolder?, p1: Int, p2: Int, p3: Int) { }
override fun surfaceDestroyed(p0: SurfaceHolder?) { }
override fun surfaceCreated(p0: SurfaceHolder?) {
startCameraSession()
}
}Back in onCreate(), accessing the same synthetic view reference enabled by Kotlin:
surfaceView.holder.addCallback(surfaceReadyCallback)Now when you run, you should see a nice little preview appearing…
Processing Images with ImageReader
Let’s quickly do the scaffolding for adding another output source to our application: ImageReader. The cool thing about Camera2 is it’s facility for wiring in any number of inputs to various outputs. So we can display a preview and process images from the very source:
//cont.
surfaceView.holder.setFixedSize(rotatedPreviewWidth, rotatedPreviewHeight)
// Configure Image Reader
val imageReader = ImageReader.newInstance(rotatedPreviewWidth, rotatedPreviewHeight,
ImageFormat.YUV_420_888, 2)imageReader.setOnImageAvailableListener({
// do something
}, Handler { true })
We configure ImageReader with the same image format as the surface view.
ImageReader renders its data to a Surface, which we can access directly:
// cont.
val previewSurface = surfaceView.holder.surface
val recordingSurface = imageReader.surfaceAdd it to our callback. By doing this we ensure that for every image that goes to the preview, one also gets sent to the ImageReader:
// cont.
val previewRequestBuilder = camera.createCaptureRequest(TEMPLATE_PREVIEW).apply {
addTarget(previewSurface)
addTarget(recordingSurface)
}And add it to the list of surfaces in our session:
// cont.
cameraDevice.createCaptureSession(mutableListOf(previewSurface, recordingSurface), captureCallback, Handler { true })Then, within the lambda of the onAvailableListener, we can now process image data in some way:
imageReader.setOnImageAvailableListener({
// do something
imageReader.acquireLatestImage()?.let { image ->
// process Image
}
}, Handler { true })The resulting image will be an Image, an object which contains a buffer of the input image, with layers and everything you need to disassemble and recombine every single bit of the image if you so desire.
That’s it! I hope this simplified your process of understanding this API. Be sure to check out the primary sources, and especially the sample project, for further clarifications. Also a quick blurb about my applications in which the above code is used:
KanjiReader is an image processing application that let’s you translate Japanese Kanji instantaneously and see a readout of its meanings and readings.
Android: https://play.google.com/store/apps/details?id=tylerwalker.io.kanjireader
iOS: https://itunes.apple.com/us/app/kanji-reader/id1457506025?mt=8
I’m going to go eat a sandwich now. As always, don’t hesitate to comment if you have questions, or just want to talk. Thanks!