Use Your TensorFlow Mobile Model in an Android App

Nowadays, modern mobile devices are extremely powerful and enable new approaches. Even if it sounds like a platitude, it is clear that some of these approaches are very interesting and are already used in some apps. One of these approaches is machine learning. In this post we want to show how to use TensorFlow Mobile to recognize houseplants in an Android app. Using a network directly on a device has several advantages. Firstly the app can process the data offline without the need of processing power on a server or bandwidth. Secondly, no sensitive data has to be sent through the network.

Previously on this blog …

This blog post is part of a TensorFlow Mobile series. The first post TensorFlow Mobile: Training and Deploying a Neural Network has focused on how to deploy a neural network and train it for our use case. In this post we’ll show how to import and integrate such a model into the app.

So as a short summary, what happened in the previous post?

• We have chosen TensorFlow Mobile as solution to create the neural network. There are two reasons, why we used TensorFlow Mobile: TensorFlow Lite is still in developer preview and furthermore TensorFlow Mobile has a bigger feature set.
• With TensorFlow Mobile we trained a model to classify images of houseplants. Our model is able to distinguish between 26 different houseplants.
• This model has been further processed for mobile use.  At the end we have an optimized model, which is available as a Protocol Buffers (protobuff) file to use in our app.

Houseplant app

The job of the application is quite simple: Recognize houseplants with Android devices using a trained TensorFlow model. To do so the application should be capable of taking a picture of the plant and then indicate which plant it is most likely. The animated gif below shows what the end result looks like.

Getting started

Now that the goal is clear, let’s get going with the implementation. At first glance, you might think that we have to create a lot of boiler plate code to develop such an app. Fortunately, Google offers four different TensorFlow examples for Android. These examples are good starting points, so we don’t have to start from scratch. The TF Classify example, which uses the Android camera to classify images in real time, roughly corresponds to the requirements of our app. That is why we will build the app on top of this example.

A disadvantage of the four TensorFlow examples is, that they are all packed into one repository. Instead of pulling the needed example out of this repository, we will rather use the more minimal repository provided for the TensorFlow for poets tutorial. The repository only includes the TF Classify example and is an ideal starting point for our houseplant app.

Once we have checked out the TF Classify project, the model and labels file have to be replaced with ours. These files are located in  src/main/assets/graph.pb and src/main/assets/labels.txt

Next we need to adjust the field variables in the ClassifierActivity to match our model.

We use 224 as our input size, because the ImageNet we trained in part one of this series was trained with an image size of 224 pixels. Also, it is important to set the values for the images mean and standard deviation based on the used model.

The input and output names are the names of the input and output nodes of the model.

If you changed the name or the location of the model and labels file you will have to change the path of the file fields as well.

Classifying plants with TensorFlow Mobile

The classification is based on the image data received from the camera of the device. Therefore, the application is using the Camera2 API from Android. The API is quite extensive, but luckily all the needed API code is packed into the abstract class CameraActivity. The class takes care of the permission handling needed for accessing the camera and is setting up the CameraConnectionFragment. This fragment handles all camera relevant configuration and also opens and closes the camera connection and the image capturing. The fragment also registers the parent activity as listener for new images.

All the classification is handled in the ClassifierActivity which is extending the CameraActivity. The ClassifierActivity receives the image data through the onImageAvailable method which is the callback method from the OnImageAvailableListener on which the Activity has been registered by the CameraConnectionFragment.

Whenever an image is available, the callback method is called and the ClassifierActivity can process the image data. Then, the data is acquired from the ImageReader passed as parameter. Before the image data can be passed to a Classifier it is converted into a Bitmap. As result, the classification provides a List of Recognitions with its confidence.

In our case the Classifier which recognizes the image is the TensorFlowImageClassifier. Such a Classifier is using the TensorFlow Android API and our model to make an assumption what is displayed in the captured image.

Useful improvements

The example application from Google is constantly classifying the input from the camera. Because we wanted to give the user time to position the camera, we added a capture button and only analyzed the image on click.

Also, to improve our model in the future, we add a feature to get feedback from the user. That is why we present the top results on a separate page where the user can confirm if one of the suggestions is right. When the user confirms the result, the image is uploaded to a cloud storage. For the upload, we use Firebase Cloud Storage, because it is easy to use within Android applications. Before uploading, the image gets annotated by adding the probability result of the model to the EXIF tags. Furthermore, the images are uploaded in different folders named after the detected plants. The categorization into folders is made depending on the result confirmed by the user. These collected images can be used in the future to generate a better model. In this case, the ordering and annotations will help to filter and improve the image base. Keep in mind that you should ask the user for permission when uploading images to a cloud storage!

Conclusion

Overall, it is impressive how fast and easily a model can be integrated into an Android application. The given tutorials and examples from Google are a good way to get started and learn more about using TensorFlow. When you have a model prepared and optimized for mobile, integrating it into an Android application is not a big deal. In our example the most challenging part was getting to grips with the Camera2 API. It might take some time to understand the API, because of its extensiveness.

If you decide to use TensorFlow Lite instead of TensorFlow Mobile, as we did in this series, you should take a look at Googles ML Kit. With ML Kit the whole process is getting easier: It offers an easy to understand API, you can directly upload custom models via the web interface and they even have a model updating mechanism, which doesn’t require the user to update the whole application. Although TensorFlow Lite has a smaller feature set, if it suffices your use case, it is a great alternative and ML Kit simplifies the model handling enormously.

Join us!

Whether you’re interested in Android development or Machine Learning, take a look at our current job offers and find what best suits your interests. Join us and implement cutting edge technology in production level projects with our broad selection of customers!