Many people use the Python language to do Image recognition and other AI tasks. There has been a huge improvement lately in order to make some popular frameworks available in R. Of course, it’s possible to run Python code inside R (using reticulate) but it’s simpler to code in just one language. Keras is a popular framework which is build on top of Tensorflow. It is available in R package keras (Rstudio documentation). this blog, I will show how I have build a Shiny application to recognize objects in an image.

Shiny app

The functionality of the application looks like this:

1. Upload an image to Shiny App
2. Perform image recognition using Keras
3. Plot detected objects in a Wordcloud and show scores in a table.

I will explain more about task 2, since that is the main functionality. Keras has a number of pretrained models for image classification. They are trained on a large dataset called ImageNet. I am using the RESNET 50 model since it’s simple to use.

Resnet 50

RESNET 50 is a Residual Network with 50 layers. As the name of the network indicates, this network uses residual learning. In general, in a deep convolutional neural network, several layers are stacked and are trained to the task at hand. The network learns several low/mid/high level features at the end of its layers. In residual learning, instead of trying to learn some features, we try to learn some residual. This residual can be simply understood as subtraction of feature learned from input of that layer. It has proved that training this form of networks is easier than training simple deep convolutional neural networks. Also the accuracy is better when adding more layers,compared to a standard neural network.

The steps needed to perform the image recognition:

• Load the image into memory and convert the size to 224×224, since that is the default size of RESNET 50.
• Convert the image to a 3D array representation
• Reshape the image array
• Preprocess the image array
• Predict the object & scores using the RESNET 50 model
• Decode the predictions to a data frame format

The model performs quite good on average, the object with the highest score should be right in about 75% of all uploaded images. The performance of the RESNET 50 model and other models can be found on the Keras website.

Have a look at the application by clicking on the (animated) screenshot below or live at shinyapps.io