Did you know that a set of computer algorithms can process a video stream in a way that allows them to detect criminal activity, control traffic jams, and even automatically detect events in sports broadcasts? Thanks to the application of machine learning (ML), the idea of acquiring so much data from a simple video doesn’t seem that unrealistic. In this article, we want to share our experience applying the pre-built logic of a machine learning algorithm for object detection and segmentation to a video.
In particular, we talk about how to configure Google Colaboratory for solving video processing tasks with machine learning. You’ll learn how to use this Google service and the free NVIDIA Tesla K80 GPU that it provides for achieving your own goals in training neural networks. This article will be useful for people who are getting familiar with machine learning and considering working with image recognition and video processing.
Apriorit was tasked with recognizing people in a video recording with the help of machine learning (ML) algorithms. We decided to begin with the basics. First, let’s consider what a video recording actually is.
From a technical point of view, any video recording consists of a series of still images in a particular format that are compressed with a video codec. Consequently, object recognition on a video stream comes down to splitting the stream into separate images, or frames, and applying a pre-trained ML image recognition algorithm to them.
To do this, we decided to use a neural network from the Mask_R-CNN repository for classifying single images. The repository contains an implementation of a convolutional neural network on Python 3, TensorFlow, and Keras. Let’s see what came out of this plan.
We developed and implemented a simple sample of Mask_RCNN that received a picture as the input and recognized objects in it. We created a sample on the basis of the demo.ipynb description taken from the Mask_R-CNN repository. Here’s the code of our sample:
In this sample, /content/drive/My Drive/Colab Notebooks/MRCNN_pure is the path to our repository with Mask_R-CNN. As a result, we got the following:
This part of the demo code looks through the images folder, randomly selects an image, and loads it to our neural network model for classification:
Let’s modify the Mask_R-CNN sample to make it recognize all images in the images folder:
After running the demo code for five minutes, the console displayed the following output:
Initially, we ran the demo code on a computer with an Intel Core i5 and 8GB of RAM without a discrete graphics card. The code crashed each time in different places, but most often it crashed in the TensorFlow framework during memory allocation. Moreover, any attempts to run any other software during the image recognition process slowed down the computer to the point of being useless.
Thus, we faced a serious problem: any experiments in getting familiar with ML required a powerful graphics card and more hardware resources. Without this, we couldn’t run any other tasks while recognizing a large number of images.
We decided to expand our hardware resources by using the Colaboratory service by Google, which is also known as Colab. Google Colab is a free cloud service that provides use of a CPU and GPU as well as a preconfigured virtual machine instance. Specifically, Google offers the NVIDIA Tesla K80 GPU with 12GB of dedicated video memory, which makes Colab a perfect tool for experimenting with neural networks.
Before explaining how to work with this Google service, we’d like to underline other beneficial Colaboratory features.
By choosing Colab for your ML experiments, you’ll get:
- support for Python 2.7 and Python 3.6 so you can improve your coding skills;
- the ability to work with Jupyter notebook so you can create, edit, and share your .ipynb files;
- the ability to connect to a Jupyter runtime using your local machine;
- many pre-installed libraries including TensorFlow, Keras, and OpenCV, as well as the possibility to interact with your custom libraries in Google Colaboratory;
- upload functionality so you can add your trained model;
- integration with GitHub so you can load public GitHub notebooks or save a copy of your Colab file to GitHub;
- simple visualization with such popular libraries as matplotlib;
- forms that can be used to parameterize code;
- the ability to store Google Colab notebooks in your Google Drive.
In order to start using the Google Colab GPU, you just need to provide access to your .ipynb script that’s implemented in a Docker container. The Docker container is assigned to you only for 12 hours. All scripts created by you are stored by default in your Google Drive in the Colab Notebooks section, which is automatically created as you connect to Colaboratory. After the expiry of 12 hours, all of your data in the container will be deleted. You can avoid this by mounting your Google Drive in your container and working with it. Otherwise, the file system of the Docker image will be available only for a limited period of time.
Let’s first explain how you can create your .ipynb notebook. Open Google Colaboratory here, select the Google Drive section, and click NEW PYTHON 3 NOTEBOOK:
Rename your notebook whatever you want by clicking on the file name. Now you need to choose your hardware. To do this, just go to the Edit section, find Notebook Settings, select GPU as the Hardware accelerator, and save changes by clicking SAVE.
After the new settings are saved, a Docker container with a discrete graphics card will become available to you. You’ll be notified about this with a “Connected” message in the upper right of the page:
If you don’t see this message, select Connect to hosted runtime.
Now you can mount your Google Drive to this container in order to relocate the source code and save the results of your work in the container. To do this, just copy the code below in the first table cell and press the Play button (or Shift + Enter).
Mount Google Drive by running this code:
You’ll get a request for authorization. Click the link, authorize, copy the verification code, paste it in the text box in your .ipynb script, and press Enter. If authorization is successful, your Google Drive will be mounted under the path /content/drive/My Drive. To follow the file tree, select Files in the left-hand menu.
Now you have a Docker container with the Tesla K80 GPU, your Google Drive as file storage, and the .ipynb notebook for script execution.
Now we’ll describe how to run our Mask_R-CNN sample for object recognition in Google Colab. We upload the Mask_RCNN repository to our Google Drive following the /content/drive/My Drive/Colab Notebooks/ path.
Then we add our sample code to the .ipynb script. When you do this, don’t forget to change your path to the Mask_RCNN folder like this:
If you do everything right, the results of code execution will provide you with an image where all objects are detected and recognized.
You can also modify the sample code to make it process all of the test images:
Using object detection in Google Colab, we received the results with recognized objects quickly, while our computer continued to perform as usual even during the image recognition process.
Let’s see how we applied this method for recognizing people in a video stream. We uploaded a test video file to our Google Drive. To train our script to work with a video stream, we used OpenCV, a popular open source computer vision library.
We don’t need the whole code that read and implemented the recognition model on one image. So instead of opening the video file, we run the video stream and move its pointer to the 1,000th frame as there are no objects to recognize in the intro of the recording.
Then we process 20,000 frames with our neural network model. The OpenCV object allows us to get images by frame from the video file using the read() method. The received image is passed to the model.detect() method and the results are visualized with the visualize.display_instances() function.
However, we faced a problem: the display_instances() function from the Mask_RCNN repository reflects detected objects in the image, but the image doesn’t returned. We decided to simplify the display_instances() function and make it return the image with displayed objects:
After processing, the frames should be bound back together into a new video file. We can also do this with the OpenCV library. All we need to do is allocate the VideoWriter object from the OpenCV library:
using the type of video we provided for the input. We get the video file type with the help of the ffprobe command:
The received object can be used for per-frame recording: writer.write(masked_frame).
At the beginning of the script, we need to specify paths to the target video files for processing: VIDEO_STREAM and VIDEO_STREAM_OUT.
Here’s the full script that we developed for video recognition:
After successful execution of the script, our video file with recognized images will be located in the path specified in VIDEO_STREAM_OUT. We run our system using a piece of a movie and receive a video file with recognized objects:
In this article, we’ve shown you how we took advantage of Google Colab and explained how you can do the following:
- Use a free Tesla K80 GPU provided by Google Colab
- Classify images with the Mask_RCNN neural network and Google Colab
- Classify objects in a video stream using Mask_RCNN, Google Colab, and the OpenCV library
At Apriorit, we have a team of dedicated professionals who can use machine learning technologies to your benefit. Image and video processing are among the key areas we’re investing our efforts in now. If you have any breathtaking ideas in mind, we’ll be glad to turn them into reality!