Artificial Intelligence for Image Processing: Methods, Techniques, and Tools

Today’s image processing solutions powered by artificial intelligence (AI) can do things that were unimaginable a few years ago. Advanced authentication systems rely on image processing for facial recognition, while providers of online services and mobile applications can enhance their solutions with such trending features as restoration of old photos, automated image and video editing, and synthetic image generation. 

In this article, we talk about digital image processing and AI and describe some AI image processing tools and techniques for developing intelligent applications. We also take a look at the most popular neural network models for working with images and videos. 

This article will be useful for technical leaders and development teams exploring the capabilities of modern AI technologies for computer vision and image processing. Information from this article will be a solid starting point in researching possible solutions for applications that require processing digital images and extracting useful data from them.

Basics of digital image processing

Generally speaking, image processing is manipulating an image in order to enhance it or extract information from it. There are two methods of image processing:

• Analog image processing is used for processing physical photographs, printouts, and other hard copies of images
• Digital image processing is used for manipulating digital images with the help of computer algorithms

In both cases, the input is an image. For analog image processing, the output is always an image. For digital image processing, however, the output may be an image or information associated with that image, such as data on features, characteristics, bounding boxes, or masks.

Today, image processing is widely used in medical visualization, biometrics, self-driving vehicles, gaming, surveillance, law enforcement, and other spheres. 

Here are some of the main purposes of image processing:

• Visualization — Represent processed data in an understandable way. For instance, giving visual form to objects that aren’t visible
• Image sharpening and restoration — Improve the quality of processed images
• Image retrieval — Help with image search
• Object measurement — Measure objects in an image
• Pattern recognition — Distinguish and classify objects in an image, identify their positions, and understand the scene

Digital image processing includes eight key phases:

Let’s look closer at what activities are usually performed at each of these phases.

1. Image acquisition — Capture an image with a sensor (such as a camera) and convert it into a manageable entity (for example, a digital image file). One popular image acquisition method is scraping.

2. Image enhancement — Improve the quality of an image in order to extract hidden information from it for further processing.

3. Image restoration — Remove possible corruptions from an image in order to get a cleaner version. This process is mostly based on probabilistic and mathematical models and can be used to get rid of blur or noise, generate missing pixels, fix camera misfocus, remove watermarks, and eliminate other image characteristics that may harm the training of a neural network.

4. Color image processing — Improve image quality and analyze image content based on color information. Depending on the image type, we can talk about pseudocolor processing (when colors are assigned grayscale values) or RGB processing (for images acquired with a full-color sensor).

5. Image compression and decompression — Reduce or restore the size and resolution of an image. These techniques are often used for image augmentation when you slightly change an original image to extend your dataset with quality relevant data. Image augmentation can help improve the way your neural network model generalizes data and make sure it provides high-quality results.

6. Morphological processing — Describe the shapes and structures of objects in an image to create datasets for training AI models. In particular, morphological analysis and processing can be applied at the annotation stage, when you describe what you want your AI model to detect or recognize.

7. Object recognition — Identify specific features of particular objects in an image. AI-based image recognition often uses such techniques as object detection, object recognition, and segmentation. This technology is at the core of solutions like driverless automotive systems, medical diagnosis systems, and AI-powered surveillance.

8. Representation and description — Visualize and describe processed data. Using special visualization tools, you can turn arrays of numbers and values — the raw output of an AI system — into readable images suitable for further analysis.

As each of these phases requires processing massive amounts of data, you can’t perform them manually. The use of AI and machine learning (ML) boosts both the speed of data processing and the quality of the final result. For instance, with the help of AI platforms, you can successfully accomplish such complex tasks as object detection, facial recognition, and text recognition. But of course, in order to get high-quality results, it’s important to pick the right methods and tools for image processing.

In the next section, we overview the key methods, libraries, and frameworks you can use to solve image processing tasks.

Image processing methods, techniques, and tools

Most images captured with regular sensors require preprocessing, as they can be misfocused or contain too much noise. Filtering and edge detection are two of the most common methods for processing digital images.

• Filtering is used for enhancing and modifying the input image. With the help of different filters, you can emphasize or remove certain features in an image, reduce image noise, and so on. Popular filtering techniques include linear filtering, median filtering, and Wiener filtering.
• Edge detection uses filters for image segmentation and data extraction. By detecting discontinuities in brightness, this method helps to find meaningful edges of objects in processed images. Canny edge detection, Sobel edge detection, and Roberts edge detection are among the most popular edge detection techniques.

There are also other popular techniques for handling image processing tasks. The wavelets technique is widely used for image compression, although it can also be used for denoising.

Some of these filters can also be used as augmentation tools. For example, in one of our recent projects, we developed an AI algorithm that uses edge detection to discover the physical sizes of objects in digital image data.

To make it easier to use these techniques as well as to implement AI-based image processing functionalities in your product, you can use specific libraries and frameworks. In the next section, we take a look at some of the most popular ones.

Open-source libraries for AI-based image processing

Computer vision libraries contain common image processing functions and algorithms. There are several open-source libraries you can use when developing image processing and computer vision features:

OpenCV

Visualization Library

VGG Image Annotator

Pillow/PIL

scikit-learn

OpenCV

The Open Source Computer Vision Library (OpenCV) is a popular computer vision library that provides hundreds of computer and machine learning algorithms and thousands of functions composing and supporting those algorithms. The library comes with C++, Java, MATLAB, Octave, and Python interfaces and supports all popular desktop and mobile operating systems.OpenCV includes various modules for tasks like machine learning, image processing, and object detection. Using this library, you can acquire, compress, enhance, restore, and extract data from images.

Visualization Library

Visualization Library is C++ middleware for 2D and 3D applications based on the Open Graphics Library (OpenGL). This toolkit allows you to build portable and high-performance applications for Windows, Linux, and Mac OS X systems.

VGG Image Annotator

VGG Image Annotator (VIA) is a web application for manual object annotation. It can be installed directly in a web browser and used for annotating detected objects in images, audio, and video records. VIA is easy to work with, doesn’t require additional setup or installation, and can be used with any modern browser.

Pillow/PIL

Pillow is a fork of Python Imaging Library (PIL) — an open-source library for performing basic image processing tasks. Using this library, you can process, rescale, and save images in different formats.

scikit-learn

Scikit-learn is an open-source Python library built on NumPy, SciPy, and matplotlib. You can use this machine learning library to preprocess images, classify them, extract features, and reduce dimensionality. 

Machine learning frameworks and image processing platforms

If you want to move beyond using simple AI algorithms, you can build custom image processing AI models. To make development a bit faster and easier, you can use special platforms and frameworks. Below, we take a look at some of the most popular ones:

TensorFlow 

Google’s TensorFlow is a popular open-source framework with support for machine learning and deep learning. Using TensorFlow, you can create and train custom deep learning models. The framework also includes a set of libraries suitable for image processing projects and computer vision applications.

PyTorch

PyTorch is an open-source deep learning framework initially created by the Facebook AI Research (FAIR) lab. This Torch-based framework has Python, C++, and Java interfaces.

Among other things, you can use PyTorch for building computer vision and natural language processing applications.

Keras Core

Keras Core (also referred to as Keras 3.0) is a high-level API for creating and training deep learning models with a user-friendly interface. 

Being rebased on top of a modular backend architecture, Keras Core makes traditional Keras workflows available on top of arbitrary deep learning frameworks like PyTorch and TensorFlow. With its help, you can design, train, and deploy all kinds of deep learning models.

MATLAB Image Processing Toolbox

MATLAB is an abbreviation for matrix laboratory. It’s the name of both a popular platform for solving scientific and mathematical problems and a programming language. MATLAB provides an Image Processing Toolbox (IPT) including multiple algorithms and workflow applications for AI-based picture analysis, processing, and visualization as well as for algorithm development.

MATLAB IPT allows you to automate common image processing workflows. This toolbox can be used for noise reduction, image enhancement, image segmentation, 3D image processing, and other tasks. Many of the IPT functions support C/C++ code generation, so they can be used for deploying embedded vision systems and desktop prototyping.

Microsoft Computer Vision

Computer Vision is a cloud-based service provided by Microsoft that gives you access to advanced algorithms for image processing and data extraction. 

Microsoft Computer Vision allows you to analyze visual features and characteristics of an image, moderate image content, and extract text from images.

Google Cloud Vision

Cloud Vision is part of the Google Cloud platform that offers a set of image processing features. It provides an API for integrating such features as image labeling and classification, object localization, and object recognition.

Cloud Vision allows you to use pre-trained machine learning models or create and train custom AI for image processing.

Google Colaboratory (Colab)

Google Colaboratory, otherwise known as Colab, is a free cloud service that can be used for developing deep learning applications from scratch. Colab makes it easier to use popular libraries and frameworks such as OpenCV, Keras Core, and TensorFlow when developing an application for image processing using AI. 

Google Colab is based on Jupyter Notebooks, allowing AI developers to share their knowledge and expertise in a comfortable way. Plus, in contrast to similar services, Colab provides free GPU resources.

In addition to different libraries, frameworks, and platforms, you may also need a large database of images to train and test your model.

There are several open databases containing millions of tagged images that you can use for training your custom machine learning applications and algorithms. ImageNet, Pascal VOC, MNIST, and MS COCO are among the most popular free databases for image processing.

Using neural networks for image processing

Many of the tools we talked about in the previous section use AI for image analysis and solving complex image processing tasks. In fact, improvements in AI and machine learning are one of the reasons for the impressive progress in computer vision technology that we can see today.

Common image processing AI examples range from simple binary classification (whether an image does or doesn’t match a specific criteria) to instance segmentation. Choosing the right type and architecture of a neural network plays an essential part in creating an efficient artificial intelligence image processing solution.

Below, we take a look at several popular types of neural networks and specify the tasks they’re most fit for.

Convolutional Neural Network

Convolutional neural networks (ConvNets or CNNs) are a class of deep learning networks that were created specifically for image processing with AI. However, CNNs have been successfully applied to various types of data, not only images. 

In these networks, neurons are organized and connected similarly to how neurons are organized and connected in the human brain. In contrast to other neural networks, CNNs require fewer preprocessing operations. Plus, instead of using hand-engineered filters (despite being able to benefit from them), CNNs can learn the necessary filters and characteristics during training.

CNNs are multilayered neural networks that include input and output layers as well as a number of hidden layer blocks which consist of:

• Convolutional layers — Filter the input image and extract specific features such as edges, curves, and colors
• Pooling layers — Improve the detection of unusually placed objects
• Normalization (ReLU) layers — Improve network performance by normalizing the inputs of the previous layer

Fully connected layers — Connect the neurons between two different layers of a CNN in order to analyze and learn the function of features extracted by the network’s convolutional layer

All CNN layers are organized in three dimensions (weight, height, and depth) and have two components:

• Feature extraction — The CNN runs multiple convolutions and pooling operations in order to detect features it will then use for image classification.
• Classification — Using the extracted features, the network algorithm attempts to predict what the object in the image could be with a calculated probability.

CNNs are widely used for implementing AI in image processing and solving such problems as signal processing, image classification, and image recognition. There are numerous types of CNN architectures, including AlexNet, ZFNet, Faster R-CNN, GoogLeNet/Inception, and YOLOv3.

Mask R-CNN

Mask R-CNN is a Faster R-CNN-based deep neural network that can be used for separating objects in a processed image or video. This neural network works in two stages:

• Segmentation – The neural network processes an image, detects areas that may contain objects, and generates proposals.
• Generation of bounding boxes and masks – The network calculates a binary mask for each class and generates the final results based on these calculations.

This neural network model is flexible, adjustable, and provides better performance when compared to similar solutions. However, Mask R-CNN struggles with real-time processing, as this neural network is quite heavy and the mask layers add a bit of performance overhead, especially compared to Faster R-CNN.

Mask R-CNN remains one of the best solutions for instance segmentation. At Apriorit, we have applied this neural network architecture and our image processing skills to solve many complex tasks, including the processing of medical image data and medical microscopic data. We’ve also developed a plugin for improving the performance of this neural network model up to ten times thanks to the use of NVIDIA TensorRT technology.

Fully convolutional networks

The concept of a fully convolutional network (FCN) was first offered by a team of researchers from the University of Berkeley. The main difference between a CNN and FCN is that the latter has a convolutional layer instead of a regular fully connected layer. As a result, FCNs are able to manage different input sizes. Also, FCNs use downsampling (striped convolution) and upsampling (transposed convolution) to make convolution operations less computationally expensive.

A fully convolutional neural network is the perfect fit for image segmentation tasks when the neural network divides the processed image into multiple pixel groupings which are then labeled and classified. Some of the most popular FCNs used for semantic segmentation are DeepLab, FCN-8, and U-Net.

U-Net

U-Net is a fully convolutional neural network that allows for fast and precise image segmentation. In contrast to other neural networks on our list, U-Net was designed specifically for biomedical image segmentation. Therefore, it comes as no surprise that U-Net is believed to be superior to Mask R-CNN, especially in such complex tasks as medical image processing.

U-Net has a U-shaped architecture and has more feature channels in its upsampling part. As a result, the network propagates context information to higher-resolution layers, thus creating a more or less symmetric expansive path to its contracting part.

At Apriorit, we successfully implemented a system with the U-Net backbone to complement the results of a medical image segmentation solution. This approach allowed us to get more diverse image processing results and permitted us to analyze the received results with two independent systems. Additional analysis is especially useful when a domain specialist feels unsure about a particular image segmentation result.

Generative networks

Generative networks are double networks that include two nets — a generator and a discriminator — that are pitted against each other. The generator is responsible for generating new data and the discriminator is supposed to evaluate that data for authenticity.

In contrast to other neural networks, you can use generative neural networks to create new synthetic images from other images or noise, as well as solve image inpainting (reconstructing missing regions in an original image) and image super-resolution (enhancing the resolution of low-quality images) tasks. Common examples of generative models include generative adversarial networks and Variational AutoEncoders.

Generative adversarial networks

Generative adversarial networks (GANs) are supposed to deal with one of the biggest challenges neural networks face these days: adversarial images.

Adversarial images are known for causing massive failures in neural networks. For instance, a neural network can be fooled if you add a layer of visual noise called perturbation to the original image. And even though the difference is nearly unnoticeable to a human, computer algorithms struggle to properly classify adversarial images (see Figure 6).

Common examples of GANs include pix2pix, EdgeConnect, and ESRGAN.

Transformer networks

Transformer neural networks are deep learning models that transform existing data into new instances. These models can be used for computer vision and natural language processing tasks. 

Transformers applied for computer vision are also called vision transformers (ViTs). They can be used for tasks like image recognition and image restoration. Some ViTs can also be used to generate images by transforming other images, textual inputs, and voice inputs into new synthetic images.

Conclusion

With the help of deep learning algorithms and neural networks, we can train computers to process and interpret images similarly to the way the human brain does. AI can detect objects and people in images and videos, recognize people’s faces, restore lost and damaged data, and even create synthetic images from other images as well as from text and voice input.

Apriorit specialists from the artificial intelligence team always keep track of the latest improvements in AI-powered image processing. We can help you build AI and deep learning solutions based on the latest field research and using leading frameworks such as Keras Core, TensorFlow, and PyTorch. We know which technologies to apply for your project and will gladly help you deliver the best results possible.