Much of the processing you apply to images and videos in OpenCV involves the concept of Fourier Transform in some capacity. Joseph Fourier was an 18th century French mathematician who discovered and popularized many mathematical concepts, and concentrated his work on studying the laws governing heat, and in mathematics, all things waveform. In particular, he observed that all waveforms are just the sum of simple sinusoids of different frequencies.
In other words, the waveforms you observe all around you are the sum of other waveforms. This concept is incredibly useful when manipulating images, because it allows us to identify regions in images where a signal (such as image pixels) changes a lot, and regions where the change is less dramatic. We can then arbitrarily mark these regions as noise or regions of interests, background or foreground, and so on. These are the frequencies that make up the original image, and we have the power to separate them to make sense of the image and extrapolate interesting data.
Note
In an OpenCV context, there are a number of algorithms implemented that enable us to process images and make sense of the data contained in them, and these are also reimplemented in NumPy to make our life even easier. NumPy has a
Fast Fourier Transform (FFT) package, which contains the fft2() method. This method allows us to compute
Discrete Fourier Transform (DFT) of the image.
Let's examine the magnitude spectrum concept of an image using Fourier Transform. The magnitude spectrum of an image is another image, which gives a representation of the original image in terms of its changes: think of it as taking an image and dragging all the brightest pixels to the center. Then, you gradually work your way out to the border where all the darkest pixels have been pushed. Immediately, you will be able to see how many light and dark pixels are contained in your image and the percentage of their distribution.
The concept of Fourier Transform is the basis of many algorithms used for common image processing operations, such as edge detection or line and shape detection.
Before examining these in detail, let's take a look at two concepts that—in conjunction with the Fourier Transform—form the foundation of the aforementioned processing operations: high pass filters and low pass filters.
A high pass filter (HPF) is a filter that examines a region of an image and boosts the intensity of certain pixels based on the difference in the intensity with the surrounding pixels.
Take, for example, the following kernel:
[[0, -0.25, 0], [-0.25, 1, -0.25], [0, -0.25, 0]]
Note
A kernel is a set of weights that are applied to a region in a source image to generate a single pixel in the destination image. For example, a ksize of 7 implies that 49 (7 x 7) source pixels are considered in generating each destination pixel. We can think of a kernel as a piece of frosted glass moving over the source image and letting through a diffused blend of the source's light.
After calculating the sum of differences of the intensities of the central pixel compared to all the immediate neighbors, the intensity of the central pixel will be boosted (or not) if a high level of changes are found. In other words, if a pixel stands out from the surrounding pixels, it will get boosted.
This is particularly effective in edge detection, where a common form of HPF called high boost filter is used.
Both high pass and low pass filters use a property called radius, which extends the area of the neighbors involved in the filter calculation.
Let's go through an example of an HPF:
import cv2
import numpy as np
from scipy import ndimage
kernel_3x3 = np.array([[-1, -1, -1],
[-1, 8, -1],
[-1, -1, -1]])
kernel_5x5 = np.array([[-1, -1, -1, -1, -1],
[-1, 1, 2, 1, -1],
[-1, 2, 4, 2, -1],
[-1, 1, 2, 1, -1],
[-1, -1, -1, -1, -1]])img = cv2.imread("../images/color1_small.jpg", 0)
k3 = ndimage.convolve(img, kernel_3x3)
k5 = ndimage.convolve(img, kernel_5x5)
blurred = cv2.GaussianBlur(img, (11,11), 0)
g_hpf = img - blurred
cv2.imshow("3x3", k3)
cv2.imshow("5x5", k5)
cv2.imshow("g_hpf", g_hpf)
cv2.waitKey()
cv2.destroyAllWindows()After the initial imports, we define a 3x3 kernel and a 5x5 kernel, and then we load the image in grayscale. Normally, the majority of image processing is done with NumPy; however, in this particular case, we want to "convolve" an image with a given kernel and NumPy happens to only accept one-dimensional arrays.
This does not mean that the convolution of deep arrays can't be achieved with NumPy, just that it would be a bit complex. Instead, ndimage (which is a part of SciPy, so you should have it installed as per the instructions in Chapter 1, Setting Up OpenCV), makes this trivial, through its convolve() function, which supports the classic NumPy arrays that the cv2 modules use to store images.
We apply two HPFs with the two convolution kernels we defined. Lastly, we also implement a differential method of obtaining a HPF by applying a low pass filter and calculating the difference with the original image. You will notice that the third method actually yields the best result, so let's also elaborate on low pass filters.
If an HPF boosts the intensity of a pixel, given its difference with its neighbors, a low pass filter (LPF) will smoothen the pixel if the difference with the surrounding pixels is lower than a certain threshold. This is used in denoising and blurring. For example, one of the most popular blurring/smoothening filters, the Gaussian blur, is a low pass filter that attenuates the intensity of high frequency signals.