Installing OpenCV

As with the Raspberry Pi OS, there are two methods to install OpenCV: the easy way, using Python’s pip install method, and the hard way, compiling the packages from source code. Unlike Raspberry Pi OS, the hard way for OpenCV is significantly more complicated and, frankly, difficult. I will present both, but strongly suggest using the easy way.

Displaying an Image

It’s actually very easy to display an image in OpenCV. The imshow() function provides this functionality. This function is used with both still and video images, and the implementation does not change between them. The imshow() function opens a new window to display the image. When you call it, you have to provide a name for the window as well as the image or frame that you want to display.

This is an important point about how OpenCV works with video. Because OpenCV treats a video as a series of individual frames, virtually all the functions used to modify or analyze an image apply to a video. This obviously includes imshow().

As you can see, the code is the same. Again, this is because OpenCV treats video as a series of individual frames. In fact, video capture depends on a loop to continuously capture the next frame. So in essence, displaying a still image from a file and an individual frame from the camera is exactly the same thing.

We use imshow() and waitKey() extensively throughout this chapter.

Opening an Image File

OpenCV makes working with images and files remarkably easy, especially considering what is happening in the background to make these operations possible. Opening an image file is no different. We use the imread() function to open image files from local storage. The imread() function takes two parameters: file name and color type flag. The file name is obviously required to open the file. The color type flag determines whether to open the image in color or grayscale.

A window opens to show an image of three colored balls on a white background (see Figure 9-3). Press any key to close it.

Due to the way Thonny interacts with the GUI system on Linux-based machines, the image window will not close properly if you were to run the code directly from Thonny. However, by running the code from the terminal, we do not have this issue.

Capturing Video

Capturing video with your camera is a little different from opening a file. There are a few more steps to use video. One change is that we have to use a loop to get multiple frames; otherwise, the OpenCV will only capture a single frame, which is not what we want. An open while loop is generally used. This captures the video until we actively stop it.

To make things easier for testing, I placed the ball directly in front of the camera (see Figure 9-4). Right now, we just want to capture the image.

To capture the video from the camera, we will create a videoCapture() object and then use the read() method in a loop to capture the frames. The read() method returns two objects: a return value and an image frame. The return value is simply an integer verifying the success or failure of the read. If the read is successful, the value is 1; otherwise, the read failed and it returns 0. To prevent errors that cause your code to error, you can test to see if the read is successful.

We care about the image frame. If the read is successful, an image is returned. If it was not, then a null object is returned in its place. Since a null object cannot access OpenCV methods, the instant you try to modify or manipulate the image, your code will crash. This is why it’s a good idea to test for the success of the read operation.

Flipping

Many times, the placement of the camera in a project is not ideal. Frequently, I’ve had to mount the camera upside down, or I’ve needed to flip the image for one reason or another.

Fortunately, OpenCV makes this very simple with the flip() method. The flip() method takes three parameters: the image to be flipped, the code indicating how to flip it, and the destination of the flipped image. The last parameter is only used if you want to assign the flipped image to another variable, but you can flip the image in place.

An image can be flipped horizontally, vertically, or both by providing the flipCode. The flipCode is positive, negative, or zero. Zero flips the image horizontally, a positive value flips it vertically, and a negative number flips it on both axes. More often than not, you will flip the image on both axes to effectively rotate it 180 degrees.

Four windows open, each with a different version of the image file. Press any key to exit.

Resizing

You can resize an image. This is useful for reducing the resources needed to process an image. The larger the image, the more memory and CPU resources needed. To resize an image, we use the resize() method . The parameters are the image you are scaling, the desired dimensions as a tuple, and the interpolation.

Interpolation is the mathematical method used for determining how to handle the removal or addition of pixels. Remember, when working with images, you are really working with a multidimensional array that contains information for each point, or pixel, that makes up the image. When you reduce an image, you are removing pixels. When you enlarge an image, you are adding pixels. Interpolation is the method by which this occurs.

There are three interpolation options. INTER_AREA is best used for reduction. INTER_CUBIC and INTER_LINEAR are both good for enlarging an image, with INTER_LINEAR being the faster of the two. If an interpolation is not provided, OpenCV uses INTER_LINEAR as the default for both reducing and enlarging.

Two windows should have opened. The first has the original image. The second displays the reduced image. Press any key to close the windows.

Color Spaces

One of the key elements of working with color is color space, which describes how OpenCV expresses color. Within OpenCV, color is represented by a series of numbers. The color space determines the meaning of those numbers.

The default color space for OpenCV is BGR. This means every color is described by three integers between 0 and 255, which correspond to the three color channels—blue, green, and red, in that order. A color expressed as (255,0,0) has a maximum value in the blue channel, and both green and red are zero. This represents pure blue. Given this, (0,255,0) is green and (0,0,255) is red. The values (0,0,0) represent black, the absence of any color, and (255,255,255) is white.

If you’ve worked with graphics in the past, BGR is the opposite of what you’re likely used to. Most digital graphics are described in terms of RGB—red, green, and blue. So this may take a little getting used to.

There are many color spaces. The ones that we care about are BGR, RGB, HSV, and grayscale. We’ve already discussed the default color space, BGR, and the common RGB color space. HSV is hue, saturation, and value. Hue represents the color on a scale of 0–180. Saturation represents how far white the color is from 0 to 255. Value is a measure of how far from black the color is from 0 to 255. If both saturation and value are 0, the color is gray. A saturation and value of 255 is the brightest version of the hue.

Hue is a little trickier. It is on a scale of 0–180, where 0 and 180 are both red. This is where remembering the color wheel is important. If 0 and 180 meet at the top of the wheel in the middle of the red space, as you move clockwise around the wheel, hue = 30 is yellow, hue = 60 is green, hue = 90 is teal, hue = 120 is blue, hue = 150 is purple, and hue = 180 brings us back to red.

The one that you most frequently encounter is grayscale. Grayscale is exactly what it sounds like: the black-and-white version of an image. It is used by feature detection algorithms to create masks. We use it when we filter for objects.

To convert an image to a different color space, you use the cvtColor method. It takes two parameters: the image and the color space constant. The color space constants are built into OpenCV. They are COLOR_BGR2RGB, COLOR_BGR2HSV, and COLOR_BGR2GRAY. Do you see the pattern there? If you wanted to convert from the RGB color space to the HSV color space, the constant would be COLOR_RGB2HSV.

This opens two windows: one with the original color image and one with the grayscale version. Click any key to exit the program and close the windows .

Color Filters

Filtering for a color takes remarkably little code, but at the same time, it can be a little frustrating because you’re generally not looking for a specific color but a color range. Colors are very rarely pure and of one value. This is why we want to be able to shift between color spaces. Sure, we could look for a red color in BGR. But to do that, we would need the specific range for each of the three values. And where that’s going to hold true with all color spaces, it’s generally easier to dial in the range you need in the HSV space.

The strategy used for filtering for a specific color is fairly straightforward, but there are a few steps involved and a couple things to keep in mind as you go.

First, we’ll make a copy of the image in the HSV color space. Then we apply our filter range and make that its own image. For this, we use the inRange() method. It takes three parameters: the image we are applying the filter to, the lower range, and the upper range of values. The inRange method scans all the pixels in the provided image to determine if they are within the specified range. It returns true, or 1, if so; otherwise, it returns 0. What this leaves us with is a black-and-white image that we can use as a mask.

Next, we apply the mask using the bitwise_and() method. This method takes two images and returns the area where the pixels match. Since that’s not quite what we are looking for, we need to do a little trickery. For our purpose, bitwise_and requires three parameters: image 1, image 2, and a mask. Since we want to return everything that is revealed by the mask, image 1 and image 2 both use our original image. Then we apply our mask by designating the mask parameter. Since we are leaving out a few optional parameters, we need to designate the mask parameter explicitly, like this: mask = mask_image. The result is an image that only shows the color we are filtering for .

Three windows open with different versions of our image. The first is the regular image. The second is a black-and-white image that acts as our mask. The third is the final masked image. Only the pixels under the white area of the mask are displayed.

Let’s take a moment to walk through the code to make it clear what we’re doing and why.

In the end, you’re left with an image with only the pixels that you are looking for. To the human eye, it is easily recognized as related groups. For the computer , it is not so. Natively, a computer does not recognize the difference between the black pixels and the blue. That’s where blob detection comes into play.

Finding a Blob

OpenCV makes blob detection fairly easy, especially after we’ve done the heavy lifting of filtering out everything we don’t want. Once the image has been filtered, we can use the mask for clean blob detection. The SimpleBlobDetector class from OpenCV identifies the location and the size of the blob.

The SimpleBlobDetector class is not quite as simple as you might think. There are a number of parameters built into it that need to be enabled or disabled. If enabled, you need to make sure that the values work for your application.

The method for setting the parameters is SimpleBlobDetector_Params(). The method for creating the detector is SimpleBlobDetector_create(). You pass the parameters to the create method to ensure everything is set properly.

Once the parameters are set and the detector is properly created, you use the detect() method to identify the keypoints. In the case of the simple blob detector, the keypoints represent the center and size of any detected blobs.

Finally, we use the drawKeyPoints() method to draw a circle around our blob. By default, this draws a small circle at the center of the blob. However, a flag can be passed that causes the size of the circle relative to the size of the blob.

This opens the three versions of the image. However, the original image now has a red circle drawn around the blue ball. In the terminal window, we’ve printed the coordinates of the center of the ball as well as its size. The center of the ball is used later in this chapter when we start to track the ball.

The Filters

The following lists the filters and their corresponding parameters.

filterByColor

This filters for the relative intensity of each binary image. It measures the intensity value at the center of the blobs and compares it to the parameter, blobColor. If they don’t match, the blob does not qualify. The intensity is measured from 0 to 255; 0 is dark and 255 is light.

filterByArea

When the individual blobs are grouped, their overall area is calculated. This filter looks for blobs between minArea and maxArea.

filterByCircularity

Circularity is calculated by the formula

This returns a ratio between 0 and 1, which is compared to minCircularity and maxCircularity. If the value is between these parameters, the blob is included in the results.

filterByInertia

Inertia is an estimation of how elongated the blob is. It is a ratio between 0 and 1. If the value is between minInertiaRatio and maxInertiaRatio, the blob is returned in the keypoint results.

filterByConvexity

Convexity is a ratio with a value between 0 and 1. It measures the ratio between convex and concave curves in a blob. The parameters for convexity are minConvexity and maxConvexity.

Blob Tracking

Two windows open: one showing the mask used for filtering the color and one with the video stream. A circle should be drawn around the blob.

I enabled filterByArea and filterByCircularity to make sure that I am only getting the ball. You will likely need to make adjustments to the detector’s parameters to fine-tune your filter.