A Few Files to Make Things Easier…

Visual Studio Code is clever, but to understand the code related to the OpenCV library, it needs some instructions. Those instructions are included in a project’s metadata file, which specifies what library and what version to include to run the code.

You can open that top folder from the project template from within Visual Studio Code, which gives you a familiar view, as shown in Figure 2-1.

Figure 2-1 shows the two Java files, so you can see the content of each; in addition, the project layout has been expanded so all the files are listed on the left.

This view should look familiar to you from the previous chapter, and you can immediately try running the HelloCv.java code using the Run or Debug link at the top of the main function.

Let’s go through the code line by line to understand what is happening behind the scenes.

This step is required because OpenCV is not a Java library; it is a binary compiled especially for your environment.

In this book, we will rely on some packaging magic, where the library is downloaded and loaded automatically for you and where, along the way, the library is placed in a location that NativeLoader handles for you. So, there’s nothing to do here except to add that one-liner to the start of each of your OpenCV programs; it’s best to put it at the top of the main function.

That second line creates a Mat object. A Mat object is, as explained a few seconds ago, a matrix. All image manipulation, all video handling, and all the networking are done using that Mat object. It wouldn’t be too much of a stretch to say that the main thing OpenCV is programmatically doing is proposing a great programming interface to work with optimized matrices, in other words, this Mat object.

In Visual Studio Code, you can already access operations directly using the autocompletion feature, as shown in Figure 2-2.

Within the one-liner, you create a 3×3 matrix, and the internal type of each element in the matrix is of type CV_8UC1. You can think of 8U as 8 bits unsigned, and you can think of C1 as channel 1, which basically means one integer per cell of the matrix.

The number of channels in a Mat is also important because each pixel in an image can be described by a combination of multiple values. For example, in RGB (the most common channel combination in images), there are three integer values between 0 and 256 per pixel. One value is for red, one is for green, and one is for blue.

Executing the previous code will give you this 50×50 Mat, where each pixel is made of three channels, meaning three values, but note that in OpenCV, RGB values are reversed by design (why, oh, why?), so it is actually BGR. The value for blue comes first in the code.

So, before the value of all the pixels, set using the function setTo, is B:190, G:119, Red:0, which gives the nice ocean sea blue shown in Figure 2-3.

Operations on Mat objects are usually done using the org.opencv.core.Core class. For example, adding two Mat objects is done using the Core.add function, two input objects, two Mat objects, and a receiver for the result of the addition, which is another Mat.

Now, we’ll let you practice a little bit and perform the same operation on a 50×50 Mat and show the result in a window again with HighGui.

The result should look like Figure 2-4.

Simple Addition

Don’t get scared; let’s quickly check with the debugger what is happening and add a breakpoint at the proper location; see the Mat objects on the Variables tab, as shown in Figure 2-7.

Before adding a breakpoint, we can see that the Marcel Mat is indeed 2304×1728, while the beach Mat is smaller at 333×500, so we definitely need to resize the second Mat to match that of Marcel. If we do not perform this resizing step, OpenCV does not know how to compute the result of the add function and gives the error message shown earlier.

Running this code, the output image looks similar to Figure 2-8.

Hmm. It’s certainly better because the program runs to the end without error, but something is not quite right. The output picture gives an impression of being over-exposed, and looks way too bright. And indeed, if you look, most of the output image pixels have maximum RGB values of 255,255,255, which is the RGB value for white.

We should do an addition in a way that keeps the feeling of each Mat but still not goes over the maximum 255,255,255 value.

Weighted Addition

Preserving meaningful values in Mat objects is certainly something that OpenCV can do. The Core class comes with a weighted version of the add function that is conveniently named addWeighted. What addWeighted does is multiply values of each Mat object, each by a different scaling factor. It’s even possible to adjust the result value with a parameter called gamma.

The output of the program execution gives something much more useful, as shown in Figure 2-9.

Finding Cat Faces in Pictures Using a Classifier

Before processing speeds got faster and neural networks were making the front pages of all the IT magazines and books, OpenCV implemented a way to use classifiers to detect objects within pictures.

Classifiers are trained with only a few pictures, using a method that feeds the classifier with training pictures and the features you want the classifier to detect during the detection phase.

The OpenCV documentation on cascade classifiers (https://docs.opencv.org/4.1.1/db/d28/tutorial_cascade_classifier.html) is full of extended details when you want to get some more background information. For now, the goal here is not to repeat this freely available documentation.

What Is a Feature?

Features are key points extracted from a set of digital pictures used for training, something that can be reused for matching on totally new digital inputs.

For example, an ORB classifier, nicely explained in “Object recognition with ORB and its Implementation on FPGA” (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.405.9932&rep=rep1&type=pdf) is a very fast binary descriptor based on BRIEF. The other famous feature-based algorithms are Scale Invariant Feature Transform (SIFT) and Speed Up Robust Features (SURF); all these are looking to match features for a set of known images (what we are looking for) on new inputs.

Listing 2-8 shows a super-brief example of how feature extraction is done. Here we are using an ORB detector to find the key points of a picture.

Mat src = Imgcodecs.imread("marcel2.jpg", Imgcodecs.IMREAD_GRAYSCALE);

ORB detector = ORB.create();

MatOfKeyPoint keypoints = new MatOfKeyPoint();

detector.detect(src, keypoints);

Mat target = src.clone();

target.setTo(new Scalar(255, 255, 255));

Features2d.drawKeypoints(target, keypoints, target);

Imgcodecs.imwrite("orb.png", target);

Listing 2-8

ORB Feature Extraction

Basically, the features are extracted into a set of points. We don’t do that usually, but drawing the key points here gives something like Figure 2-13.

Cascade classifiers are called that because they internally have a set of different classifiers, each of them getting a deeper, more detailed chance of a match at a cost of speed. So, the first classifier will be very fast and get a positive or a negative, and if positive, it will pass the task on to the next classifier for some more advanced processing, and so on.

Haar-based classifiers, as proposed by Paul Viola and Michael Jones, are based on analyzing four main features, as shown in Figure 2-14.

Since the features are easily computed, the number of images required for training Haar-based object detection is quite low. Most importantly, up until recently, with low CPU speeds on embedded systems, those classifiers had the advantage of being very fast.

Where in the World Is Marcel?

So, enough talking and reading about research papers. In short, a Haar-based cascade classifier is good at finding features of faces, of humans or animals, and can even focus on eyes, noses, and smiles, as well as on features of a full body.

The classifiers can also be used to count the number of moving heads in video streams and find out whether anyone is stealing things from the fridge after bedtime.

OpenCV makes it a no-brainer to use a cascade classifier and get some instant gratification. The way to put cascade classifiers into action is as follows:

1. 1.

   Load the classifier from an XML definition file containing values describing the features to look for.

    
2. 2.

   Access a Mat object called detectMultiScale directly on the input Mat object, and a Mat object called MatOfRect, which is a specific OpenCV object designed to handle lists of rectangles nicely.

    
3. 3.

   Once the previous call is finished, MatOfRect is filled with a number of rectangles, each of them describing a zone of the input image, where a positive has been found.

    
4. 4.

   Do some artsy drawing on the original picture to highlight what was found by the classifier.

    
5. 5.

   Save the output.

    

The Java code for this is actually rather simple and just barely longer than the equivalent Python code. See Listing 2-9.

String classifier

    = "haarcascade_frontalcatface.xml";

CascadeClassifier cascadeFaceClassifier

    = new CascadeClassifier(classifier);

Mat cat = Imgcodecs.imread("marcel.jpg");

MatOfRect faces = new MatOfRect();

cascadeFaceClassifier.detectMultiScale(cat, faces);

for (Rect rect : faces.toArray()) {

     Imgproc.putText(

       cat,

       "Chat",

      new Point(rect.x, rect.y - 5),

      Imgproc.FONT_HERSHEY_PLAIN, 10,

      new Scalar(255, 0, 0), 5);

     Imgproc.rectangle(

       cat,

       new Point(rect.x, rect.y),

       new Point(rect.x + rect.width, rect.y + rect.height),

      new Scalar(255, 0, 0),

      5);

}

Imgcodecs.imwrite("marcel_blunt_haar.jpg", cat);

Listing 2-9

Calling a Cascade Classifier on an Image

After running the code in Listing 2-9, you will realize quickly what the problem is with the slightly naïve approach. The classifier is finding a lot of extra positives, as shown in Figure 2-15.

There are two techniques to reduce the number of detected objects.

• Filter the rectangles based on their sizes in the loop on rectangles. While this is often used thanks to its simplicity, this adds a chance to focus on false positives.

• Pass in extra parameters to detectMultiScale, specifying, among other things, a certain number of required neighbors to get a positive or indeed a minimum size for the returned rectangles.

The full version has all the parameters shown in Table 2-3.

Table 2-3

detectMultiScale Parameters

Parameter	Description
image	Matrix of the type CV_8U containing an image where objects are detected.
objects	Vector of rectangles where each rectangle contains the detected object; the rectangles may be partially outside the original image.
scaleFactor	Parameter specifying how much the image size is reduced at each image scale.
minNeighbors	Parameter specifying how many neighbors each potential candidate rectangle should have to be retained.
flags	Not used for a new cascade.
minSize	Minimum possible object size. Objects smaller than this value are ignored.
maxSize	Maximum possible object size. Objects larger than this value are ignored if the maxSize == minSize model is evaluated on a single scale.

Based on the information from Table 2-3, let’s apply some sensible parameters to detectMutiScale, as shown here:

• scalefactor=2

• minNeighbors=3

• flags=-1 (ignored)

• size=300x300

Building on the code in Listing 2-9, let’s replace the line containing detectMultiScale with this updated one:

cascadeFaceClassifier.detectMultiScale(cat, faces, 2, 3, -1, new Size(300, 300));

Applying the new parameters, and (why not?) running a debug session on the new code, gives you the layout shown in Figure 2-16.

The output is already included in the layout, but you will notice the found rectangles are now limited to only one rectangle, and the output gives only one picture (and, yes, there can be only one Marcel).

Finding Cat Faces in Pictures Using the Yolo Neural Network

We have not really seen how to train cascade classifiers to recognize things we want them to recognize (because it’s beyond the scope of this book). The thing is, most classifiers have a tendency to recognize some things better than others, for example, people more than cars, turtles, or signs.

Detection systems based on those classifiers apply the model to an image at multiple locations and scales. High-scoring regions of the image are considered detections.

The neural network Yolo uses a totally different approach. It applies a neural network to the full image. This network divides the image into regions and predicts bounding boxes and probabilities for each region. These bounding boxes are weighted by the predicted probabilities.

Yolo has proven fast in real-time object detection and is going to be our neural network of choice to run in real time on the Raspberry Pi in Chapter 3.

Later, you will see how to train a custom Yolo-based model to recognize new objects that you are looking for, but to bring this chapter to a nice and exciting end, let’s quickly run one of the provided default Yolo networks, trained on the COCO image set, that can detect a large set of 80 objects, cats, bicycle, cars, etc., among other objects.

As for us, let’s see whether Marcel is up to the task of being detected as a cat even through the eyes of a modern neural network.

The final sample introduces a few more OpenCV concepts around deep neural networks and is a great closure for this chapter.

You probably know already what a neural network is; it is modeled on how the connections in the brain work, simulating threshold-based neurons triggered by incoming electrical signals. If my highly neuron-dense brain were a drawing, it would look something like Figure 2-17.

I actually do hope reality is slightly different and my brain is way more colorful.

What you see in Figure 2-17 at first sight is the configuration of the network. Figure 2-17 shows only one hidden layer in between the input and output layers, but standard deep neural networks have around 60 to 100 hidden layers and of course way more dots for inputs and outputs.

In many cases, the network maps the image, with a hard-coded size specific to that network, from one pixel to one dot. The output is actually slightly more complicated, containing among other things probabilities and names, or at least an index of names in a list.

During the training phase, each of the arrows in the diagram, or each neuron connection in the network, is slowly getting a weight, which is something that impacts the value of the next neuron, which is a circle, in the next hidden or output layer.

When running the code sample, we will need both a config file for the graphic representation of the network, with the number of hidden layers and output layers, and a file for the weights, which includes a number for each connection between the circles.

Let’s move from theory to practice now with some Java coding. We want to give some file as the input to this Yolo-based network and recognize, you guessed it, our favorite cat.

Running the example on images of Marcel actually brings really high confidence scores that are close to a perfect location, as shown in Figure 2-18.

Running the inference on many cats also gives great results, but on the second run we are missing one of the cats because of its close position to another detected cat and because of our introduction of the postprocessing step, which removes overlapping matching boxes. See Figure 2-19.

An exercise for you at this point is to change the parameters of the Dnn.NMSBoxes function call to see whether you can get the two boxes to show at the same time.

The problem with static images is that it is difficult to get the extra context that we have in real life. This is a shortcoming that goes away when dealing with sets of input images coming from a live video stream.

So, with Chapter 2 wrapped up, you can now do object detection on pictures. Chapter 3 will take it from here and use the Raspberry Pi to teach you about on-device detection.