The most perfect technique is that which is not noticed at all.
Pablo Casals
The previous chapter was an introduction to Origami and how to perform mostly single-step processing operations on simple mats and images.
While that was already a very good show to highlight the library’s ease of use, the third chapter wants to take you one step further by combining simple processing steps together to reach a bigger goal. From performing content analysis, contour detection, shape finding, and shape movements, all the way to computer-based sketching and landscape art, you name it, many an adventure awaits here.
We will start again on familiar ground by manipulating OpenCV mats at the byte level, to grasp in even more detail the ins and outs of image manipulation.
The learning will be split into two big sections. First will be a slightly art-focused section, where we play with lines, gradations, and OpenCV functions to create new images from existing ones. You will be using already known origami/opencv functions, but a few other ones will also be introduced as needed to go with the creative flow.
It was one of the original plans of Origami to be used to create drawings. It just happened that to understand how simple concepts were brought together, I had to play with image compositions and wireframes that actually came out better than I thought they would. Even more so, it was easy to just add your own touch and reuse the creations later on. So that first part is meant to share this experience.
Then, in a second part, we will move onto techniques more focused on image processing. Processing steps will be easier to grasp at that stage, after reviewing steps with immediate feedback from the art section.
Processing steps in OpenCV are easy most of the time, but the original samples in C++ make it quite hard to read through the lines of pointers. I personally find, even with the Clojure learning curve included, that Origami is an easier way to get started with OpenCV: you can focus on the direct impact of your lines of code, and try writing each step in different ways without restarting everything by getting instant feedback each time, until eventually it comes into place nicely. Hopefully, the second part of the chapter will make you comfortable enough that you will want to go and challenge the examples even more.
Note that it is probably a good idea to read this chapter linearly so that you do not miss new functions or new tricks along the way. However, nothing prevents you from just jumping in where you feel like it, of course. It is a recipe book after all!
3.1 Playing with Colors
Problem
In the previous chapter, you already saw various techniques to change colors in a mat.
You would like to get control over how to specify and impact colors , for example, increasing or decreasing their intensity, by applying specific factors or functions on the mats.
Solution
Here, you will learn about the following: how to combine operations like converting an image color channel using the already known cvt-color; how to use other OpenCV functions like threshold to limit channel values; how to create masks and use them with the function set-to; and how to use functions to combine separate versions of a mat.
You will review also in more detail how to use the transform! function to create basic art effects.
How it works
To play with mats, we will be using another set of cats and flowers, but you can of course try applying the functions on your own photos any time.
The namespace header of the chapter, with all the namespace dependencies, will use the same namespaces required in the last chapter, namely, opencv3.core and opencv3.utils as well as opencv3.colors.rgb from origami’s opencv3 original namespaces.
The required section looks like the following code snippet.
It is usually a good idea to create a new notebook for each experiment, and to save them separately.
Applying Threshold on a Colored Mat
Back to the basics . Do you remember how to threshold on a mat, and keep only the values in the matrix above 150?
Yes, you’re correct: use the threshold function.
The input matrix contains various values, some below and some above the threshold value of 150. When applying threshold, the values below are set to 0 and the ones above are set to threshold’s second parameter value, 255.
This results in the following matrix (Figure 3-1):
That was for a one-channel mat , but what happens if we do the same on a three-channel mat?
Converting the colors to BGR duplicates each of the values of the one-channel mat to the same three values on the same pixel.
Applying the OpenCV threshold function right afterward applies the threshold to all the values over each channel. And so the resulting mat loses the 100 values of the original mat and keeps only the 255 values.
A 3×3 matrix is a bit too small to show onscreen, so let’s use resize on the input matrix first.
Applying a similar threshold on the preceding mat keeps the light gray, which has a value above the threshold, but removes the darker gray by turning it to black.
Notice the use of a specific interpolation parameter with resize, INTER_AREA, which nicely cuts the shape sharp, instead of interpolating and forcing a blur.
Anyway, back to the exercise, and you probably have it at this point: applying a standard threshold pushes forward vivid colors.
Let’s see how that works on a mat loaded from an image, and let’s load our first image of the chapter (Figure 3-4).
We start by applying the same threshold that was applied on the mat loaded from a matrix, but this time on the rose image .
In a nicely shot photograph, this actually gives you an artistic feeling that you can build upon for cards and Christmas presents!
Let’s now apply a similar technique on a completely different image. We’ll turn the picture to black and white first and see what the result is.
This time, the picture is of playful kittens, as shown in Figure 3-6.
If you apply a similar threshold but on the grayscale version , something rather interesting happens.
Cool; this means that the shape we wanted to stand out has been highlighted.
Something similar to this can be used to find out shapes and moving objects; more in recipe 3-6 and 3-7.
For now, and to keep things artistic, let’s work on a small function that will turn all the colors under a given threshold to one color, and all the values above the threshold to another one.
First, turning to a different color space, namely HSV
creating a mask from the threshold applied with THRESH_BINARY setting
creating a second mask from the threshold applied with THRESH_BINARY_INV setting, thus creating a mask with opposite values from the first one
converting the two masks to gray, so they are only made of one channel
setting the color of the work mat using set-to, following the first mask
setting the color of the work mat using again set-to, but following the second mask
That’s it!
In coding happiness, we will create a low-high! function that does the algorithm described in the preceding.
The low-high! function is composed of cvt-color!, threshold, and set-to, all functions you already have seen.
We will call it on the rose picture, with a threshold of 150 and a white smoke to light blue split.
Great. But, you ask, do we really need to create two masks for this? Indeed, you do not. You can do a bitwise operation perfectly on the first mask. To do this, simply comment out the second mask creation and use bitwise-not! before calling set-to the second time.
From there, you could also apply thresholds on different color maps, or create ranges to use as threshold values.
Another idea here is, obviously, to just hot-space-queen-ize any picture.
In case you are wondering, the following snippet does that for you.
This really just is calling low-high! four times, each time with colors from the Queen album Hot Space, from 1982.
You really know how to set the mood
And you really get inside the groove
Cool cat
Queen – “Cool Cat”
Channels by Hand
Whenever you are about to play with channels of a mat, remember the opencv split function. The function separates the channels in a list of independent mats, so you can entirely focus on only one of them.
You can then apply transformations to that specific mat, without touching the others, and when finished, you can return to a multichannel mat using the merge function, which does the reverse and takes a list of mats , one per channel, and creates a target mat combining all the channels into one mat.
To see that in action, suppose you have a simple orange mat (Figure 3-10).
If you want to turn the orange mat into a red one, you would simply set all the values of the green channel to 0.
So, you start by splitting the RGB channels into three mats; then, set all the values of the second mat to 0 and merge all three mats into one.
First, let’s split the mat into channels, and see the content of each of them.
In happy coding, this gives
The three channels are now separated into three elements in the list. You can look at the content of each channel simply by using dump.
For example, dump of the blue channel:
or dump of the green channel :
Finally, dump of the red channel:
From there, let’s turn all those 154 values in the green channel to 0.
And then, let’s merge all the different mats back to a single mat and get Figure 3-11.
The green intensity on all pixels in the mat was uniformly set to 0, and so with all the blue channel values already set to 0, the resulting mat is a completely red one.
We can combine all the different steps of this small exercise and create the function update-channel!, which takes a mat, a function, and the channel to apply the function to and then returns the resulting mat.
Let’s try a first version using u/mat-to-bytes and u/bytes-to-mat! to convert back and forth between mat and byte arrays.
This gets complicated, but is actually the easiest version I could come up with to explain the flow of the transformation .
split the channels into a list
retrieve the target channel’s mat
convert the mat to bytes
apply the function to every byte of the channel mat
turn the byte array back to a mat
set that mat to the corresponding channels in the list
merge the channels into the resulting mat
This should now, at least, read almost sequentially as in the following:
Now let’s get back to my sister’s cat, who’s been sleeping on the couch for some time. Time to tease him a bit and wake him up.
With the help of the update-channel! function, let’s turn all the blue and green channel values to their maximum possible values of 255. We could have written a function that applies multiple functions at the same time, but for now let’s just call the same function one by one in a row.
This newly created function can also be combined with converting colorspace.
Thus, switching to HSV color space before calling update-channel! gives you full control over the mat’s color.
The preceding code applies a blue filter, leaving saturation and brightness untouched, thus still keeping the image dynamics .
Of course, you could try with a pink filter, setting the filter’s value to 150, or red, by setting the filter’s value to 120, or any other possible value. Try it out!
Personally, I also like the YUV switch combined with maximizing all the luminance values (Y).
Transform
If you remember transform , you could also apply different sorts of transformation using the opencv transform function.
To understand the background of transform a bit, let’s get back once again to the usual byte-per-byte matrix manipulation, first on a one-channel 3×3 mat that we would like to make slightly darker.
This can be viewed with the following code (Figure 3-15).
Then we define a 1×1 transformation matrix, with one value of 0.7.
Next, we apply the transformation in place and also dump the result to see the values out from the transformation.
Calling the transform function has the effect of turning all the values of the input matrix to their original value multiplied by 0.7.
The result is shown in the following matrix:
It also means that the visuals of the mat have become darker (Figure 3-16):
The bytes of the source mat are all multiplied by the value in the 1×1 mat;
It’s actually easy to apply custom transformation .
Those transformations work much the same for mats with multiple channels. So, let’s grab an example and move to a colored colorspace (yeah, I know) using cvt-color!
Because the mat is now made of three channels, we now need a 3×3 transformation matrix.
The following transformation mat will give more strength to the blue channel.
[2 0 0] boosts the values of the blue channel by 2, and does not affect green or red output values
[0 1 0] keeps the green channel as is, and does not contribute to other channels in the output
[0 0 1] keeps the red channel as is, and similarly does not contribute to other channels in the output
Since there is definitely no way we can leave my sister’s cat in peace, let’s apply a similar transformation to it.
The code is exactly the same as the preceding small mat example, but applied on an image .
If you wanted blue in the input to also influence red in the output, you could use a matrix slightly similar to the following:
You can understand why by now, right? [2 0 1.1] means that the blue in the input is gaining intensity, but that it also contributes to the intensity of red in the output.
You should probably try a few transformation matrices by yourself to get a feel for them.
So, now, how could you increase the luminosity of a mat using a similar technique?
Yes, that’s right: by converting the matrix to HSV colorspace first, then multiplying the third channel and keeping the others as they are.
The following sample increases the luminosity by 1.5 in the same fashion .
Artful Transformations
To conclude this recipe, let’s play a bit with luminosity and contours to create something a bit artistic.
We want to create a watercolor version of the input picture, by maximizing the luminosity. We also want to create a “contour” version of the image, by using opencv’s canny quick feature of contour detection. Then finally, we will combine the two mats for a pencil-over-watercolor effect.
First, let’s work on the background. The background is created by performing two transformations in a row: one to max out the luminosity in the YUV color space, the other to get it more vivid by increasing blue and red colors.
If you get a result that is too transparent, you could also add another transformation at the end of the pipeline to increase contrast ; this is easily done in another colorspace, HSV.
Next is the foreground. The front cat is created using a call to opencv’s canny function. This time, this is done in the one-channel gray color space.
Then, the two mats are combined using a simple call to the function bitwise-and, which merges two mats together by doing simple “and” bit operations.
While the pink color may not be your favorite, you now have all the tools to modify to your liking the flows presented in this recipe to create many variations of artful cats, with different background colors and also different foregrounds.
But please. No dogs.
3.2 Creating Cartoons
Be yourself. No one can say you’re doing it wrong.
Problem
You have seen a very simple way of doing cartoon artwork using canny, but you would like to master a few more variations of doing cartoony artwork.
Solution
Most of the cartoon-looking transformations can be creating using a variation of grayscale, blur, canny, and the channel filter functions that were seen in the previous recipe.
How it works
You have already seen the canny function , famous for easily highlighting shapes in a picture. It can actually also be used for cartooning a bit. Let’s see that with my friend Johan.
In this recipe, Johan was loaded with the following snippet:
A naïve canny call would look like this, where 10.0 and 90.0 are the bottom and top thresholds for the canny function, 3 is the aperture, and true/false means basically superhighlight mode or standard (false).
You already know that we can use the result of the canny function as a mask and for example do a copy of blue over white (Figure 3-25).
That is quite a few lines showing in the picture. By reducing the range between the two threshold values, we can make the picture significantly clearer and look less messy.
The result is nice, but it still seems that there are quite a few extra lines that should not be drawn.
The technique usually used to remove those extra lines is to apply a median-blur or a gaussian-blur before calling the canny function.
Gaussian blur is usually more effective; do not hesitate to go big and increase the size of the blur to at least 13×13 or even 21×21, as shown in the following:
Do you remember the bilateral filter function ? If you use it after calling the canny function, it also gives some interesting cartoon shapes, by putting emphasis where there are more lines coming out of the canny effect.
You would remember that the focus of the bilateral filter is on reinforcing the contours. And indeed, that is what is achieved here.
So, play around with parameters and see what works for you. The whole Origami setup is there to give immediate feedback anyway.
Also, canny is not the only option. Let’s see other techniques to achieve cartoon effects.
Bilateral Cartoon
The bilateral filter is actually doing a lot of the cartoon work, so let’s see if we can skip the canny processing and stick with just using the bilateral filter step.
turn the input image to gray
apply a very large bilateral filter
apply successive smoothing functions
then turn back to an RGB mat
A possible implementation is shown in the following:
The output of cartoon-0! applied to Johan makes it to Figure 3-30.
Here again, the parameters of the bilateral filter pretty much make all the work.
Changing (bilateral-filter! 10 250 30) to (bilateral-filter! 9 9 7) gives a completely different feeling.
Grayed with Update Channel
The last technique of this recipe will take us back to use the update-channel! function written in the previous recipe.
turns the gray channel’s value to 0 if the original value is less than 70;
turns it to 100 if the original value is greater than 80 but less than 180; and
turns it to 255 otherwise.
This gives the following slightly long but simple pipeline:
The output of the pipeline looks great, but the pixels have had quite a bit of processing, so it is hard to tell what’s inside each of them at this stage, and postprocessing after that needs a bit of care.
Say you want to increase the luminosity or change the color of the preceding output; it is usually better to switch again to HSV color space and increase the luminosity before changing anything on the colors, as highlighted in the following:
As a bonus, we also just flipped the image horizontally to end this recipe on a forward-looking picture!
3.3 Creating Pencil Sketches
Problem
You have seen how to do some cartooning for portraits , but would like to give it a more artistic sense by combining front sketching with deep background colors.
Solution
To create backgrounds with impact, you will see how to use pyr-down and pyr-up combined with smoothing methods you have already seen.
To merge the result, we will again be using bitwise-and.
How it works
The goal here is to create a painted-looking version of that picture.
A goal without a plan is just a wish.
Phase 1: we completely remove all the contours of the picture by smoothing out the edges and doing loops of decreasing the resolution of the picture. This will be the background picture.
Phase 2: We do the opposite, meaning we focus on the contours, by applying similar techniques to what was done in the cartoon recipe, where we turn the picture to gray, find all the edges, and give them as much depth as possible. This will be the front part.
Phase 3: Finally, we combine the results of phase 1 and phase 2 to get the painting effect that we are looking for.
Background
pyr-down! is probably new to you. This decreases the resolution of an image . Let’s compare the mats before and after applying the change of resolution done by the following snippet.
Before:
After:
Basically, the resolution of the mat has been divided by 2, rounded to the pixel. (Yes, I have heard stories of 1/2 pixels before, but beware… those are not true!!)
To create the background effect, we actually need a mat of the same size as the original, so there is a need to resize the output to the size of the original.
The first idea is of course to simply try the usual resize! function:
Let’s try something else. There is a reverse function of pyr-down, named pyr-up, which doubles the resolution of a mat. To use it effectively, we can apply pyr-up in a loop , and loop the same number of times as done with pyr-down.
The background is finalized by applying blur in the mat in between the pyr-down and pyr-up dance.
So:
The output is kept for later, and that’s it for the background ; let’s move to the edge-finding part for the foreground.
Foreground and Result
The foreground is going to be mostly a copy-paste exercise of the previous recipe. You can of course create your own variation at this stage; we will use here a cartooning function made of a median-blur and an adaptive-threshold step.
To finish the exercise, we now combine the two mats using bitwise-and. Basically, since the edges are black, a bitwise-and operation keeps them black, and their values will be copied over as they are to the output mat.
This will have the consequence of copying the edges over unchanged onto the target result, and since the remaining part of the edges mat is made of white, bitwise-and will be the value of the other mat, and so the color of the background mat will take precedence.
With the adaptive threshold step, you can tune the way the front sketching looks.
We used 9 as edges-thickness and 7 as edges-number in the first sketch; let’s see what happens if we put those two parameters to 5.
It’s now up to you to play and improvise from there!
Summary
the factors, e.g., the number of loops in the dance, used to turn the resolution down and then turn it up again
the parameters of the bilateral filter of the background
the parameters of the adaptive threshold of the foreground
The sketch! function is made of smoothing! and edges!. First, let’s use smoothing! to create the background.
Then edges! to create the foreground.
Finally, we can use sketch!, the combination of background and foreground.
Calling sketch! is relatively easy. You can try the following snippet:
A few others have been put in the samples, but now is indeed the time to take your own pictures and give those functions and parameters a shot.
3.4 Creating a Canvas Effect
Problem
Creating landscape art seems to have no more secrets for you, but you would like to emboss a canvas onto it, to make it more like a painting .
Solution
This short recipe will reuse techniques you have seen, along with two new mat functions: multiply and divide.
With divide, it is possible to create burning and dodging effects of a mat, and we will use those to create the wanted effect.
With multiply, it is possible to combine mats back with a nice depth effect, and so by using a paper-looking background mat, it will be possible to have a special draw on canvas output.
How it works
We will take another picture from the French Alps—I mean why not!—and since we would like to make it look slightly vintage, we will use an image of an old castle.
We first start by applying a bitwise-not!, then a gaussian-blur on a gray clone of the source picture; this is pretty easy to do with Origami pipelines.
We will need a grayed version for later as well, so let’s keep the two mats gray and gaussed separate.
We will use this gaussed mat as a mask. The magic happens in the function dodge!, which uses the opencv function divide on the original picture, and an inverted version of the gaussed mat.
Hmmm… okay. What does divide do? I mean, you know it divides things, but at the byte level, what is really happening?
Let’s take two matrices, a and b, and call divide on them for an example.
The output of the divide call is
which is
which gives
then, given that OpenCV considers that dividing by 0 equals 0:
Now, let’s call dodge! on the gray mat and the gaussed mat:
Apply the Canvas
Now that the main picture has been turned to a crayon-styled art form, it would be nice to lay this out on a canvas-looking mat. As presented, this is done using the multiply function from OpenCV.
Now we will create the apply-canvas! function, which takes the front-end sketch, and the canvas, and applies the multiply function between them. (/ 1 256.0) is the value used for the multiplication; since these are gray bytes here, the bigger the value the whiter, and so here (/ 1 256.0) makes the dark lines stand out quite nicely on the final result.
Whoo-hoo. Almost there; now let’s call this newly created function
Now is obviously the time for you to go and find/scan your own old papers, to try a few things using this technique ; or why not reuse the cartoon functions from previous recipes to lay on top of the different papers?
3.5 Highlighting Lines and Circles
Problem
This recipe is about teaching how to find and highlight lines , circles, and segments in a loaded mat.
Solution
A bit of preprocessing is usually needed to prepare the image to be analyzed with some canny and smoothing operations .
Once this first preparation step is done, finding circles is done with the opencv function hough-circles.
The version to find lines is called hough-lines, with its sibling hough-lines-p, which uses probability to find better lines.
Finally, we will see how to use a line-segment-detector to draw the found segments.
How it works
Find Lines of a Tennis Court with Hough-Lines
The first part of this tutorial shows how to find lines within an image. We will take the example of a tennis court .
Preparing the target for the hough-lines function is done by converting the original tennis court picture to gray, then applying a simple canny transformation.
Lines are collected in a mat in the underlying Java version of opencv, and so, no way to avoid this, we will also prepare a mat to receive the resulting lines .
The hough-lines function itself is called with a bunch of parameters. The full underlying polar system explanation for the hough transformation can be found on the OpenCV web site:
https://docs.opencv.org/3.3.1/d9/db0/tutorial_hough_lines.html
You don’t really need to read everything just now, but it’s good to realize what can be done and what cannot.
For now, we will just apply the same parameters suggested in the linked tutorial.
The resulting mat of lines is made of a list of rows with two values, rho and theta, on each row.
Creating the two points required to draw a line from rho and theta is a bit complicated but is described in the opencv tutorial .
For now, the following function does the work for you.
Note that when calling hough-lines, changing the parameter with value 1 to a value of 2 gives you way more lines, but you may need to filter the lines yourself afterward.
Also by experience , changing the Math/PI rounding from 180 to 90 gives fewer lines but better results.
Hough-Lines-P
Another variant of the hough-lines function, named hough-lines-p , is an enhanced version with probabilistic mathematics added, and it usually gives a better set of lines by performing guesses.
To try hough-lines with P, we will this time take the example of… a soccer field.
As per the original hough-lines example, we turn the soccer field to gray and apply a slight gaussian blur to remove possible imperfections in the source image.
Let’s now make a canny version of the court to create the edges.
Now, we call hough-lines-p. The parameters used are explained in line in the following code snippet. Lines are expected to be collected from the newly created edges mat .
The parameters are ready; let’s call hough-lines-p, with the result being stored in the lines mat.
This time, the lines are slightly easier to draw than with the regular hough-lines function. Each line of the result mat is made of four values, for the two points needed to draw the line.
Finding Pockets on a Pool Table
No more running around on a court; let’s move to… the billiard table!
In a similar way, opencv has a function named hough-circles to look for circle-looking shapes. What’s more, the function is pretty easy to put in action.
You can’t knock on opportunity’s door and not be ready.
Bruno Mars
Let’s get the pool table ready first.
With hough-circles, it seems you can actually get better results by bypassing the canny step in the preprocessing .
The following snippet now shows where to put values for the min and max radius of the circles to look for in the source mat.
Here again, circles are collected in a mat, with each line containing the x and y position of the center of the circle and its radius.
Finally, we simply draw circles on the result mat with the opencv circle function.
So defining precisely what is searched for is the recipe for success in most of your OpenCV endeavors (and maybe other ones too…).
And so, to avoid false positives here, it is also probably a good idea to filter on colors before accepting and drawing the lines. Let’s see how to do this next.
Finding Circles
In this short example, we will be looking for red circles in a mat where circles of multiple colors can be found.
You may not see it if you are reading straight from the black-and-white version of the book, but we will be focusing on the large bottom left circle, which is of a vivid red.
If you remember lessons from the previous recipes, you already know we need to change the color space to HSV and then filter on a hue range between 0 and 10.
The following snippet shows how to do this along with some extra blurring to ease processing later on.
Now we can apply the same hough-circles call as was seen just previously; again, the circle will be collected in the circle mat, which will be a 1×1 mat with three channels .
Using Draw Segment
Sometimes, the easiest may be to simply use a technique using the provided segment detector. It is less origami friendly, since the methods used are straight Java method calls (so prefixed with a dot “.”), but the snippet is rather self-contained.
Let’s try that on the previously seen soccer field. We’ll load it straight to gray this time and see how the segment detector behaves.
We call detect on the line-segment-detector, using Clojure Java Interop for now.
At this stage, the lines mat metadata is 161*1*CV_32FC4, meaning 161 rows, each made of 1 column and 4 channels per dot, meaning 2 points per value.
The detector has a helpful drawSegments function, which we can call to get the resulting mat .
3.6 Finding and Drawing Contours and Bounding Boxes
Problem
Since identifying and counting shapes are at the forefront of OpenCV usage, you would probably like to know how to use contour-finding techniques in Origami .
Solution
Apart from the traditional cleanup and image preparation, this recipe will introduce the find-contours function to fill in a list of contours.
Once the contours are found, we need to apply a simple filter to remove extremely large contours like the whole pictures as well as contours that are really too small to be useful.
Once filtering is done, we can draw the contours using either handmade circles and rectangles or the provided function draw-contours.
How it works
Sony Headphones
They are not so new anymore, but I love my Sony headphones . I simply bring them everywhere, and you can feed your narcissism and get all the attention you need by simply wearing them. They also get you the best sound, whether on the train or on the plane…
Let’s have a quick game of finding my headphones’ contours.
My headphones still have a cable, because I like the sound better still, whatever some big companies are saying.
First, we need to prepare the headset to be easier to analyze. To do this, we create a mask of the interesting part, the headphones themselves.
Then with the use of the mask, we create a masked-input mat that will be used to ease the finding contours step.
Have you noticed? Yes, there was an easier way to create the input, by simply creating a noninverted mask in the first place, but this second method gives more control for preparing the input mat.
So here we basically proceed in two steps. First, set all the pixels of the original mat to black when the same pixel value of the mask is 1. Next, set all the other values to white, on the opposite version of the mask .
Now that the mat that will be used to find contours is ready, you can almost directly call find-contours on it.
find-contours takes a few obvious parameters, and two ones, the last two, that are a bit more obscure.
RETR_LIST is the simplest one, and returns all the contours as a list, while RETR_TREE is the most often used, and means that the contours are hierarchically ordered.
CHAIN_APPROX_NONE means all the points of the found contours are stored. Usually though, when drawing those contours, you do not need all of the points defining them. In case you do not need all of the points, you can use CHAIN_APPROX_SIMPLE, which reduces the number of points defining the contours.
It eventually depends how you handle the contours afterward. But for now, let’s keep all the points!
Alright, now let’s draw rectangles to highlight each found contour. We loop on the contour list, and for each contour we use the bounding-rect function to get a rectangle that wraps the contour itself.
The rectangle retrieve from the bounding-rect call can be used almost as is, and we will draw our first contours with it.
Right. Not bad. It is pretty obvious from the picture that the big rectangle spreading over the whole picture is not very useful. That’s why we need a bit of filtering.
not too small, meaning that the area they should cover is at least 10,000, which is a surface of 125×80,
nor too big, meaning that the height shouldn’t cover the whole picture.
That filtering is now done in the following snippet.
And so, drawing only the interesting-contours this time gives something quite accurate.
Drawing circles instead of rectangles should not be too hard, so here we go with the same loop on interesting-contours , but this time, drawing a circle based on the bounding-rect.
Finally, while it’s harder to use for detection processing, you can also use the opencv function draw-contours to nicely draw the free shape of the contour.
We will still be looping on the interesting-contours list. Note that the parameters may feel a bit strange, since draw-contours uses an index along with the list instead of the contour itself, so be careful when using draw-contours.
Things are not always so easy, so let’s take another example up in the sky!
Up in the Sky
This second example takes hot-air balloons in the sky, and wants to draw contours on them.
Unfortunately, using the same technique as previously shown to prepare the picture does not reach a very sexy result .
So, let’s try another technique. What would you do to get a better mask?
Yes—why not? Let’s filter all this blue and create a blurred mask from it. This should give you the following snippet.
We will now use the complement version of the mask to find the contours.
Using the finding-contours function has no more secrets to hide from you. Or maybe it does? What’s the new-point doing in the parameter list? Don’t worry; it is just an offset value, and here we specify no offset, so 0 0.
Contours are in! Let’s filter on the size and draw circles around them. This is simply a rehash of the previous example.
Next, let’s filter ahead of the drawing, and let’s use the bounding-rect again to draw rectangles.
And yes indeed, if you checked its content, my-contours has only three elements.
3.7 More on Contours: Playing with Shapes
Problem
Following on the previous recipe, you would like to see what’s returned by the function find-contours. Drawing contours with all the dots is nice, but what if you want to highlight different shapes in different colors?
Also, what if the shapes are hand-drawn, or not showing properly in the source mat?
Solution
We still are going to use find-contours and draw-contours as we have done up to now, but we are going to do some preprocessing on each contour before drawing them to find out how many sides they have.
approx-poly- dp is the function that will be used to approximate shape, thus reducing the number of points and keeping only the most important dots of polygonal shapes. We will create a small function, approx, to turn shapes into polygons and count the number of sides they have.
We will also look at fill-convex-poly to see how we can draw the approximated contours of handwritten shapes.
Lastly, another opencv function named polylines will be used to draw only wireframes of the found contours.
How it works
Highlight Contours
The goal here is to draw the contours of each shape with different colors depending on the number of sides of each shape.
The shapes mat is loaded simply with the following snippet:
As was done in the previous recipe, we first prepare a thresh mat from the input by converting a clone of the input to gray, then applying a simple threshold to highlight the shapes.
Ok, the thresh is ready, so you can now call find-contours on it.
To draw the contours , we first write a dump function that loops on the contours list and draws each one in magenta.
But, as we have said, we would like to use a different color for each contour, so let’s write a function that selects a color depending on the sides of the contour.
Unfortunately, even with CHAIN_APPROX_SIMPLE passed as parameter to find-contours, the number of points for each shape is way too high to make any sense.
So, let’s work on reducing the number of points by converting the shapes to approximations.
Two functions are used from opencv, arc-length, and approx-poly-dp. The factor 0.02 is the default proposed by opencv; we will see its impact with different values slightly later in this recipe.
Using this new approx function, we can now count the number of sides by counting the number of points of the approximation.
The following is the how-many-sides function that simply does that.
Everything is in place; let’s rewrite the dumb draw-contours! function into something slightly more evolved using which-color .
Note how the circle still goes slightly overboard, with too many sides , but that was to be expected.
Hand-Drawn Shapes
But perhaps you were going to say that the shapes were nicely showing already, so you still have some doubts about whether the approximation is really useful or not. So, let’s head to a beautiful piece of hand-drawn art that was prepared just for the purpose of this example.
First, let’s call find-contours and draw the shapes defined by them.
Now this time, let’s try something different and use the function fill-convex-poly from the core opencv package.
It’s not very different from draw-contours, and we indeed just loop on the list and use fill-convex-poly on each of the contours .
As we can see, the contours and shapes are found and can be drawn.
Another way to draw the contours is to use the function polylines. Luckily, the function polylines hides the loop over each element of the contours, and you can just pass in as parameters the contour list as is.
Alright, but again those shapes for now all have too many points.
Let’s again use the approx function that was created, and enhance it so we can specify the factor used by approx-poly-dp.
A higher factor means we force the reduction of points to a greater extent. And so, to that effect, let’s increase the usual value of 0.02 to 0.03.
3.8 Moving Shapes
Problem
This is based on a problem found on stack overflow.
https://stackoverflow.com/questions/32590277/move-area-of-an-image-to-the-center-using-opencv
The problem was “Move area of an image to the center,” with the base picture shown in Figure 3-80.
Solution
I like this recipe quite a lot, because it brings in a lot of origami functions working together toward one goal, which is also the main theme of this chapter.
First, add borders to the original picture to see the boundaries
Switch to the HSV color space
Create a mask by selecting only the color in-range for yellow
Create a submat in the original picture from the bounding rect of the preceding mask
Create the target result mat, of the same size as the original
Create a submat in the target mat, to the place the content. That submat must be of same size, and it will be located in the center.
Set the rest of the target mat to any color …
We’re done!
Let’s get started.
How it works
Alright, so the first step was to highlight the border of the mat, because we could not really see up to where it was extending.
We will start by loading the picture and adding borders at the same time.
We then switch to hsv color space and create a mask on the yellow mark , and this is where Origami pipelines make it so much easier to pipe the functions one after the other.
Next is to find the contours in the newly created mask mat. Note here the usage of RETR_EXTERNAL, meaning we are only interested in external contours, and so the lines inside the yellow mark will not be included in the returned contour list.
Let’s now create an item mat, a submat of the original picture, where the rectangle defining it is made from the bounding rect of the contours.
We now create a completely new mat, of the same size of the item submat, and copy into the content of the segmented item. The background color has to be the same as the background color of the result mat.
Now let’s find the location of the rect that will be the target of the copy. We want the item to be moved to the center, and the rect should be of the same size as the original small box mat.
Alright, everything is in place; now we create the result mat and copy the content of the segmented item through a copy, via the submat, at the preceding computed centered location .
And that’s it.
3.9 Looking at Trees
Problem
This is another recipe based on a stack overflow question. The interest this time is to focus on a tree plantation, and before counting the trees, being able to highlight them in an aerial picture.
The referenced question is here:
Solution
Recognizing the trees will be done with a call to in-range as usual. But the results, as we will see, will still be connected to each other, making it quite hard to actually count anything.
We will introduce the usage of morphology-ex! to erode the created mask back and forth, thus making for a better preprocessing mat , ready for counting.
How it works
Eventually, you would want to count the trees, but right now it is even difficult to see them with human eyes. (Any androids around?)
Let’s start by creating a mask on the green of the trees.
The trick of this recipe comes here. We will apply a MORPH_ERODE followed by a MORPH_OPEN on the in-range-pict mat. This will have the effect of clearing up the forest, and gives each tree its own space .
Morphing is done preparing a mat to pass, as parameter, a kernel matrix created from a small ellipse.
If you call dump on elem, you will find its internal representation.
We then use this kernel matrix, by passing it to morpholy-ex!.
To finish, we just apply a simple coloring on the original mat to highlight the position of the trees for the human eye. (Still no androids around?)
This could be great to do in real time over a video stream.
You also already know what exercise awaits you next. Count the number of trees in the forest by using a quick call to find-contours …
This is of course left as a free exercise to the reader!
3.10 Detecting Blur
Problem
You have tons of pictures to sort, and you would like to have an automated process to just trash the ones that are blurred.
Solution
The solution is inspired from the pyimagesearch web site entry http://pyimagesearch.com/2015/09/07/blur-detection-with-opencv/ , which itself is pointing at the paper variation of the Laplacian by Pech-Pacheco et al, “Diatom autofocusing in brightfield microscopy: A comparative study.”
It does highlight cool ways of putting OpenCV and here origami into actions quickly for something useful.
Basically, you need to apply a Laplacian filter on the one-channel version of your image. Then, you compute the deviation of the result from the preceding and check if the deviation is below a given threshold.
The filter itself is applied with filter-2-d!, while the variance is computed with mean-std-dev.
How it works
The Laplacian matrix/kernel to be used for the filter puts emphasis on the center pixel and reduces emphasis on the left/right top/bottom ones.
This is the Laplacian kernel that we are going to use.
Let’s apply this kernel with filter-2-d!, followed by a call to mean-std-dev to compute the median and the deviation.
When processing a picture, you can view the results of the averages with dump, since they are matrices. This is shown in the following:
Finally, the value to compare to detect blur will be the deviation raised to the power of 2.
We will then get a value that will be compared to 50. Lower than 50 means the image is blurred. Greater than 50 means the image is showing as not blurred.
Let’s create an is-image-blurred? function made of all the preceding steps:
Now let’s apply that function to a few pictures.
Now, probably time to go and sort all your beachside summer pictures…
But yes, of course, yes, agreed, not all blurred pictures are to be trashed.
3.11 Making Photomosaics
Problem
In a project lab, now maybe 20 years ago, I saw a gigantic Star Wars poster, made of multiple small scenes of the first movie, A New Hope.
The poster was huge, and when seen from a bit far away, it was actually a picture of Darth Vader offering his hand to Luke.
The poster left a great impression, and I always wanted to do one of my own. Recently, I also learned there was a name for this type of created picture: photomosaic .
Solution
The concept is way simpler than what I originally thought. Basically, the hardest part is to download the pictures.
You mainly need two inputs, a final picture, and a set of pictures to use as subs.
The work consists of computing the mean average of the RGB channels for each picture, and creating an index from it.
Once this first preparation step is done, create a grid over the picture to be replicated, and then for each cell of the grid, compute the norm between the two averages: the one from the cell, and the one from each file of the index.
Finally, replace the sub of the big picture with the picture from the index that has the lowest mean average, meaning the picture that is visually closer to the submat.
Let’s put this in action!
How it works
The first step is to write a function that computes the mean average of the colors of a mat. We use again mean-std-dev for that effect, and since we are only interested in the mean for this exercise, this is the result returned by the function.
Let’s call this on any picture to see what happens.
The return values are shown in the following. Those values are the mean average for each of the three RGB channels.
Let’s sidestep a bit and compare the norms of three matrices : ex1, ex2, and ex3. Looking at their content, you can “feel” that ex1 and ex2 are closer than ex1 and ex3.
This is confirmed by the result of the output of the norm function, which calculates the distance between the matrices.
And this is what we are going to use. First, we create an index of all the files available. The index is a map created by loading each image as a mat, and computing its mean-average-bgr.
The output of the function is a map where each element is a set of key,val like filepath -> mean-average-bgr.
To find the closest image now that we have an index, we compute the norm of the mat (or submat later on) considered, and all the possible mean-bgr matrices of our index.
We then sort and take the lowest possible value . This is what find-closest does.
apply-to-vals is a function that takes a hashmap and a function, applies a function to all the values in the map, and leaves the rest as is.
The hardest part is done; let’s get to the meat of the photomosaic algorithm.
The tile function is a function that creates a grid of the input picture and retrieves submats, one for each tile of the grid.
It then loops over all the submats one by one, computes the submat’s mean color average using the same function, and then calls find-closest with that average and the previously created index.
The call to find-closest returns a file path, which we load a submat from and then replace the tile’s submat in the target picture, just by copying the loaded mat with the usual copy-to.
See this in the function tile written here.
The main entry point is a function named photomosaic, which calls the tile algorithm by just creating the index of averages upfront, and passing it to the tile function.
Folder of jpg images
The picture we want to mosaic
The size of the grid
Here is a simple sample:
Cats used in the pictures are all included in the examples, not a single cat has been harmed, and so now is probably your turn to create your own awesome-looking mosaics … Enjoy!