Do-It-Yourself Video Stream

You could skip this small section of the recipe, but it’s actually quite informative to know what is behind the scenes.

The simple idea of video stream manipulation starts with creating an opencv videocapture object that accesses available video devices.

That object can then return you a mat object, just like all the mat objects you have used so far. It is possible to then act on the mat object, and in the simplest case show the mat in a frame on the screen.

Origami uses something similar to u/imshow to display mats taken from video, but for this very first example let’s simply use u/imshow to display the mat.

Here, we do require another namespace : [opencv3.video :as v], but later on you will see that this step is not necessary, and you would require that extra video namespace only when using opencv video functions directly.

Let’s see how it goes by going through the following code example.

First, we create the videocapture object, which can access all the webcams of your host system.

We then open the camera with ID 0. That is probably the default in your environment, but we will also see later how to play with multiple devices.

We need a window to display the frame recorded from the device, and sure enough, we’ll create a binding named window. This window will be set to have a black background.

We then create a buffer to receive video data, as a regular OpenCV mat.

The core video loop will copy content to the buffer mat using the read function on the capture object, and then it will show the buffer in the window, using the function u/re-show.

At this stage, you should see frames showing up in a window on your screen, as in Figure 4-1.

Finally, when the loop has finished, the webcam is released using the release function on the capture object.

This should also have the effect of turning off the camera LED of your computer. One thing to think about at the end of this small exercise is … yes, this is a standard mat object that was used as a buffer in the display loop , and so, yes, you could already plug in some text or color conversion before displaying it.

One-Function Webcam

Now that you understand how the underlying webcam handling is done, here is another slightly shorter way of getting you to the same result, using u/simple-cam-window .

In this small section, we want to quickly review how to take the stream and manipulate it using that function.

In its simplest form, simple-cam-window is used with the identity function as the parameter. As you remember, identity takes an element and returns it as is.

Providing you have a webcam connected, this will start the same streaming video with the content of the stream showing in a frame.

The function takes a single parameter, which is the function applied to the mat before it is shown inside the frame.

Sweet. We’ll get back to it in a few seconds, but for now, here’s what you’ll find: simply converting the recording frames to a different colormap , you could pass an anonymous function only using apply-color-map!.

With the immediate result showing in Figure 4-2.

In the second version of u/simple-cam-window , you can specify settings for the frame and the video recording, all of this as a simple map passed as the first parameter to simple-cam-window.

For example:

In the map, the video key specifies the device ID, the device to take stream from, and the size of the frame to record. Note that if the size is not according to what the device is capable of, the setting will be silently ignored.

In the same parameter map, the frame key can specify the parameter, as seen in previous chapter, with the background color, the title, and the size of the window.

Ok, great; all set with the basics. Let’s play a bit.

Transformation Function

The identity function takes an element and returns it as is. We saw how identity worked in the first cam usage, by returning the mat as it was recorded by the opencv framework.

Not so difficult with all the knowledge you have gathered so far. Let’s write a small origami pipeline in a function my-fn! to do the image transformation:

Note here that the pipeline returns the transformed mat. Now let’s use this newly created pipeline on a still image .

And let’s enjoy a simple winter feline output (Figure 4-3).

And then, if you are in Starbucks and using your laptop webcam, you can use the new function my-fn! straight onto a video stream by passing it as an argument to simple-cam-window .

Which would give you something like Figure 4-4.

Two Frames, or More, from the Same Input Source

This is a convenient method when trying to apply two or more functions from the same source. This is really only a matter of using the clone function to avoid memory conflicts with the source buffer.

Here, we create a function that takes the buffer as input, and then concatenates two images created from the same buffer. The first image on the left will be a black-and-white version of the stream, while the right one will be a flipped version of the buffer.

Note that we use the clone twice for each side of the concatenation (Figure 4-5).

You can push this method even further by cloning the input buffer as many times as you want; to highlight this, here is another example of applying a different color map three times onto the same input buffer.

And the result is shown in Figure 4-6.

Finding Movement in Black and White

The first step is to take a buffer and create a cleaned (via a blur) gray version of it.

We are not interested to display this mat, but just to perform arithmetic on it; we will convert the mat to a 32-bit float mat, or in opencv language CV_32F.

This function will be used to prepare a gray version of the mat. Let’s now work on computing the accumulated average and a diff between the average and the most recent buffer.

We’ll create another function, find-movement, which will highlight, in black and white, recent movement in the picture.

That function will get a Clojure atom, avg, as a parameter to keep track of the average value of the video’s incoming mat objects. The second parameter is the usual buffer mat passed to the callback. The function will display the frame-delta.

In the first if switch, we make sure the average mat, stored in the atom, is initialized with a proper value from the incoming stream.

Then a diff is computed using absdiff, onto which we apply a short threshold-dilate-cvt-color pipeline to show the movements directly.

We finally define a function wrapping the find-movement function, with an inlined Clojure atom. That atom will contain the average of the mat objects.

Time to put those functions in action with u/simple-cam-window.

This is shown in Figure 4-17.

We would like to show movements here, but because the amount of black ink required to print this is going to scare the publisher, let’s add a bitwise operation to do a black-on-white instead and see how the live progression goes.

Let’s update the find-movement function with a bitwise-not! call on the frame-delta mat. Before that, we need to convert the matrix back to something we can work on, using opencv’s convert-to! function, with a type target CV_8UC3, which is usually what we work with.

Good; let’s call simple-cam again. Wow. Figure 4-18 now looks a bit scary.

And if you stop getting agitated in front of your computer, the movement highlights are stabilizing and slowly moving to a fully white mat, as shown in the progression of Figure 4-19.

Find and Draw Contours

At this stage, it would be easy to find and draw contours to highlight movement on the original colored buffer.

Let’s find contours of the nice movement mat that you managed to create.

A few more lines are added to the find-movement function, notably the finding contours on the delta mat and the drawing on the color mat.

You have seen all of this find-contours dance in the previous chapter, so let’s get down to the updated code.

Calling this new version of the find-movement function gives something like Figure 4-20, but you can probably be way more creative from there.

On a Still Image

Here we want to call grabcut and separate a depth layer from the other ones.

The idea with grabcut is to prepare to use either a rectangle or a mask on the input picture and pass it to the grabcut function.

The result stored in that single output mask will contain a set of 1.0, 2.0, or 3.0 scalar values depending on what grabcut thinks is part of each of the different layers.

Then we use opencv compare on this mask and another fixed 1×1 mat of the scalar value of the layer we would like to retrieve. We obtain a mask only for the layer of interest.

Finally, we do a copy of the original image, on an output mat, using the mask created in step 2.

Ready? Let’s go for a cat example.

First, we load one of those cat pictures that we love so much and turn it to a proper working-size mat object.

The loaded cat picture is shown in Figure 4-21.

Then, we define a mask mat , which will receive the output of the grabcut call, namely, per-pixel information about the layer info.

We also define a rectangle for the region of interest (ROI) of where we want the grabcut to be done, here almost the full picture, mostly just removing the borders.

Now that we have all the required inputs for grabcut, let’s call it with the mask, the ROI, and the grabcut init param , here GC_INIT_WITH_RECT. The other available method is to use GC_INIT_WITH_MASK , which as you probably have guessed is initialized with a mask instead of a rect.

Grabcut has been called. To get an idea of the retrieved content of the output, let’s quickly see the matrix content on a small submat of the mask .

If you try it yourself, you would see values like

Another submat dump elsewhere in the mat gives a different result:

In turn, this gives

We can guess from this different matrix that the layer is different.

The idea here is to retrieve a mask made of all the same values, so now let’s create a mask from all the pixels that are contained in layer 3, meaning that they are made of 3.0 values.

We’ll call this the fg-mask, for foreground mask .

The cat foreground mask is shown in Figure 4-22.

We can then use copy-to from the original input image, and the fg-mask on a new black mat of the same size as the input.

And we get the mat of Figure 4-23.

Notice how we get a bit of an approximation where the two kittens cross each other, but overall the result is pretty effective.

Before moving on, let’s quickly retrieve the complementary mask, the background mask , by focusing on the layer with scalar values of 2.0.

First, we create a mask again to receive the output, this time bg-mask.

The result for the background mask is shown in Figure 4-24.

Then, simply do a copy similar to the one that was done for the foreground.

And the result is shown in Figure 4-25.

Now that you have seen how to separate the different layers on a still image, let’s move on to video streaming.

On a Video Stream

As you may have noticed, the grabcut step in the preceding example was very slow, mostly due to a lot of heavy computations done to achieve a clean separation of the different layers. But how bad is it?

Let’s give it a quick try with a first dumb version of a real-time grabcut.

We’ll call this function in-front- slow , and basically just compile the steps we have just seen in the still example in a single function.

And then, let’s use this function as a callback to our now-familiar u/simple-cam-window .

This slowly gives the output seen in Figure 4-26.

As you will quickly realize, this is not very usable as is on a video stream.

The trick here is actually to turn down the resolution of the input buffer, do the grabcut on that lower-resolution mat , and get the grabcut mask. Then, do the copy using the full-sized picture and the mask retrieve from grabcut on a lower resolution.

This time, we’ll create an in-front function , which will be a slightly updated version of the preceding, but now including a pyr-down–pyr-up dance around the grabcut call (Figure 4-27).

To make this easier, we’ll set the number of iterations of the dance as a parameter of the callback .

Then, call simple-cam-window with this new callback.

It’s hard to get the feeling of speed by just reading, so do go ahead and try this locally.

Usually, a factor of 2 for the resolution-down dance is enough, but it depends on both your video hardware and the speed of the underlying processor.

Using Hough-Circles

First, we’ll focus on finding the proper hsv range by taking a one-shot picture of the orange.

First, let’s put the orange on the table (Figure 4-28).

We first switch to hsv color space, then apply the in-range function, and finally dilate the found orange shape a bit so it’s easier for the coming hough-circle call.

In origami, this gives

Now, you’ll remember how to do hough-circles from Chapter 3, so no need to spend too much time on that here. The important thing in this part is to have the proper radius range for the orange, and here we take a 10–50 pixels diameter to identify the orange.

At this stage, you should have only one matching circle for the orange. It is quite important to work on this step until exactly one circle is found.

As a check, printing the circle mat should give you a 1×1 mat, like the following:

Once you have the mat nailed, let’s draw a pink circle on the original image (Figure 4-29).

(def output (clone img))

(dotimes [i (.cols circles)]

  (let [ circle (.get circles 0 i) x (nth circle 0) y (nth circle 1) r (nth circle 2)  p (new-point x y)]

  (opencv3.core/circle output p (int r) color/ color/magenta- 3)))

Everything is there, so let’s wrap up our discoveries as a single function working on the buffer from the video stream; we’ll call that function my-orange!, which is a recap of the previous steps.

Now it’s a simple matter of again passing that callback function to the simple-cam-window.

Figures 4-30 and 4-31 show how the orange is found properly, even in low-light conditions. Winter in the French Alps after a storm did indeed make the evening light, and everything under it, a bit orange.

Using Find-Contours

Instead of looking for a perfect circle, you may be looking for a slightly distorted shape, and this is when using find-contours actually gives better results than hough-circles.

Here we combine the same hsv range found a few minutes ago to select the orange and apply the find-contours technique from Chapter 3.

The find-my-orange! callback brings back the familiar find-contours and draw-contours function calls. Note that we draw the contour of found shapes only if those are bigger than the smallest expected size of the orange.

Giving this callback to simple-cam-window shows Mei playing around with a pink-colored orange in Figure 4-32.

Still Image

The first test is to be able to find the body soap in a simple still picture, like the one in Figure 4-34.

To get started, we need a few more Java object imports, namely, the detector and the extractor , which we will initialize straightaway before doing any processing.

Basic setup is done; we then load the body soap background through a short origami pipeline and ask the detector to detect points on it.

The coming step is not required whatsoever, but drawing the found keypoints gives an idea of where the matcher thinks the important points are in the mat.

This gives a bunch of blue circles, as shown in Figure 4-35.

Of course, it may be useful to clean up and remove imperfections before retrieving keypoints , but let’s check how the matching works on the raw mat.

Note how the intensity of the points is already pretty strong on the body soap itself.

We now repeat the same steps for a body soap–only mat.

Here again, this drawing points part is not required but it helps to give a better idea of what is going on.

The detector result is in Figure 4-36, and again, the keypoints look to be focused on the label of the body soap.

The next step is to extract two feature sets that will then be used with the matcher.

This is simply a matter of calling compute on the extractor with the sets of found points from the previous step.

Now, on to the matching step . We create a matcher through DescriptorMatcher and give it a way to find out matches.

In IT, brute force is always the recommended way to find a solution. Just try every single solution and see if any match.

As was said in the solution summary, each match is rated on how good it is through its distance value .

If printed, each match looks something like the following:

With the distance value , the score of the match itself usually shows up as a value between 0 and 300.

So now, let’s create a quick Clojure function to sort and filter good matches. This is simply done by filtering on their distance property. We will filter on matches that are below 50. You may reduce or increase that value as needed, depending on the quality of the recording.

The draw-matches method is a coding nightmare, but it can be seen as mostly a wrapper around the nightmarish drawMatches Java method from the OpenCV.

We mostly pass the parameters the way they are expected using Java interop and some cleanup on each parameter. We also create the output mat bigger, so that we can fit in the background picture and the body soap on the same mat.

And now, with all this, we can draw the matches found by the matcher, using the preceding function.

We pass it the first and second mats, as well as their respective found key points and the set of matches.

This, surprisingly after all the obscure coding, works very well, as shown in Figure 4-37.

Video Stream

Compared to what you have just been through, the video stream version is going to feel like a breath of fresh air.

We will create a where-is-my-body-soap! function that will reuse the matcher defined in the preceding and run the detector, extractor, and match within the stream callback on the buffer mat.

The previously defined draw-matches function is also reused to draw the matches on the real-time stream.

And you can use that callback to simple-cam-window but … Ah! It seems Mei has found the body soap just before this recipe feature detection could be run!

Figure 4-38 shows both on the video stream.

This brings this recipe, chapter, and book to a humble end. We do hope this gave you plenty of ideas for things to try out by playing with the Origami framework and bringing light to your creation.