Quick Comparison

Windows Store Support

Windows Store support has long been requested by Kinect developers, and the Kinect for Windows v2 now enables this. The APIs are pretty much the same as the ones for WPF. While the market for general Window Store Kinect consumer apps is admittedly bleak, such apps are perfect for special use cases like booths or commercial use because of their accessibility. A great example of a Windows Store app that uses Kinect is 3D Builder, which allows you to scan objects and prepare them for 3D printing.

Unity Support

Much-anticipated Unity support is now available for the Kinect for Windows v2. Most developers should be familiar with Unity by now, but for those still programing their games in BASIC, you should take a serious look at Unity. Unity makes it exceedingly easy for any developer to make nearly AAA-quality games for dirt cheap. With the new Kinect plugin, we can now make complex games with the Kinect without having to maintain an entire development team. In a few easy steps, we can add skeletal tracking to Unity games and do cool stuff like avateering or green-screening. While a paid license was initially required, after the release of Unity 5, anyone can use the Kinect plugin with their apps for free.

Unity apps can be deployed to both the desktop and the Windows Store. A thorough discussion on Unity is available in Chapter 7.

Hand-Gesture Support

There is now support for hand-gesture tracking . The Kinect can let us know whether a user’s hands are in any one of three recognized gestures or two non-gestures. The three recognized gestures, as shown in Figure 2-2, are Open, when a user opens their palm flat; Closed, when a user clasps their hand in a clenched fist; and Lasso, a gesture that is oft-mistaken for flipping the bird but is really a pointing gesture with two fingers (the middle and index). The non-gestures are Unknown and NotTracked. Recognizing these hand gestures in our applications enables better user experiences by facilitating user interactions with prompts (e.g., holding Closed grip to signify that the user confirms the selection).

Figure 2-2. Open, Closed, and Lasso hand gestures

Visual Gesture Builder

There was no official tool with which to integrate gestures into apps with the Kinect for Windows v1, which meant that the average developer had to essentially track gestures with an assortment of conditional statements. This was very prone to error, not to mention irritating to program. The Kinect for Windows v2 comes with Visual Gesture Builder, a tool that uses machine-learning techniques to help you train gestures and integrate them into your app. The tool is very easy to use and requires no prior knowledge of machine learning.

Face & HD Face API

The original Kinect SDK tracked up to 40 points on a user’s face. As with skeletal gestures, we had to make assumptions based on the positional heuristic of face joints to see if a user was making a certain facial gesture. The kind folks at Microsoft went ahead and included a handful of facial gesture–detection algorithms with the Kinect for Windows v2 SDK. These gestures include closed eyes, eyes looking away/not interested, open mouth, moving mouth, happy expression, and neutral expression. Additionally, the Kinect can detect if the user is wearing glasses. The use for this Face API is immediately obvious: we can now determine if a spectator is engaged with our product or not.

There is also an additional HD Face API. HD Face supports a whopping 613 HD face points. It is 3D, whereas the regular Face API is 2D. HD Face can be used for advanced character rigging and character facial animation, among other things.

Kinect Fusion

Kinect Fusion enables 3D object scanning and model creation using the Kinect. The original Kinect Fusion was pretty nifty, but because of throughput issues, low camera resolution, and weakly optimized code, the results were somewhat lackluster. The new Kinect Fusion, depicted in Figure 2-3, has been drastically improved and is much more performant.

Figure 2-3. Kinect Fusion reconstruction of the author working on his book

Kinect Studio

If you have worked with Kinect Studio which was introduced in Kinect for Windows SDK v1.7 onward, you will be glad to know that Kinect Studio makes a comeback in SDK v2. Kinect Studio is an application that allows you to record data with your Kinect and replay it for your apps so as to avoid having to physically debug each time around. It is much more comprehensive than its earlier incarnation, and you now have the capability to toggle the data streams you want to capture. A rundown of the tool will be provided later in this chapter.

Kinect Ripple

The Kinect Ripple is one of the coolest Kinectabilities (can this be a word?) that I have had a chance to play with. Unfortunately, it was never advertised anywhere but within the core Kinect developer circles and in India, and thus never received all the attention that it deserved. Basically, the concept is that you have a projector displaying an image on a wall or screen and another projector displaying an image on the floor. The Kinect is positioned underneath the wall projection, facing away, like in Figure 2-4. The floor projection will become interactive, and you can step on it to activate buttons or tiles. The one on the wall can show a presentation or content based on the interaction with the floor projection. The whole thing can be programmed in JavaScript.

Figure 2-4. A Kinect Ripple system set up at Microsoft’s Hyderabad office

Kinect RoomAlive Toolkit

Another one of the Kinect’s coolest capabilities is RoomAlive . True to its name, it brings your living room to life, and it achieves this by enabling you to do immersive, dynamic projection mapping. The toolkit takes care of calibrating the combined Kinect and projector units (dubbed procams), and any surface covered by these units receives a projection-mapping treatment.

In Figure 2-5, we have a real example of a person playing with a RoomAlive app. Creatures are able to erupt from any surface and attack the user. The user can in turn retaliate by hitting them with their limbs or by firing their rifle. You can see where the HoloLens, in particular with its robot shooting game, got its inspiration.

Figure 2-5. A whack-a-mole type game with a toy rifle (© Microsoft)

Setting up the system is beyond the scope of this book, but if you want to learn more, you can visit www.github.com/Kinect/RoomAliveToolkit . (A future blog post by the author maybe in the works however.

Multi-App Support

The Kinect for Windows v2 now allows for the ability to have multiple apps using the Kinect at the same time. It is a bit of a niche capability, but a very much welcome one in several circumstances. If we created a Dance Dance Revolution–type game, for example, we could set up two monitors with two instances of the game running at the same time for two participants competing against each other, while only running one Kinect. Another use case could be having a therapist run a Kinect exercise suite app for one of her patients while at the same time running a general body-analysis app. The analysis app might need to be utilized for many different exercises, so it saves having to integrate it with every new exercise app that is developed.

Readers

Most of the Kinect’s data is read through readers. The reader’s job is to read from the Data Sources and fire an event each time there is a data frame available for use, which I will explain briefly.

Readers can access frames in either one of two methods: Events and Polling. If you have worked with .NET before, Events should be nothing new to you, but for those of you have not, the gist is that a class can use events to notify other classes that something of interest has occurred. In the case of the Kinect SDK, readers raise events that let us know whenever there is a new frame available for us to use. For your typical WPF or Windows Store application, events are the way to go.

The other method, Polling, is a reverse Event of sorts. If you have dabbled with Microsoft’s XNA, this should be familiar to you. Instead of waiting for the specific class to drive an interrupt, our application code continuously calls or polls the class asking for the current status of whatever you are interested in. So, for example, if you want to know if the Jump button was pressed, your application loop checks to see the status of Jump on each loop, and if it has been pressed you execute the code to handle this. The end result generally is that while you will find out faster whether something of interest has occurred, the system will be taxed unnecessarily as many to most of the calls will return a negative. Of greater concern is the added complexity for any application other than those with the smallest of scopes. Typically, you will go with polling if you are working with game loops or something of the sort.

Accessing Data Sources with readers is as simple as simple can get. Listing 2-1 demonstrates how we would access ColorFrameSource and open a ColorFrameReader.

Listing 2-1. Opening a Data Source with a Reader

private ColorFrameReader colorFrameReader = null;
...
this.colorFrameReader = this.kinect.ColorFrameSource.OpenReader();

When a reader is not being utilized, it is best to pause it to minimize the creation of unnecessary frames. If we reach a point in the application where we no longer need to use the Kinect, we can shut off the Kinect sensor altogether, as we saw in Listing 1-5 in Chapter 1. However, if we only need to temporarily pause the collection of data from one Data Source, we are better off pausing the readers, as shown in Listing 2-2.

Listing 2-2. Pausing a Reader

this.colorFrameReader.IsPaused = true;
//to re-enable
this.colorFrameReader.IsPaused = false;

If we are done with the reader for good, it is best practice to dispose of it, as shown in Listing 2-3.

Listing 2-3. Disposing of a Reader

if (this.colorFrameReader != null)
{
    // ColorFrameReader is IDisposable
    this.colorFrameReader.Dispose();
    this.colorFrameReader = null;
}

Unlike in the first Kinect SDK, we can now have multiple readers per source. This permits us to modularize our code. An example of this could be for a first-person sports game in which we could have the majority of the screen dedicated to an activity—say, a user trying to hit a target with a bow and arrow—and on the corner of the screen we could have an optional 3D avatar visualization of the person that could be turned off and on. Both the main game and the 3D avatar visualization use the BodyFrameReader, but since we want to be able to toggle the 3D avatar visualization and use it for other sports activities like dancing, we put it in its own code module and give it its own reader that can be turned off individually.

Frames

The frame contains all our precious, precious data, in addition to metadata such as dimensions or formatting. Frames are obtained from FrameReferences, which are in turn obtained from readers. FrameReferences exist mainly to help you keep your frames in sync. Other than the frame itself, FrameReferences contain a TimeSpan (.NET structure that describes an interval of time) called RelativeTime, which informs us when the frame was created. The start time of RelativeTime is unknown to us, so we compare it with the RelativeTimes of other types of frames to see if they were produced at around the same time.

To access the frames, we add an event handler to our reader, as in Listing 2-4, and whenever our reader raises an event, we will be provided with the most recently produced frame.

Listing 2-4. Adding an Event Handler to a Reader

this.colorFrameReader.FrameArrived += this.Reader_ColorFrameArrived;

To access the frame, we should make use of the Dispose pattern. Kinect data is very memory intensive, and it can easily eat up all our system resources if we are not careful. We do this by relying on the using keyword, which will automatically dispose of unneeded resources when we are finished. Accessing frames with the use of this pattern is demonstrated in Listing 2-5.

Listing 2-5. Accessing a Frame from an Event Handler

private void Reader_ColorFrameArrived(object sender, ColorFrameArrivedEventArgs e)
{
    using (ColorFrame colorFrame = e.FrameReference.AcquireFrame())
    {
        if (colorFrame != null)
        {
            ...
            //do work with Frame
            ...
        }
    }
}

Once the event handler is called, we can obtain the FrameReference from the event arguments e and then call its AcquireFrame() method to extract the frame. Whenever we receive a frame, we should endeavor to make a local copy or to directly access its data buffer to avoid holding up the system.

Feeding a Recording into a Kinect Application

Start by unplugging your Kinect from your computer. We do not want to stream the Kinect to our application instead of the recording. On the Play page, ensure that you are still connected to the Kinect service. It should say Connected next to the Play button. Then, make sure all the Data Sources you are interested in are enabled on the bottom half of the Play page. Do not play the video yet.

Since we have not created a full-fledged Kinect application yet, we are going to use one of the sample applications. Search for SDK Browser v2.0 on your computer and run it. Find the Color Basics-WPF sample and install it to a location of your choice. Open the solution file (.sln) in Visual Studio and then compile the project (green arrow or F5). The video window should be black and should say “Kinect not available!” on the bottom. Now, return to Kinect Studio and click the Play button. It should still say “Kinect not available!” on the bottom, but the video feed should work as if you had a real Kinect connected to it. If it is not working, make sure you recorded your Kinect Studio footage with the color Data Source enabled and that it is also enabled on the playback page.