Foreword by Mansib Rahman

It may not be a surprise to many that the current state of affairs for the Kinect is in flux. While the core APIs and drivers are relatively stable, Microsoft has somewhat dropped the ball on further development. That is not necessarily a bad thing. They have taken their learnings and applied them to the development of newer technologies, such as HoloLens and other mixed-reality headsets. The Kinect for Windows v2 still offers a strong entry point for hobbyists, developers on a budget, and research/commercial developers looking for a proven depth-sensing technology.

So, when Microsoft (somewhat quietly) announced Kinect support for Windows 10, you can imagine I was pretty ecstatic. I had no expectations, so anything at all was welcomed by me. I really wanted to ensure that this book included what was essentially the start of the last chapter of Kinect for Windows v2, even at the risk of delaying publication by several months. Unfortunately, I simply did not have enough experience with the new programming paradigms for Windows 10 UWP and Kinect. I happened to know someone who did. Dwight, the author of this appendix, is a Kinect MVP and had the fortune of learning about Kinect development for Windows 10 before it was announced by Microsoft publicly. Thus, I did not hesitate to enlist his expertise and have him draft the following pages. Aside from some grammar and syntactic edits, I let him have purview of this entire appendix.

A lot of the theory and backstory in this appendix may be confusing for those not experienced with the Microsoft development stack. The story from Microsoft is not clear in itself, and Dwight was compelled to work with what he had at his disposal. If you find it difficult to grasp, do not hesitate to jump right to the code.

Additionally, I should warn you that the content in this appendix reflects the current state of affairs, which is subject to change by Microsoft at any time, depending on their overarching Windows 10 and mixed-reality strategies. This is the uncool part about being a developer (though admittedly, it does help justify our paychecks). As a result, some of the content here may not reflect what is current at the time of publishing.

Introduction

There are two different approaches to building applications for Windows 10. The first approach is to utilize a software development kit targeting Windows 10 traditional Win32 desktop-based application s running on x86 and x64-bit platforms, which have been around for the past 30 years. The second approach is to utilize an SDK targeting Windows 10 Universal Windows Platform (UWP). The second approach has only been around more recently.

To target the Windows 10 Win32 desktop, everything you’ve learned up to this point is how you would go about it. You would utilize the Kinect for Windows v2 SDK in both .NET and C++ libraries; there is nothing special you need to do. Just follow the steps mentioned in the previous chapters, and your application will work. This brings up a very important point: applications that you built using the Kinect for Windows v2 on Windows 8 will work unchanged on Windows 10 desktops.

On the other hand, to target the Windows 10 UWP and its store, there is a new paradigm you must learn. The architecture has changed, the design philosophy has changed, and the programming model is different. You can pretty much say it’s a new SDK and framework for building Windows 10 UWP applications altogether, which follow the Windows 10 UWP development mantra.

This section will take you through this new paradigm, the history of the Kinect v2 for UWP, architecture, and programming model, and show step by step the building blocks for how to build a Windows 10 UWP store application with Kinect.

One Billion Devices

One of the main goals of Windows 10 UWP is to make sure it runs on as many Windows devices as possible. The goal is to reach one billion-plus devices over the next year. This includes device form factors such as IoT , hobby or prototyping programmable boards such as Raspberry Pi, Particle, and Arduino, mobile phones, tablets, desktops, laptops, and wearables. Even the more specific form factors such as HoloLens (head-mounted displays), Surface Hub (wall-mounted, large, shared-screen experience), and Xbox gaming and entertainment consoles are a part of this grouping. This means that, as a developer, when you create your UWP application it has the potential to run on over one billion devices, which means the potential to reach over one billion customers.

Reaching this many customers and running on this many types of devices requires a change in thinking and a change in development strategies. This is especially the case when it comes to display and form factors, as well as how an application runs and how an application closes down. The latter is typically known as lifecycle management. While the form factor is addressed through a fast and fluid design philosophy, there are many books and articles on how this is accomplished within the Windows 10 SDK. The lifecycle management, on the other hand, needs some more attention, as it relates to Kinect for Windows v2 development.

Lifecycle Management

The lifecycle management of a UWP application compared to legacy Win32 desktop applications is a lot like your father’s root beer, where anything may go into the drink and anything can happen if he stays on it too long. Traditional Win32 applications are typically controlled by the user, and anything can go in it. The application starts when the user wants, and it can stay running as long as the user wants; anything is likely to happen. The user terminates and closes down the application when the user wants by pressing the x-close button.

UWP applications, however, do not necessarily behave this way. The lifecycle of a UWP application is modeled as a state machine. A state machine, in this example, is a UWP application where states are maintained and can transition to different states based on events and data flowing between the states. A user can initiate the state machine, but there are many things outside the user’s control and within the UWP engine that can change the state of the application. A UWP application contains three main states: Running, Suspended, and Not Running. An application can go from Running to Suspended to Not Running. An application can also go from Suspended back to Running again without going to Not Running. Figure A1-1 shows UWP states an application can be in. As a Kinect UWP developer, these states must be accounted for in your application. The following is a quick diagram showing the possible states and events you have access to for determining which state you came from or are about to go into.

Figure A1-1. Diagram of UWP states

System Environment Changes

Traditionally, legacy-based applications have free rein as to what resources are available for use. When running legacy applications with administrative permissions, the application can access and modify registry settings, connected devices, and any file on the system, including system files, and can basically communicate with any other running applications. These legacy applications took the concepts of standard C—trust the programmer and don’t prevent the programmer from doing what needs to be done—and ran with it. Legacy applications became kings of the computer. Well, in the hands of untrustworthy developers, these king-like legacy applications led to viruses, and eventually led to a concept called secure computing.

Fast forward to 30 years later, when trustworthy developers and secure applications are at the top of everyone’s list. As a developer now, we must be security conscious. Along with this responsibility comes the idea of secure applications running on one billion devices, and the philosophy is now the anti-C theorem: do not “trust the programmer”; you must prevent the programmer from doing whatever he or she wants. This, for the most part, is what Windows 10 and the UWP is all about—providing a safe environment and giving control back to the user and safe engine and taking it out of the hands of the untrustworthy programmer. In UWP applications, a user must decide if they trust the programmer. By default, nearly all capabilities within the application are restricted from running. The user has the ability to allow various capabilities to run within an app. As a developer, if you don’t request the permissions for the capabilities you need for your application from the user, it will run with the bare minimum of features, basically just displaying some hard-coded text and embedded pictures. Keep in mind that the user also has the ability to say no, which means as a developer your application must account for this.

A UWP application provides a manifest file that contains declared capabilities that a developer requests from a user in order to run on a target system. These capabilities can be declared in the package manifest file or programmatically. The capabilities are grouped into general capabilities, device capabilities, and special capabilities.

General capabilities apply to the most common UWP application scenarios and contain features such as access to the Music, Pictures, Videos, and Removable Storage folders and libraries. If your application requires network connectivity, there’s the Internet and public networks capabilities, as well as home and work networks. There is also support for appointments, contacts, code generation, AllJoyn, phone calls, VOIP calling, 3D objects, chat messaging, and a few more.

If your application requires a scenario for accessing peripheral and internal devices, you must include device capabilities. The Kinect, for example, is a peripheral device that has many sensors and capabilities.

If I may, I’d like to go on a tangent for a second. In the new Windows 10 and beyond world, we must think about the Kinect for Windows v2 as a set of capabilities instead of as just a device. Just for a second, you could re-imagine the development strategy of UWP and Kinect. Microsoft has always been more focused on the software side rather than the hardware side. For this one- billion-device plan to work, they have to build software and frameworks for device-agnostic capabilities that have depth, perception, color, sound, and location. Well, that’s exactly what the UWP SDK provides—a generic pattern where we as developers can now build device-agnostic applications. We can build a suite of applications that take advantage of capabilities rather than tie ourselves to specific devices. These capabilities can be embedded in apparel (wearables), glasses (HMD), TVs, phones, laptops, desktops, household appliances, cars, and other creative mediums. Now, hopefully, you can imagine how emerging technologies can lead to emerging experiences that explode like tiny DNA explosions, changing your molecular structure into a new being.

Sorry for the aside; let’s get back to the point. A Kinect UWP application needs to explicitly request one or more of these capabilities. For example, if you want to access any generic microphone connected either internally or externally (such as the Kinect for Windows v2 microphone array), you must add the Microphone capability to the package manifest. There are device manifests for Location-aware scenarios (GPS sensors), Proximity (Bluetooth/Wi-Fi or Internet), Webcam (i.e., Kinect’s web camera), USB, Human Interface Device (HID), Bluetooth GATT, Bluetooth RFCOMM, and Point of Sale (POS) features. Even though these capabilities seem specific and special enough, there’s actually one last unique category left: the special capabilities.

The special capabilities are meant to be for specific scenarios that include Enterprise authentication, Share User certificates, and a special use case for the documents library when accessing files offline or in OneDrive, or when your UWP can’t use a file-picker user interface control for selecting files in the folder of your choosing.

Now that you have a general understanding of capabilities, it’s time to talk about how packaging and installation works in a UWP application.

Packaging and Store Submission

Packaging is where a developer takes all the files necessary to run the application-included configuration entries and combines them into one file or folder for the purpose of transferring the package to the user. Back in the day, this was done with a packaging utility or installer software, such as InstallShield or WISE. In more modern applications such as .NET, we have the Wix Toolset and set up projects included inside of Visual Studio .NET. These utilities package the application and make it easy to transfer the application to a user system. One of the issues with packaging has always been deciding where you should store the package for high visibility, and also how you know what version of the software is inside the package. Along with answering these questions, it would be great to also have a rating and/or ranking system and comments for other users. Taking a lesson from other vendors, Microsoft realized a repository of trusted packages and versioning was needed, and while they were at it, they added a way to allow developers to monetize the application, along with a way for users to show their trust in the developer, or their distrust. This repository is called the Windows App Store. For UWP applications, you can easily create an App Store–ready package, and Microsoft provides a PowerShell script to make it easy to install the package on a user system for development and testing purposes if you don’t care to submit your application to the store. This is known as sideloading your application. However, if you do decide to submit your application, it will go through a certification process. This certification process will run automatic tests to check a system shutdown scenario, residual entries upon installation and uninstallation steps, OS Version check, compatibility issues, crashes, user-access control, writing to correct folders, multi-user sessions, and other tests. Once the tests have been passed, a few poor souls at Redmond then have the task of manually reviewing the application and testing it to make sure there’s no offensive content and that it is in line with Microsoft store policies. If the manual test is approved, the application is published to the Microsoft App Store .

Now, with the Windows 10 paradigm out of the way, let’s take a trip down memory lane for a second to understand how the Kinect’s new UWP APIs made it into being.

Microsoft Media Foundation (MF)

Microsoft Media Foundation is a technology that has been around since Windows Vista, and its core purpose is to be a low-level set of APIs for media development. It is the replacement for Microsoft DirectShow, Microsoft Direct X Media Objects (DMO), and Microsoft Video for Windows frameworks. MF provides a developer with the ability to create videos and audio, enhance them, and create audio and video editing–type applications and other advanced media-based filters. It is the core functionality behind Windows 8/8.1 Media Capture XAML elements and is used to provide Internet Explorer and Microsoft Edge with their video rendering capabilities. Another way of stating it is as follows: MF is the hard-core media pundit. All things go through MF, ’cause it’s one bad MF. In previous releases of MF, there was no ability to treat videos as frames or pixel buffers except through the MF Pipeline, also known as the topology. The MF topology is a graph, or a series of tasks handled by individual COM-based components that take some input stream, process the data, and return 0 or more output streams.

There are several different types of components, such as sources, sinks, and transforms. Sources provide the source of the data-input stream. Sinks provide a destination (think file on a drive, or endpoint to a network stream). Transforms are middleware that aggregate, divide, remix, modify, encode, decode, compress, decompress, convert, and anything else you can think of that happens in the middle of processing media files.

Microsoft created a Media Foundation Source that adheres to the Media Foundation Pipeline specifically for the Kinect v2. This source captures all the video and audio streams that the Kinect device offers. There is a video source input for Color, Depth, and IR. This can be seen in the Figure A1-6 using the MF topology editor (topoedit.exe). I won’t go into detail about the MF topology, as that is beyond the scope of this book. However, I will say that the MF topology provides a low-level architecture to expose the Kinect device as a set of media sources for video and audio to the Windows OS. This mechanism is what allows the Kinect device to be used with Skype, the Windows 10 Camera, and other applications. It is also what allows the Kinect to be exposed to the Windows 10 UWP MediaCapture API and XAML controls.

Figure A1-6. Kinect MF Custom Source in TopoEdit

The Kinect Media Foundation library can be found at C:WindowsSystem32KinectKinectMFMediaSource.dll

In previous versions of MF, it was very difficult to get to the underlying audio and video frames and buffers. With the Windows 10 Anniversary update, Microsoft provided a set of new APIs in the Windows.Media.Capture and Windows.Media.Capture.Frames namespaces that makes it very easy to access video frames and buffers for doing computer vision.

Device Perception Provider

Windows 10 also provides a set of APIs in the Windows.Devices.Perception and Windows.Devices.Perception.Providers namespaces that allows hardware vendors to expose depth and infrared streams of data from their devices into a generic class for perception: the perception frame. This is made possible by the Device Perception Provider. If a device wants to provide access to its data streams to Windows 10 as perception frames, it must conform to the DevicePerception Provider ABIs. In previous versions of this provider, color, depth, and infrared streams were made available; however, this provider was purposed only for use with Windows Hello. Windows Hello only utilizes the infrared camera for its functionality, and thus the Analog team removed the color and depth streams from this provider.

By default, you can find the device providers at ComputerHKEY_LOCAL_MACHINESOFTWAREMicrosoftAnalogProviders.

In Figure A1-7, you can see some of the default providers available on a Windows 10 Anniversary Update.

Figure A1-7. Registry hive showing default perception providers

Specifically, the Kinect perception provider can be found at %SystemDrive%windowssystem32Kinect.PerceptionFrameProvider.dll.

As an Independent Hardware Vendor (IHV), Original Equipment Manufacturer (OEM), or hobbyist maker, you can implement your own PerceptionFrameProvider. To do so, you have to create a driver for your device that can generate data to send to the SensorDataService through the IPerceptionFrameProvider and IPerceptionFrameProviderManager interfaces. Implementing these interfaces and the exact steps required is beyond the scope of this book .

Updated Kinect Driver

The Kinect for Windows v2 hardware driver was updated to support sending data to the sensor provider so that the provider could create perception frames to send to the SensorDataService. The Kinect Device Perception Provider is what allows the Kinect to also support Windows Hello. This driver was also updated to support Media Foundation for sending color, depth, IR, and audio streams into the MF Pipeline. With all these new updates, one would think that the old functionality would take a hit and not work anymore, but that is not the case. Microsoft did not modify the old behavior of the driver. This means it still works the same way for previous applications. It is backward-compatible with the pre–Windows 10 UWP SDK and framework. This means that applications built with the .NET framework and C++ libraries will still continue to function as normal. Also, as an added bonus, the new UWP framework does not hinder the old framework and actually can work side by side simultaneously. Figure A1-8 shows an application running the DirectX Depth Sample found in the previous SDK and the UWP Camera Frames sample simultaneously.

Figure A1-8. .NET and UWP Kinect applications running simultaneously

If you didn’t catch this the first time I mentioned it, this means you can run your old .NET or C++ Kinect-based applications at the same time as you run newer UWP store Kinect-based applications. Along with running the applications side by side, as you start to develop with UWP Kinect-based applications, you’ll notice a similar programming pattern when it comes to asynchronous behavior and listening for events and frames. Let’s discuss that next.

Similarities in the Architecture

I know that it was stated earlier that developing a Kinect application for UWP is a completely new paradigm, and this leaves the impression that there will be a steep learning curve. However, to our advantage, the learning curve isn’t very steep at all. This is because if you’ve ever done any Kinect programming in the past, even by reading through this book up to this point, you’ll be comforted to know that there are some very similar development patterns with UWP. For example, Windows 10 UWP applications are based on an asynchronous design pattern, so if you’ve worked with the await keyword in C#, nothing has changed. Next, there is the concept of frames. If you remember, in previous chapters we had ColorFrames, DepthFrames, and InfraredFrames. In UWP, we have Media Capture frames and perception frames. Similarly, if you remember, each type of frame was able to generate various events, such as the FrameArrived event. In UWP, we have the same pattern. Perception and Media Capture frames also generate an arrived event, and as developers we can subscribe to these events to listen for data. Another similarity we have is the concept of disposing of a frame as fast as we possibly can once we’re done with it. In UWP Kinect-based applications, we follow the same principle. As you can see, the learning curve isn’t as steep as it may appear when it comes to media frames. With the architecture out of the way, let’s start creating our UWP Kinect-based Store app.

How to Get the Latest Drivers

The next and easiest, but probably most time-consuming, step is to upgrade your Windows 10 to the Anniversary edition: version 10.0.14394.xxx or above. If you’re not sure what version of Windows 10 you’re running, click the Start button or press the Windows key and type version. In Figure A1-9, you’ll see the System Information application reveal what version of Windows 10 you are running. If you’re on version 10.0.14394 or above, you’re good to go. If not, go ahead and head over to your system settings, navigate to Windows Update, and update to the Anniversary edition .

Figure A1-9. Windows 10 System Information and version details

If you don’t have the Anniversary edition, you can still get the latest driver, which is the preview “test” version but is the latest none the less. To get this version, you will also have to be a Windows 10 Insider and opt in for test drivers. Search how to opt in for test drivers on Windows 10. It’s a registry setting. Turn on Windows Hello, and you should be ready to go. I won’t go into the specifics simply because I recommend you go the safer route of updating to the Anniversary edition.

Once you are on the Anniversary edition, plug your device in and open up Device Manager by clicking on the Start button and typing device manager. Locate the Kinect device, expand it, right-click on WDF KinectSensor Interface 0, and select Update driver. Let Windows choose the best and most updated driver. Figure A1-10 shows the menu item to select.

Figure A1-10. Selecting Update driver in Device Manager

If everything goes well without an error, then your driver should be updated to the latest version. Figure A1-11 shows the properties of the latest Kinect UWP driver from Device Manager.

Figure A1-11. Kinect’s latest driver details

To verify your Kinect driver is working with UWP, run the Windows 10 Camera application and cycle through your cameras, verifying that your Kinect turns on and that the Kinect v2 is listed as the camera. Figure A1-12 shows the Kinect device as a camera available for selection in the UWP Camera application. No camera feed might show.

Figure A1-12. Kinect Device selected in the Camera application

Building a UWP Store application

We will start up by building a very simple UWP Kinect-based application that can show the color web camera stream and allow you to take a picture of the color stream. The application will make use of the Media Frame Source Group, Media Frame Reference, Media Frame, Software Bitmap, XAML Image Element, and XAML Writeable Bitmap.

Let’s start from scratch building a Visual Studio 2015 C# UWP project. Name the project WorkingWithMediaCaptureFrames.

Add the namespaces seen in Listing A1-1 to the MainPage.xaml.cs file.

Listing A1-1. Media Capture Namespaces

using Windows.Media.Capture;
using Windows.Media.Capture.Frames;

These are the core namespaces required for the Media Capture and Media Capture Frames classes.

Add the remaining namespaces to include asynchronous behavior, software bitmaps, and file storage functionality, as shown in Listing A1-2.

Listing A1-2. Namespaces for Async, Imaging, and File Storage

using System.Threading.Tasks;
using Windows.Graphics.Imaging;
using Windows.UI.Xaml.Media.Imaging;
using Windows.Storage;
using Windows.Storage.Pickers;
using Windows.Storage.Streams;                                                                       

Add the private class member variable seen in Listing A1-3.

Listing A1-3. MediaCapture Variable Declaration

// MF Capture device to start up
        MediaCapture _mediaCapture;

This variable represents the Media Foundation Media Capture device. Basically, this represents the Kinect Device as a Media Foundation UWP object. Now, add the remaining private member class variables that will be used to help us display and save the color image, as shown in Listing A1-4.

Listing A1-4. Imaging Variable Declarations

private enum FileFormat
{
    Jpeg,
    Png,
    Bmp,
    Tiff,
    Gif
}

// Constants that let us decide on which frame source kind we want
const int COLOR_SOURCE = 0;
const int IR_SOURCE = 1;
const int DEPTH_SOURCE = 2;
const int LONGIR_SOURCE = 3;
const int BODY_SOURCE = 4;
const int BODY_INDEX_SOURCE = 5;

// UWP BitmapSource
private SoftwareBitmapSource source = new SoftwareBitmapSource();

// XAML Bitmap for updating bitmaps in XAML
private WriteableBitmap extBitmap;

The preceding variables are used to help us determine which file format we’d like to use to save the color image, which Media Capture frame source types we are going to display, the UWP bitmap source, and the XAML bitmap source, respectively. The remaining steps deal more with how to access various data streams and work with the individual streams (i.e. color, depth, and infrared).

How to Access Data Streams

To get access to the various data streams from a Media Capture device, we must request permission to access the device. We can do this declaratively using the package.appxmanifest file. Double-click on the package manifest and click on the Capabilities tab. Select Webcam and Pictures Library. Figure A1-13 shows the package.appmanifest file opened inside Visual Studio 2015.

Figure A1-13. Declaring capabilities with package manifest

Selecting these options will have the application asking or requesting permission to read and write to the Pictures Library, as well as asking for permission to show and use the device camera (the Kinect, in our case).

Once we have requested the proper capabilities, we focus our attention on how to access them. UWP treats a Media Capture device as a set of features. This set or collection is known as a MediaFrameSourceGroup. For each MF device, if it can send at least one type of data stream, then it is represented as one of this group. This is where things start to differ from traditional Kinect development. Normally, with the Kinect device you’d call GetDefault(), which will provide you with the default device that represents the Kinect.

In the UWP paradigm, you query for a group, the MediaFrameSourceGroup. The concept is that one or more devices will have a group of sensors. Plus, as a developer you may want to access these sensors synchronously. Basically, each device (a.k.a. MediaFrameSourceGroup) will return with data that contains one or more sources of information regarding each data stream that may be exposed by that device (a.k.a. MediaFrameSourceGroup). This group is more akin to a single device with a collection of capabilities. The MediaFrameSourceGroup is an abstraction over that aforementioned idea. A MediaFrameSourceGroup provides a collection of sources. These are the Data Sources. They are represented in the UWP platform as a class called SourceInfos. The SourceInfos is a collection that contains a collection of data streams. A data stream is of a particular type; for example, it could be a color stream, depth stream, infrared stream, or custom stream. We determine which type of stream it is by the property of source info called the SourceKind. SourceInfos also contain information about the source’s DisplayName, Location, and CameraIntrinsics. It’s here that you can filter out which device group you want to listen to for frame events. Let’s add the method shown in Listing A1-5 to the class.

Listing A1-5. Color Frame Initialization

async void InitializeColorFrames()
{
    // Devices come grouped together as a set of capabilities
    // Find all the media frame source devices - as a collection of groups
    var allGroups = await MediaFrameSourceGroup.FindAllAsync();

    // Let's filter using LINQ, based on a device group with a DisplayName of "Kinect"
    var eligibleColorGroups = allGroups
        .Where(g => g.SourceInfos.FirstOrDefault(info => info.SourceGroup.DisplayName.Contains("Kinect") ) != null).ToList();

    // Check to see if we found a device
    if (eligibleColorGroups.Count == 0)
    {
        //No kinect camera found
        return;
    }

    //Only one Kinect camera supported, so always take the first in the list
    var kinectColorGroup = eligibleColorGroups[0];
    try
    {
        // Initialize MediaCapture with selected group.
        // This can raise an exception if the source no longer exists,
        // or if the source could not be initialized.
        await InitializeMediaCaptureAsync(kinectColorGroup);
    }
    catch (Exception exception)
    {
        //Release any resources if something goes wrong
        await CleanupMediaCaptureAsync();
        return;
    }

    // Let's get a Device Capability; in this case, let's get the color source stream
    MediaFrameSourceInfo colorInfo = kinectColorGroup.SourceInfos[COLOR_SOURCE];

    if (colorInfo != null)
    {
        // Access the initialized frame source by looking up the the ID of the source found above.
        // Verify that the ID is present, because it may have left the group
        // while we were busy deciding which group to use.
        MediaFrameSource frameSource = null;
        if (_mediaCapture.FrameSources.TryGetValue(colorInfo.Id, out frameSource))
        {
            // Create a frameReader based on the color source stream                                                
            MediaFrameReader frameReader = await _mediaCapture.CreateFrameReaderAsync(frameSource);

            // Listen for Frame Arrived events
            frameReader.FrameArrived += FrameReader_FrameArrived;

            // Start the color source frame processing
            MediaFrameReaderStartStatus status = await frameReader.StartAsync();

            // Status checking for logging purposes if needed
            if (status != MediaFrameReaderStartStatus.Success)
            {

            }
        }
        else
        {
            // Couldn't get the color frame source from the MF Device

        }
    }
    else
    {
        // There's no color source

    }                                                                              
}

The preceding method uses the static MediaFrameSourceGroup.FindAllAsync() method to get back a collection of device “groups” that have a synchronous collection of data streams. A LINQ statement follows, which filters the collection of groups based on the SourceInfo’s DisplayName. If the groups’ DisplayName contains the word Kinect, then the group represents the Kinect’s collection of synchronized capabilities.

private async Task InitializeMediaCaptureAsync(MediaFrameSourceGroup sourceGroup)
{
    if (_mediaCapture != null)
    {
        return;
    }

    // Initialize mediacapture with the source group.
    _mediaCapture = new MediaCapture();
    var settings = new MediaCaptureInitializationSettings
    {
        SourceGroup = sourceGroup,

        // This media capture can share streaming with other apps.
        SharingMode = MediaCaptureSharingMode.SharedReadOnly,

        // Only stream video and don't initialize audio capture devices.
        StreamingCaptureMode = StreamingCaptureMode.Video,

        // Set to CPU to ensure frames always contain CPU SoftwareBitmap images
        // instead of preferring GPU D3DSurface images.
        MemoryPreference = MediaCaptureMemoryPreference.Cpu
    };

    await _mediaCapture.InitializeAsync(settings);
}

/// <summary>
/// Unregisters FrameArrived event handlers, stops and disposes frame readers
/// and disposes the MediaCapture object.
/// </summary>
private async Task CleanupMediaCaptureAsync()
{
    if (_mediaCapture != null)
    {
        using (var mediaCapture = _mediaCapture)
        {
            _mediaCapture = null;

        }
    }
}

How to Work with Color

Up to this point, we’ve discussed the basic setup for accessing the Kinect device through Media Foundation. We have the mediaCapture object, and we have the Kinect source data streams through two different objects: mediaSourceGroup (a synchronized grouping of all the source streams, similar to the MultiSourceFrameReader in the previous SDK) and SourceInfo object, which represents the individual source stream and description. Let’s revisit a section of code in the InitializeColorFrames() method. There’s a section of code we haven’t discussed yet, and this section of code is how we work with the core source data streams . See Listing A1-8.

Listing A1-8. Handling the Color Data Source

//...
// Let's get a Device Capability, in this case let's get the color source stream
MediaFrameSourceInfo colorInfo = kinectColorGroup.SourceInfos[COLOR_SOURCE];

if (colorInfo != null)
{
    // Access the initialized frame source by looking up the ID of the source found above.
    // Verify that the ID is present, because it may have left the group
    // while we were busy deciding which group to use.
    MediaFrameSource frameSource = null;
    if (_mediaCapture.FrameSources.TryGetValue(colorInfo.Id, out frameSource))
    {
        // Create a frameReader based on the color source stream
        MediaFrameReader frameReader = await _mediaCapture.CreateFrameReaderAsync(frameSource);

        // Listen for Frame Arrived events
        frameReader.FrameArrived += FrameReader_FrameArrived;

        // Start the color source frame processing
        MediaFrameReaderStartStatus status = await frameReader.StartAsync();

        // ...                                                                                                    

The preceding section of code queried all the source groups and filtered down the specific device source group we’re interested in. If you remember from earlier, it was the source group that had a display name of “Kinect” contained inside its string’s full name. Once we’ve selected the source group, we can get to the capability list, better known as the SourceInfo collection. In the preceding code I use an indexed array, as follows:

//...
// Let's get a Device Capability, in this case let's get the color source stream
        MediaFrameSourceInfo colorInfo = kinectColorGroup.SourceInfos[COLOR_SOURCE];

Where COLOR_SOURCE is a constant representing the positional index that is retrieved from the MF Kinect’s color stream representation. However, if you’re working with other source groups for other devices, this may not be the best way to go, because each device has the option of changing how it presents the collection of source streams and in what order. Again, remember the SourceInfo collection allows you to determine which source stream you want using SourceKind, MediaStreamType, Device Information, and many other properties. I chose the simplest way for this demo.

The next lines of code should be somewhat familiar:

MediaFrameSource frameSource = null;
if (_mediaCapture.FrameSources.TryGetValue(colorInfo.Id, out frameSource))
{
    // Create a frameReader based on the color source stream
    MediaFrameReader frameReader = await _mediaCapture.CreateFrameReaderAsync(frameSource);

    // Listen for Frame Arrived events
    frameReader.FrameArrived += FrameReader_FrameArrived;

    // Start the color source frame processing
    MediaFrameReaderStartStatus status = await frameReader.StartAsync();                                                                                                    

These lines use the mediaCapture.FrameSources.TryGetValue() method to get a generic MediaFrameSource. This MediaFrameSource represents a generic version of the ColorFrameSource, DepthFrameSource, and InfraredFrameSource found in previous versions. Once we have the frame source, we use the mediaCapture object to create a MediaFrameReader. This frame reader is used to listen for FrameArrived events. The MediaFrameReader also supports the polling design. There is a TryAcquireLatestFrame() method that you can use to get the latest frame in a polling pattern if you need to use that design. The MediaFrameReader also contains the methods to StartAsync() and StopAsync() the frames from being processed on the device.

Let’s now add the FrameReader_FrameArrived method to our class, as shown in Listing A1-9.

Listing A1-9. FrameArrived Event Handler

private void FrameReader_FrameArrived(MediaFrameReader sender, MediaFrameArrivedEventArgs args)
{
    // Try to get the FrameReference
    using (var frameRef = sender.TryAcquireLatestFrame())
    {
        if(frameRef != null)
        {
            if (frameRef.SourceKind == MediaFrameSourceKind.Color)
            {
                // Process the frame drawing the color pixels onto the screen
                ProcessColorFrame(frameRef);
            }
        }
    }
}

This method provides us with a signature that takes in a MediaFrameReader, as well as MediaFrameArrivedEventArgs parameters. Normally in .NET programming the EventArgs parameter provides context information about the event being generated; however, in this framework EventArgs are not used at all, so you can safely ignore that parameter. In this method, we use the MediaFrameReader parameter to try to get the latest frame. This method returns a MediaFrameReference, similar to in the previous SDK. Once we have the media frame reference, we can access the specific frame we’re interested in. Some of you may have noticed that it’s generic. In the previous framework, we had a specific frame reference—one for Color, one for Body, one for Depth, and one for Infrared. In this new UWP framework, there’s only one. This means we have to check for which type of frame reference it is by using the SourceKind enumeration property.

Now, let’s add the ProcessColorFrame method, as shown in Listing A1-10.

Listing A1-10. Processing Color Data

private void ProcessColorFrame(MediaFrameReference  clrFrame)
{
    try
    {
        // Get the Individual color Frame
        var vidFrame = clrFrame?.VideoMediaFrame;
        {
            if (vidFrame == null) return;

            // create a UWP SoftwareBitmap and copy Color Frame into Bitmap
            SoftwareBitmap sbt = new SoftwareBitmap(vidFrame.SoftwareBitmap.BitmapPixelFormat, vidFrame.SoftwareBitmap.PixelWidth, vidFrame.SoftwareBitmap.PixelHeight);
                    vidFrame.SoftwareBitmap.CopyTo(sbt);

            // PixelFormat needs to be in 8bit BGRA for Xaml writeable bitmap
            if (sbt.BitmapPixelFormat != BitmapPixelFormat.Bgra8)
            sbt = SoftwareBitmap.Convert(vidFrame.SoftwareBitmap, BitmapPixelFormat.Bgra8);

            if (source != null)
            {
                // To write out to writeable bitmap which will be used with ImageElement, it needs to run
                // on UI Thread thus we use Dispatcher.RunAsync()...
                var ignore = Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, () =>
                    {
                        // This code runs on UI Thread
                        // Create the writeableBitmap for ImageElement display                                                                  
                        extBitmap = new WriteableBitmap(sbt.PixelWidth, sbt.PixelHeight);

                        // Copy contents from UWP software Bitmap
                        // There are other ways of doing this instead of the double copy, 1st copy earlier
                        // this is a second copy.
                        sbt.CopyToBuffer(extBitmap.PixelBuffer);
                        extBitmap.Invalidate();

                        // Set the imageElement source
                        var ig = source.SetBitmapAsync(sbt);
                        imgView.Source = source;

                    });
            }
        }
    }
    catch (Exception ex)
    {

    }
}

The process color frame method is used to retrieve either the VideoMediaFrame or a BufferMediaFrame. The BufferMediaFrame is used to retrieve body skeletal data through an IBuffer interface. The VideoMediaFrame is used for all other data. We use the VideoMediaFrame to determine if we have data in the Direct3DSurface or SoftwareBitmap, get access to the depth- and infrared-specific properties, get access to camera intrinsics, and determine video format. The video media frame is the key to processing and displaying frames.

Once we have the video media frame, we use the frame’s SoftwareBitmap to copy the contents of the bitmap to another SoftwareBitmap, which we can use inside a writeable bitmap for saving out to a file and also for re-copying to a SoftwareBitmapSource class for use with the XAML Image element.

As an aside, because we are updating an XAML user interface control (the Image element), we are confined to only updating this control on the user interface thread. The way to do this is to use the Dispatcher.RunAsync() method call. This is the reason for all the SoftwareBitmap copy code living inside the RunAsync call.

Finally, let’s add the final helper methods and user interface so we have something to show. Add the helper methods shown in Listing A1-11 to your class.

Listing A1-11. Helper Methods for Screenshotting

private async Task<StorageFile> WriteableBitmapToStorageFile(WriteableBitmap WB, FileFormat fileFormat, string fileName = "")
{
    try
    {
        string FileName = string.Empty;
        if (string.IsNullOrEmpty(fileName))
        {
            FileSavePicker savePicker = new FileSavePicker();
            savePicker.SuggestedStartLocation = PickerLocationId.PicturesLibrary;

            // Drop-down of file types the user can save the file as
            savePicker.FileTypeChoices.Add("jpeg", new List<string>() { ".jpg", ".jpeg" });

            // Default file name if the user does not type one in or select a file to replace
            savePicker.SuggestedFileName = "WorkingWithMediaCapture.jpg";
            fileName = (await savePicker.PickSaveFileAsync()).Name;
        }
                                                                                                            
        FileName = fileName;

        Guid BitmapEncoderGuid = BitmapEncoder.JpegEncoderId;
        switch (fileFormat)
        {
            case FileFormat.Jpeg:
            //  FileName = string.Format("{0}.jpeg", fileName);
            BitmapEncoderGuid = BitmapEncoder.JpegEncoderId;
            break;

            case FileFormat.Png:
            //  FileName = string.Format("{0}.png", fileName);
            BitmapEncoderGuid = BitmapEncoder.PngEncoderId;
            break;

            case FileFormat.Bmp:
            //  FileName = string.Format("{0}.bmp", fileName);
            BitmapEncoderGuid = BitmapEncoder.BmpEncoderId;
            break;

            case FileFormat.Tiff:
            //  FileName = string.Format("{0}.tiff", fileName);
            BitmapEncoderGuid = BitmapEncoder.TiffEncoderId;
            break;

            case FileFormat.Gif:
            //  FileName = string.Format("{0}.gif", fileName);
            BitmapEncoderGuid = BitmapEncoder.GifEncoderId;
            break;
        }

        var file = await Windows.Storage.KnownFolders.PicturesLibrary.CreateFileAsync(FileName, CreationCollisionOption.GenerateUniqueName);
        using (IRandomAccessStream stream = await file.OpenAsync(FileAccessMode.ReadWrite))
        {
            BitmapEncoder encoder = await BitmapEncoder.CreateAsync(BitmapEncoderGuid, stream);
            Stream pixelStream = WB.PixelBuffer.AsStream();
            byte[] pixels = new byte[pixelStream.Length];
            await pixelStream.ReadAsync(pixels, 0, pixels.Length);

            encoder.SetPixelData(BitmapPixelFormat.Bgra8, BitmapAlphaMode.Ignore,
                (uint)WB.PixelWidth,
                (uint)WB.PixelHeight,
                96.0,
                96.0,
                pixels);
            await encoder.FlushAsync();
        }
        return file;                                                                                                     
    }
    catch (Exception ex)
    {
        return null;
        // throw;
    }
}

private void btnExternalCapture_Click(object sender, RoutedEventArgs e)
{
    if (extBitmap != null)
    {
        var ignore = Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, async () =>
        {
            var file =
                await WriteableBitmapToStorageFile(extBitmap, FileFormat.Jpeg, "WorkingWithMediaCaptureFrames.jpg");
            fileLocation.Text = file.Path;
        });
    }
}

The preceding two methods take the writeable bitmap and save the contents out to a file in the Pictures library. The second method is an event handler that listens for the button client event for the External Capture button in the XAML shown in Listing A1-12 .

Listing A1-12. MainPage.xaml Markup

<Grid Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">
    <StackPanel  Margin="5" VerticalAlignment="Center" HorizontalAlignment="Center">
        <TextBlock >External Cameras View:</TextBlock>
        <Image x:Name="imgView" Stretch="UniformToFill" MaxWidth="600" ></Image>
        <Button x:Name="btnExternalCapture" HorizontalAlignment="Right" Click="btnExternalCapture_Click">Save External Camera Shot</Button>
        <TextBlock>FilePath:</TextBlock>
        <TextBox x:Name="fileLocation" TextWrapping="Wrap" />
    </StackPanel>
</Grid>

Compile your application and run it. The final result is shown in Figure A1-14.

Figure A1-14. Using Color stream with Kinect in UWP application

How to Work with Depth

Working with Depth is exactly the same as working with Color frames. Let’s revisit the code to retrieve Color Source Info, which gives us access to the color frames:

//...
// Let's get a Device Capability, in this case let's get the color source stream
MediaFrameSourceInfo colorInfo = kinectColorGroup.SourceInfos[COLOR_SOURCE];

To use Depth, we simply change out which SourceInfo item we want. We can simply change this to DEPTH_SOURCE, where DEPTH_SOURCE is a constant representing the positional index that is retrieved from MF Kinect’s Depth stream representation.

//...
// Let's get a Device Capability, in this case let's get the depth source stream
MediaFrameSourceInfo depthInfo = kinectColorGroup.SourceInfos[DEPTH_SOURCE];

Again, remember that if you’re working with other SourceGroups (i.e. other devices), this may not be the best way to do this, because each device has the option to change how it presents the collection of source streams and in what order. The SourceInfo collection allows you to determine which source stream you want by using the SourceKind, MediaStreamType, Device Information, and many other properties. I chose the simplest way for this demo.

How to Work with IR

Working with Infrared is exactly the same as working with Color frames. Let’s revisit the code to retrieve Color Source Info, which gives us access to the color frames:

//...
// Let's get a Device Capability, in this case let's get the color source stream
MediaFrameSourceInfo colorInfo = kinectColorGroup.SourceInfos[COLOR_SOURCE];

To use either the default IR or the LongIR, we simply change out which SourceInfo item we want. We can simply change this to IR_SOURCE or LONGIR_SOURCE, where IR_SOURCE or LONGIR_SOURCE is a constant representing the positional index that is retrieved from the MF Kinect’s Infrared stream representation.

//...
// Let's get a Device Capability, in this case let's get the IR source stream
MediaFrameSourceInfo irInfo = kinectColorGroup.SourceInfos[IR_SOURCE];
MediaFrameSourceInfo longIRInfo = kinectColorGroup.SourceInfos[LONGIR_SOURCE];

How to Work with Custom Data Streams (Preview)

The Media Capture frames API supports devices’ being able to yield custom data streams. This means devices like the Kinect can send body, body index, and other data through the custom data streams pattern. This also means other devices, like Leap Motion, can send hand-tracking data. There is no limit to the types of data that can be sent through the custom data streams.

As of the time of this writing, the Kinect device is the only device that makes use of this custom data stream, and its current implementation is in preview, which means it’s likely to change by the time this feature is finalized.

// The layout of a pose frame buffer is:
//   1. PoseTrackingFrameHeader: Common structure for all providers.
//      1. CustomDataTypeGuid: Guid corresponding to specific provider (16 bytes).
//      2. EntitiesCount: How many entities are in this frame (4 bytes).
//      3. EntityOffsets: Offsets of entity data from buffer start (4 * EntitiesCount bytes).
//
//   2. Provider-specific, per-frame data.
//
//   3. PoseTrackingEntityData #0: First entity data. Common structure for all providers.
//      1. DataSizeInBytes: Size of entire entity (PoseTrackingEntityData + custom data) in bytes (4 bytes).
//      2. EntityId: Allows correlation between frames (16 bytes).
//      3. PoseSet: Guids for Body (such as Kinect), Handtracking, etc.: defines the meaning of each Pose (16 bytes).
//      4. PosesCount: Count of poses in this entity data (4 bytes).
//      5. IsTracked: Whether or not this entity is being tracked (1 byte).
//      6. Poses: Array of common structure TrackedPose (sizeof(TrackedPose) * PosesCount bytes).
//      7. Customer-specific data for this entity (DataSizeInBytes - sizeof(PoseTrackedEntityData) - sizeof(TrackedPose) * (PosesCount -1) bytes)
//
//   4. Provider-specific data for entity #0 in this frame.
//
//   5. PoseTrackingEntityData #1: Second entity data. Common structure for all providers.
//
//   6. Provider-specific data for entity #1 in this frame.

struct PoseTrackingFrameHeader
{
    // This Guid allows the consumer of the buffer to verify that
    // it has been written by the expected provider
    GUID CustomDataTypeGuid;
    uint32_t EntitiesCount;
    uint32_t EntityOffsets[1]; // actual length is EntitiesCount
};

struct PoseTrackingEntityData                                                      
{
    uint32_t DataSizeInBytes;
    GUID EntityId;
    GUID PoseSet;
    uint32_t PosesCount;
    bool IsTracked;
    TrackedPose Poses[1]; // actual length is PosesCount
};

How to Use Audio

There is nothing special about using Kinect audio within a UWP application. As a matter of fact, we lose some capabilities here. In previous frameworks, we had the ability to do beamforming and to determine where sound was coming from using the Microphone array. We could even figure out which sound was coming from which tracked user. I’m afraid to say these features are not implemented in UWP audio for Kinect. I’m also afraid to say I haven’t heard any rumors about whether there will even be an update to include these features in the near future. However, if you know Microsoft, if we as a community provide enough vocal feedback and devise a real usage scenario for them, they will consider it. After all, that is how we got the exclusive mode option in the Kinect, as well as many other features.

Using audio in a UWP Kinect-based application requires working with the Audio graph class. The Kinect microphone array is treated like any other standard microphone. This means that the standard audio samples that come with the UWP samples will work for Kinect. I won’t go into detail on this since no special settings or functionality are required to get it working.

What About Kinect Fusion, Face, HD Face, Visual Gesture Builder, and Kinect Studio Tools?

I’m sad to say that in this current release there are no features or capabilities to work with Kinect Face, HD Face, Fusion, Visual Gesture Builder, or Kinect Studio tools for unit testing and automating replay mechanisms. These topics suffer the same fate as the more specific Kinect Audio capabilities. I do offer some advice and alternatives though.

Kinect Fusion is all about volumetric rendering and can be accomplished by extrapolating on the basic Data Sources. We have all the core components in the Media Capture APIs to establish a generic framework for doing volumetric rendering.

Kinect Face can be accomplished using the Windows.Media classes. There are classes inside of UWP that provide the ability to locate a face and various face points using any camera, not just Kinect. More advanced face features can be accomplished using OpenCV and other computer-vision tools. HD Face can specifically be accomplished using volumetric rendering on top of Windows.Media classes and OpenCV.

Visual Gesture Builder and Kinect Studio tools, on the other hand, are separate applications that will require more involved effort. Their current versions may be able to work with UWP applications as is, but it will require some thought .

Summary

Microsoft changed the game plan when it comes to developing Kinect-based applications for the UWP Store. They have a new paradigm, framework, and philosophy for dealing with device-agnostic apps. This change does not put Kinect in the tomb, but rather forces the developer to treat the device as a group of capabilities that can be used across Windows 10 devices. Learning the new paradigm doesn’t mean you have to start from scratch, especially since a lot of the patterns and design dealing with frames, events, and asynchronous behavior are practically the same. Where Microsoft will take the Kinect from here on out is left to be seen.