Drawing Joints

We will get started with a simple dots-for-joints demonstration. We will draw purple dots on the color camera feed for the location of tracked joints and gray dots for the location of inferred joints, which are joints that the Kinect cannot see at the moment but assumes are present. Start a new C# WPF project and include the standard assortment of namespaces: System, System.Collections.Generic, System.Linq, System.Windows, System.Windows.Media, System.Windows.Media.Imaging, and Microsoft Kinect. Some extra namespaces that also need to be included this time around are System.Windows.Shapes and System.Windows.Controls so that we can draw joints as circles on a Canvas control.

Listing 5-1. MainWindow.xaml.cs Private Variable Declaration for Displaying Skeletal Joints

public partial class MainWindow : Window
{

    private KinectSensor kinect;
    private MultiSourceFrameReader multiSourceFrameReader;                
    //private BodyFrameReader bodyFrameReader;
    private CoordinateMapper coordinateMapper;                

    FrameDescription colorFrameDescription;
    private WriteableBitmap colorBitmap;

    private bool dataReceived;                
    private Body[] bodies;                
    ...
}

In Listing 5-1, we opted to create a MultiSourceFrameReader instead of a BodyFrameReader. This will allow us to read frames from multiple Data Sources relatively synchronously without having to open multiple readers. You should keep in mind that there is no Data Source called MultiFrameSource. It is mostly an abstraction of convenience, and we could have called the readers for the color and skeletal Data Sources individually instead.

By this point in the book, you might be wondering about how the depth camera and color camera have different images that by default do not align. Quite simply, if you look at the front of your Kinect, the cameras are not in the same spot, not to mention have different resolutions. As a consequence, we have to align the images. We can do this mathematically, but fortunately the Kinect has already done this for us. In Listing 5-1, we declare an instance of the CoordinateMapper utility class to let us align points in both images along with the positions of skeletal joints. It is not a static class and is a property of each individual KinectSensor object.

Culminating the private variables, we have the dataReceived boolean, which will toggle whether we draw our joints based on the availability of new data, and the bodies array, which will hold the skeletal data collected by the Kinect.

Listing 5-2. MainWindow.xaml.cs Constructor and Variable Assignment for Displaying Skeletal Joints

public MainWindow()
{

    kinect = KinectSensor.GetDefault();

    multiSourceFrameReader = kinect.OpenMultiSourceFrameReader(FrameSourceTypes.Color | FrameSourceTypes.Body);                
    multiSourceFrameReader.MultiSourceFrameArrived += MultiSourceFrameReader_MultiSourceFrameArrived;                

    coordinateMapper = kinect.CoordinateMapper;                
    colorFrameDescription = kinect.ColorFrameSource.CreateFrameDescription(ColorImageFormat.Bgra);

    colorBitmap = new WriteableBitmap(colorFrameDescription.Width, colorFrameDescription.Height, 96.0, 96.0, PixelFormats.Bgra32, null);
                                                                                                                  
    kinect.Open();

    DataContext = this;
    InitializeComponent();

}

public ImageSource ImageSource
{
    get
    {
        return colorBitmap;
    }
}

In Listing 5-2, we initialize our reader and the other variables necessary for skeletal tracking. Note that the method signature for creating MultiSourceFrameReaders is different from that for the other Data Sources. Instead of calling KinectSensor.XFrameSource.OpenReader() as with the others, we called the KinectSensor’s OpenMultiSourceFrameReader(FrameSourceTypes enabledFrameSourceTypes) method. As mentioned previously, there is no MultiFrameSource Data Source property in the KinectSensor object to access directly. FrameSourceTypes indicates which Data Sources we want our reader to offer frames from. It is an enum that can be delineated by the overloaded | operator (which acts as a bitmask). Like the other readers, we can assign it event handlers to be triggered when frames arrive. We also assign CoordinateMapper and WriteableBitmap in the constructor.

Listing 5-3. MainWindow.xaml.cs Event Handler for Displaying Skeletal Joints

private void MultiSourceFrameReader_MultiSourceFrameArrived(object sender, MultiSourceFrameArrivedEventArgs e)
{

    bool dataReceived = false;
    MultiSourceFrame multiSourceFrame = e.FrameReference.AcquireFrame();

    using (ColorFrame colorFrame = multiSourceFrame.ColorFrameReference.AcquireFrame())
    {
        if (colorFrame != null)
        {
            using (KinectBuffer colorBuffer = colorFrame.LockRawImageBuffer())
            {
                colorBitmap.Lock();

                if ((colorFrameDescription.Width == colorBitmap.PixelWidth) && (colorFrameDescription.Height == colorBitmap.PixelHeight))
                {
                    colorFrame.CopyConvertedFrameDataToIntPtr(
                        colorBitmap.BackBuffer,
                        (uint)(colorFrameDescription.Width * colorFrameDescription.Height * 4),
                        ColorImageFormat.Bgra);

                    colorBitmap.AddDirtyRect(new Int32Rect(0, 0, colorBitmap.PixelWidth, colorBitmap.PixelHeight));
                }

                colorBitmap.Unlock();
            }
        }
    }

    using (BodyFrame bodyFrame = multiSourceFrame.BodyFrameReference.AcquireFrame())
    {
        if (bodyFrame != null)
        {
            if (bodies == null)
            {
                bodies = new Body[bodyFrame.BodyCount];
            }
            bodyFrame.GetAndRefreshBodyData(bodies);
            dataReceived = true;
        }
    }

    if (dataReceived)
    {
        canvas.Children.Clear();                                                                                                              

        foreach (Body body in bodies.Where(b => b.IsTracked))
        {
            foreach (var joint in body.Joints)
            {
                CameraSpacePoint position = joint.Value.Position;
                if (position.Z < 0)
                {
                    position.Z = 0.1f;
                }

                ColorSpacePoint colorSpacePoint = coordinateMapper.MapCameraPointToColorSpace(position);

                if (joint.Value.TrackingState == TrackingState.Tracked)
                {
                    DrawJoint(new Point(colorSpacePoint.X, colorSpacePoint.Y), new SolidColorBrush(Colors.Purple));
                }
                if (joint.Value.TrackingState == TrackingState.Inferred)
                {
                    DrawJoint(new Point(colorSpacePoint.X, colorSpacePoint.Y), new SolidColorBrush(Colors.LightGray));
                }
            }
        }
    }                                                                                                              
}

In Listing 5-3, we have a large event handler method that encapsulates the code for both displaying color images and grabbing skeletal frames. In a larger application, we would probably split this up by passing the MultiSourceFrameArrivedEventArgs e or the frames to separate methods. For simplicity’s sake, I put them together here. It is noteworthy that there is a bit of Inception going on with the frame references for MultiSourceFrames. With the other Data Sources, the process of grabbing data can be summated with xFrameArrivedEventArgs ➤ xFrameReference ➤ xFrame ➤ Underlying Data Buffer. With multi-source data, the process is MultiSourceFrameEventArgs ➤ MultiSourceFrameReference ➤ MultiSourceFrame ➤ xFrameReference ➤ xFrame ➤ Underlying Data Buffer. The important thing to know is that the MultiSourceFrames contain the references for the other frames, and the rest of the paradigm works as you would expect.

There is nothing interesting about our code to acquire and present color data; it is the same as in Chapter 3. In the using (BodyFrame ...) portion of the code, we create a new array of bodies if none exists and then update it with the bodyFrame.GetAndRefreshBodyData(bodies) statement. We set the dataReceived flag to true so that we know to execute the joint-drawing portion of the code. This is reset to false every time the event handler is called.

We could have chosen to draw the joints directly onto the Image WPF control, writing over the relevant color pixels in our colorBitmap, but instead we will draw the joints on a Canvas control that will be overlaid on top of our color image. This is mainly a matter of preference, though there are some trade-offs. For example, if we wanted to take a screenshot, we would not be able to use the code from Chapter 3 to do so if we were using the Canvas control. We would instead have to rely on the RenderTargetToBitmap class to take a snapshot of the Image and Canvas controls simultaneously. We opted for Canvas out of convenience.

We start off the drawing portion of our code by calling canvas.Children.Clear(), which deletes all the existing joint dots on the canvas. These were created as child WPF controls of our Canvas (called canvas).

Using LINQ, we loop through each body in our bodies array that is being tracked to apply our drawing code. The Joints property of Body contains a dictionary of key-value pairs for each joint in the tracked body. If we wanted to access a specific joint directly, we could query the JointType in the dictionary as such:

Joint j = joints[JointType.ElbowRight];

Note that the identifiers for the JointTypes are all written as camel-cased renditions of those listed in Figure 2-1 (i.e., Elbow_Right is written as ElbowRight in this context).

Looping through the joints, we start off by extracting the Position property of each joint as seen from the Depth camera, which is defined as a CameraSpacePoint. Next, we clamp its depth value (position.Z), ensuring that it is above 0, to prevent the CoordinateMapper from returning negative infinity values when we convert it to a ColorSpacePoint. These depth values can be negative as a result of algorithms used by the Kinect SDK to determine the position of inferred joints. It might seem odd that the depth value is relevant at all in the conversion from a 3D coordinate system to a 2D coordinate system, but it makes sense. If you think about it, the smaller the distance between the camera and a person, the larger their skeleton (and their joints) should appear on the screen. Thus, the relationship between the depth value and the size of the joints in the color space is inversely proportional.

After the depth value is clamped, we need to convert the depth positions to the color camera view positions. Thus, we call the MapCameraPointToColorSpace(CameraSpacePoint cameraPoint) method of CoordinateMapper to find the location of the joint on our color image. Checking the TrackingState property of each joint, we determine whether to color it purple for Tracked or light gray for Inferred. We give the coordinates of the ColorSpacePoint as a Point object to our custom DrawJoint(Point jointCoord, SolidColorBrush s) method, along with the desired color of the joint on the canvas.

Listing 5-4. MainWindow.xaml.cs Drawing Method for Displaying Skeletal Joints

private void DrawJoint(Point jointCoord, SolidColorBrush s)
{
    if (jointCoord.X < 0 || jointCoord.Y < 0)
        return;

    Ellipse ellipse = new Ellipse()
    {
        Width = 10,
        Height = 10,
        Fill = s
    };

    Canvas.SetLeft(ellipse, (jointCoord.X/colorFrameDescription.Width) * canvas.ActualWidth - ellipse.Width / 2);
    Canvas.SetTop(ellipse, (jointCoord.Y/colorFrameDescription.Height) * canvas.ActualHeight - ellipse.Height / 2);
    canvas.Children.Add(ellipse);
}

In Listing 5-4, we have the method that is responsible for drawing the joints on our canvas. We first confirm that the joint coordinates on the color image have positive values. We create a new ellipse object for the joint and give it a 10-pixel diameter and our chosen brush as the fill color. We then set the x coordinate and y coordinate of the ellipse on the Canvas with the Canvas class’ SetLeft(UIElement element, double length) and SetTop(UIElement element, double length) methods, respectively. The (0, 0) coordinate for the Canvas is the top-left pixel, as with images. There is an interesting bit of calculating we must do to determine where we put the circle on the Canvas:

Canvas.SetLeft(ellipse, (jointCoord.X/colorFrameDescription.Width) * canvas.ActualWidth - ellipse.Width / 2);

The preceding calculation also applies for the y coordinate. Normally, ColorSpacePoints refer to Cartesian coordinates on the color camera image. This image is normally 1920 x 1080 pixels, but for whatever reason, it might be displayed in a different resolution. In our case, we set the image to 1024 x 576 in our application so that it can fit better in our monitor. Thus, we must scale down the coordinate value so that it fits in this lower-resolution image. In the example of the x coordinate, we divide its value by the width of the original image and then multiply it by the width of our Canvas. SetLeft(...) and SetTop(...) set elements on the canvas by their top-left corners, so we subtract the radius of the ellipse from the length to ensure it is set to the desired location by its center.

Listing 5-5. MainWindow.xaml Front End for Displaying Skeletal Joints

<Window [...]
        Title="Skeletal Joints" Height="604" Width="1028" ResizeMode="NoResize">
    <Grid>
        <Image Source="{Binding ImageSource}" Width="1024" Height="576" />
        <Canvas Name="canvas" Width="1024" Height="576" />              
    </Grid>
</Window>

In the front end (Listing 5-5), we add the Canvas element right after the Image element so that it is displayed in front of the image and not behind. The rest of the XAML is otherwise uninteresting. Compile and run the application, then stand a short distance away from the Kinect. If you can see the joint dots on your screen like as in Figure 5-2, then you can try sitting down or appearing in other positions to see that it is working as anticipated.

Figure 5-2. Skeletal joints displayed on a user. Notice the light gray dots under the waist, which represent the inferred positions of the leg joints. The torso and waist joints are being partially shifted as a result of the table’s obstructing the lower right side of the user’s abdomen.

Drawing Bones

There is no concept of “bones” in the Kinect SDK. We have to enumerate the various connections between the provided joints ourselves before we draw them. The process of drawing them is similar to doing so for the joints. The primary difference is that instead of drawing ellipses, we will be drawing lines.

Listing 5-6. MainWindow.xaml.cs Additional Private Variables to Display Skeletons

...
private Body[] bodies; //previously declared private variables

private List<Tuple<JointType, JointType>> bones;                    
private List<SolidColorBrush> bodyColors;                    
...

In Listing 5-6, we make a list of tuples to represent the various joint relationships in the body that constitute bones. Additionally, we make a list for the colors we will use to paint each body.

Listing 5-7. MainWindow.xaml.cs Additional Constructor Code to Display Skeletons

public MainWindow()                                                                                                              
{
    bones = new List<Tuple<JointType, JointType>>();

    //Torso
    bones.Add(new Tuple<JointType, JointType>(JointType.Head, JointType.Neck));
    bones.Add(new Tuple<JointType, JointType>(JointType.Neck, JointType.SpineShoulder));
    bones.Add(new Tuple<JointType, JointType>(JointType.SpineShoulder, JointType.SpineMid));
    bones.Add(new Tuple<JointType, JointType>(JointType.SpineMid, JointType.SpineBase));
    bones.Add(new Tuple<JointType, JointType>(JointType.SpineShoulder, JointType.ShoulderRight));
    bones.Add(new Tuple<JointType, JointType>(JointType.SpineShoulder, JointType.ShoulderLeft));
    bones.Add(new Tuple<JointType, JointType>(JointType.SpineBase, JointType.HipRight));
    bones.Add(new Tuple<JointType, JointType>(JointType.SpineBase, JointType.HipLeft));

    //Left Arm
    bones.Add(new Tuple<JointType, JointType>(JointType.ShoulderLeft, JointType.ElbowLeft));
    bones.Add(new Tuple<JointType, JointType>(JointType.ElbowLeft, JointType.WristLeft));
    bones.Add(new Tuple<JointType, JointType>(JointType.WristLeft, JointType.HandLeft));
    bones.Add(new Tuple<JointType, JointType>(JointType.HandLeft, JointType.HandTipLeft));
    bones.Add(new Tuple<JointType, JointType>(JointType.WristLeft, JointType.ThumbLeft));

    //Right Arm
    bones.Add(new Tuple<JointType, JointType>(JointType.ShoulderRight, JointType.ElbowRight));
    bones.Add(new Tuple<JointType, JointType>(JointType.ElbowRight, JointType.WristRight));
    bones.Add(new Tuple<JointType, JointType>(JointType.WristRight, JointType.HandRight));
    bones.Add(new Tuple<JointType, JointType>(JointType.HandRight, JointType.HandTipRight));
    bones.Add(new Tuple<JointType, JointType>(JointType.WristRight, JointType.ThumbRight));

    //Left Leg                                                                          
    bones.Add(new Tuple<JointType, JointType>(JointType.HipLeft, JointType.KneeLeft));
    bones.Add(new Tuple<JointType, JointType>(JointType.KneeLeft, JointType.AnkleLeft));
    bones.Add(new Tuple<JointType, JointType>(JointType.AnkleLeft, JointType.FootLeft));

    //Right Leg
    bones.Add(new Tuple<JointType, JointType>(JointType.HipRight, JointType.KneeRight));
    bones.Add(new Tuple<JointType, JointType>(JointType.KneeRight, JointType.AnkleRight));
    bones.Add(new Tuple<JointType, JointType>(JointType.AnkleRight, JointType.FootRight));

    bodyColors = new List<SolidColorBrush>();

    bodyColors.Add(new SolidColorBrush(Colors.Red));
    bodyColors.Add(new SolidColorBrush(Colors.Green));
    bodyColors.Add(new SolidColorBrush(Colors.Orange));
    bodyColors.Add(new SolidColorBrush(Colors.Blue));
    bodyColors.Add(new SolidColorBrush(Colors.Yellow));
    bodyColors.Add(new SolidColorBrush(Colors.Pink));

    //remaining code in constructor
    kinect = KinectSensor.GetDefault();
    ...

In the MainWindow constructor shown in Listing 5-7, we add a bunch of “bones” to the list, which are supposed to be reasonable-looking connections between the body joints that the SDK gives us access to. Like the joints, they do not necessarily depict our anatomy exactly as in a textbook, but instead have some abstractions applied to facilitate working with them through software. For example, there are two bones between our wrist and elbow joint, the radius and the ulna, but in our demonstration code we have only one bone, described as Tuple<JointType, JointType>(JointType.ElbowLeft, JointType.WristLeft). In addition to the bones, we add six colors to the bodyColors list for the six different skeletons.

Listing 5-8. MainWindow.xaml.cs Modified FrameArrived Event Handler

...
//color and body frames grabbed and processed before this
if (dataReceived)
{
    canvas.Children.Clear();

    //Add this line
    int colorIndex = 0;                

    foreach (Body body in bodies.Where(b => b.IsTracked))
    {
          //Add the following lines
        SolidColorBrush colorBrush = bodyColors[colorIndex++];                
        Dictionary<JointType, Point> jointColorPoints = new Dictionary<JointType, Point>();                

        foreach (var joint in body.Joints)
        {

            //[...] depth clamping omitted for brevity

            ColorSpacePoint colorSpacePoint = coordinateMapper.MapCameraPointToColorSpace(position);

               //Add this line
            jointColorPoints[joint.Key] = new Point(colorSpacePoint.X, colorSpacePoint.Y);                

            //[...] DrawJoint calls omitted for brevity

        }

          //Add this foreach loop
        foreach (var bone in bones)                
        {                
            DrawBone(body.Joints, jointColorPoints, bone.Item1, bone.Item2, colorBrush);                
        }                
    }                                                                                                              
}

Listing 5-8 contains the skeleton- and joint-drawing portions of the MultiSourceFrameReader_MultiSourceFrameArrived event handler. We keep track of body colors with colorIndex so that we can maintain a unique color for each body in the frame with the SolidColorBrush colorBrush = bodyColors[colorIndex++]; statement. A dictionary is created to keep track of the ColorSpacePoints being mapped for each joint in the body. Once we have this data, we loop through all the joint-joint tuples in our bones list and draw them. We have a new DrawBone(...) method for this; we will not be modifying the DrawJoint(...) method from earlier, which is still needed for the joints.

Listing 5-9. MainWindow.xaml.cs Drawing Method for Bones

private void DrawBone(IReadOnlyDictionary<JointType, Joint> joints, IDictionary<JointType, Point> jointColorPoints, JointType jointType0, JointType jointType1, SolidColorBrush color)
{
    Joint joint0 = joints[jointType0];
    Joint joint1 = joints[jointType1];

    if (joint0.TrackingState == TrackingState.NotTracked || joint1.TrackingState == TrackingState.NotTracked)
        return;

    if (jointColorPoints[jointType0].X < 0 || jointColorPoints[jointType0].Y < 0 || jointColorPoints[jointType1].X < 0 || jointColorPoints[jointType0].Y < 0)
        return;

    Line line = new Line()
    {
        X1 = (jointColorPoints[jointType0].X / colorFrameDescription.Width) * canvas.ActualWidth,
        Y1 = (jointColorPoints[jointType0].Y / colorFrameDescription.Height) * canvas.ActualHeight,
        X2 = (jointColorPoints[jointType1].X / colorFrameDescription.Width) * canvas.ActualWidth,
        Y2 = (jointColorPoints[jointType1].Y / colorFrameDescription.Height) * canvas.ActualHeight,
        StrokeThickness = 5,
        Stroke = color
    };
    canvas.Children.Add(line);
}

In the DrawBone(...) method featured in Listing 5-9, we begin by refusing to draw any bones for situations where the necessary data is not present. This is namely when any of the joints are not tracked, or when their color-image coordinates are not positive values. Next, we create a line with the coordinates of the joints as its start and end points, along with the desired color inputted from bodyColor[colorIndex++]. Notice that the joint coordinates fed into the line are scaled as we did with the joints. Finally, we added the line to the canvas. Compiling the application should result in a scene like that in Figure 5-3.

Figure 5-3. Skeleton displayed on a user

Showing Clipped Edges

More often than not, users will not fully position their body within the view range of the Kinect. The Kinect SDK has an easy way for us to find out if such is the case. Each Body object has a ClippedEdge property that tells us on which edges of the image the user’s skeleton is being clipped on. This can be useful in many ways, such as knowing when to tell the user to move back into the Kinect’s view range, as in Figure 5-4.

Figure 5-4. Clipped edges shown as red lines on the sides. The left edge is being shown as clipped because the person’s right leg joints are inferred as being past that edge on the frame .

Drawing clipped edges in our existing application only requires minor edits. We will include a method to draw lines on the sides of the image representing the clipped edges and call it from the event handler for MultiSourceFrame.

Listing 5-10. MainWindow.xaml.cs Method for Drawing Clipped Edges

private void DrawClippedEdges(Body body)
{
    FrameEdges clippedEdges = body.ClippedEdges;

    if (clippedEdges.HasFlag(FrameEdges.Bottom))
    {
        Line edge = new Line()                                                                                      
        {
            X1 = 0,
            Y1 = canvas.ActualHeight - 9,
            X2 = canvas.ActualWidth,
            Y2 = canvas.ActualHeight - 9,
            StrokeThickness = 20,
            Stroke = new SolidColorBrush(Colors.Red)
        };
        canvas.Children.Add(edge);
    }

    if (clippedEdges.HasFlag(FrameEdges.Top))
    {
        Line edge = new Line()
        {
            X1 = 0,
            Y1 = 0,
            X2 = canvas.ActualWidth,
            Y2 = 0,
            StrokeThickness = 20,
            Stroke = new SolidColorBrush(Colors.Red)
        };
        canvas.Children.Add(edge);
    }

    if (clippedEdges.HasFlag(FrameEdges.Left))
    {
        Line edge = new Line()
        {
            X1 = 0,
            Y1 = 0,
            X2 = 0,
            Y2 = canvas.ActualHeight,
            StrokeThickness = 20,
            Stroke = new SolidColorBrush(Colors.Red)
        };
        canvas.Children.Add(edge);
    }

    if (clippedEdges.HasFlag(FrameEdges.Right))
    {
        Line edge = new Line()
        {
            X1 = canvas.ActualWidth - 9,
            Y1 = 0,
            X2 = canvas.ActualWidth - 9,
            Y2 = canvas.ActualHeight,
            StrokeThickness = 20,
            Stroke = new SolidColorBrush(Colors.Red)
        };
        canvas.Children.Add(edge);                                                                                       
    }
}

In Listing 5-10, we can see that checking whether an edge is clipped is a matter of checking which FrameEdges are in body.ClippedEdges. Once we have this, we can draw a line on the canvas for each flagged frame edge. Finally, the DrawClippedEdges(Body body) method needs to be called in the loop in which we go through all the bodies and draw their joints and bones, as follows:

foreach (Body body in bodies.Where(b => b.IsTracked))
{
    //[...] Rest of code to gather and draw joints

    foreach (var bone in bones)
    {
        DrawBone(body.Joints, jointColorPoints, bone.Item1, bone.Item2, colorBrush);
    }

    //Add this line to call DrawClippedEdges
    DrawClippedEdges(body);                

}

Showing Hand States

While the Kinect supports more advanced gesture-recognition capabilities, being able to near-instantaneously determine whether the hands are clenched (closed), open, or in a lasso state is very convenient for smaller projects where we do not want to spend time defining custom gestures.

As with the clipped edges, showing hand states will not require a significant alteration of our existing application. We will add a method to paint a circle around the hand depending on the hand state. We will have Red represent a closed first, Green an open one, and Blue for one in the lasso form.

Listing 5-11. MainWindow.xaml.cs Method to Show Hand State with Colored Circles

private void DrawHandStates(HandState handState, Point handCoord)
        {
            switch (handState)
            {
                case HandState.Closed:
                    Ellipse closedEllipse = new Ellipse()
                    {
                        Width = 100,
                        Height = 100,
                        Fill = new SolidColorBrush(Color.FromArgb(128, 255, 0, 0))
                    };
                    Canvas.SetLeft(closedEllipse, (handCoord.X / colorFrameDescription.Width) * canvas.ActualWidth - closedEllipse.Width / 2);
                    Canvas.SetTop(closedEllipse, (handCoord.Y / colorFrameDescription.Height) * canvas.ActualHeight - closedEllipse.Width / 2);
                    canvas.Children.Add(closedEllipse);
                    break;                                                                                       

                case HandState.Open:
                    Ellipse openEllipse = new Ellipse()
                    {
                        Width = 100,
                        Height = 100,
                        Fill = new SolidColorBrush(Color.FromArgb(128, 0, 255, 0))
                    };
                    Canvas.SetLeft(openEllipse, (handCoord.X / colorFrameDescription.Width) * canvas.ActualWidth - openEllipse.Width / 2);
                    Canvas.SetTop(openEllipse, (handCoord.Y / colorFrameDescription.Height) * canvas.ActualHeight - openEllipse.Width / 2);
                    canvas.Children.Add(openEllipse);
                    break;

                case HandState.Lasso:
                    Ellipse lassoEllipse = new Ellipse()
                    {
                        Width = 100,
                        Height = 100,
                        Fill = new SolidColorBrush(Color.FromArgb(128, 0, 0, 255))
                    };
                    Canvas.SetLeft(lassoEllipse, (handCoord.X / colorFrameDescription.Width) * canvas.ActualWidth - lassoEllipse.Width / 2);
                    Canvas.SetTop(lassoEllipse, (handCoord.Y / colorFrameDescription.Height) * canvas.ActualHeight - lassoEllipse.Width / 2);
                    canvas.Children.Add(lassoEllipse);
                    break;
            }
        }

In Listing 5-11, we see the method that is used to paint circles representing hand states onto the canvas. HandState is an enum with five possible values, the aforementioned Open, Closed, and Lasso, as well as Unknown and NotTracked. The switch-case statement finds out what state the hand is in and then basically draws the circle in a manner similar to the DrawJoint method using the ColorSpacePoint coordinates of the hand. The method is called twice per hand, as follows:

...
//[...] Rest of code to gather and draw joints and bones
DrawClippedEdges(body);
                                                                                  
//Add these lines to MultiSourceFrameReader_MultiSourceFrameArrived
DrawHandStates(body.HandRightState, jointColorPoints[JointType.HandRight]);                  
DrawHandStates(body.HandLeftState, jointColorPoints[JointType.HandLeft]);                  
...

This method is called in the MultiSourceFrame event handler right after (or before) the DrawClippedEdges(Body body) method. The HandState enumeration can be obtained from the HandRightState and HandLeftState properties of Body. The color space coordinates of the joints that we derived earlier are used here to tell the method where to draw the hand-state circles. Compiling and running the application at this point should result in a scene similar to that in Figure 5-5.

Figure 5-5. Hand states depicted with colored circles

It is worth knowing that Body has HandLeftConfidence and HandRightConfidence properties that return a TrackingConfidence value for the left and right hand states, respectively. TrackingConfidence is nothing but an enum with two values, High and Low, whose meanings are self-described.

Obtaining Face Data

Data from the Face API is obtained in a manner similar to doing so for the other Data Sources. The primary differences are that we must designate a separate FaceFrameSource and FaceFrameReader for each face we are looking to pick out and that we have to extract a FaceFrameResult, which is what contains the recognized facial features and data, from each FaceFrame.

Listing 5-21. Private Variables Declaration to Obtain Face Data

KinectSensor kinect;

const float InfraredSourceValueMaximum = (float)ushort.MaxValue;
const float InfraredSourceScale = 0.75f;
const float InfraredOutputValueMinimum = 0.01f;
const float InfraredOutputValueMaximum = 1.0f;

MultiSourceFrameReader multiSourceFrameReader;
FrameDescription frameDescription;
Body[] bodies;

FaceFrameSource[] faceFrameSources;                    
FaceFrameReader[] faceFrameReaders;                    
FaceFrameResult[] faceFrameResults;                    

WriteableBitmap infraredBitmap;
ushort[] infraredPixels;
DrawingGroup drawingGroup;
DrawingImage drawingImageSource;
List<Brush> faceBrushes;                                                                                       

In Listing 5-21, we declare arrays in which to contain the frame sources, readers, and results for all the faces that can be simultaneously tracked at an instant. Note that we also added an array for tracked bodies. While tracking body data is not a prerequisite to tracking facial data, in practice you will probably always do so anyway. Among other reasons, this is so we can update the FaceFrameSources to ensure that they are tracking faces from bodies visible in the scene.

Listing 5-22. Constructor for App Using Face Data

public MainWindow()
{
    kinect = KinectSensor.GetDefault();

    InfraredFrameSource infraredFrameSource = kinect.InfraredFrameSource;

    multiSourceFrameReader = kinect.OpenMultiSourceFrameReader(FrameSourceTypes.Infrared | FrameSourceTypes.Body);
    multiSourceFrameReader.MultiSourceFrameArrived += MultiSource_FrameArrived;

    frameDescription = infraredFrameSource.FrameDescription;

    bodies = new Body[kinect.BodyFrameSource.BodyCount];

    FaceFrameFeatures faceFrameFeatures =                
        FaceFrameFeatures.BoundingBoxInInfraredSpace                
        | FaceFrameFeatures.PointsInInfraredSpace                
        | FaceFrameFeatures.RotationOrientation                
        | FaceFrameFeatures.FaceEngagement                
        | FaceFrameFeatures.Glasses                
        | FaceFrameFeatures.Happy                
        | FaceFrameFeatures.LeftEyeClosed                
        | FaceFrameFeatures.RightEyeClosed                
        | FaceFrameFeatures.LookingAway                
        | FaceFrameFeatures.MouthMoved                
        | FaceFrameFeatures.MouthOpen;                

    faceFrameSources = new FaceFrameSource[6];                
    faceFrameReaders = new FaceFrameReader[6];                
    faceFrameResults = new FaceFrameResult[6];                

    faceBrushes = new List<Brush>()                
    {                
        Brushes.Pink,                
        Brushes.Orange,                
        Brushes.BlueViolet,                
        Brushes.Aqua,                
        Brushes.LawnGreen,                
        Brushes.                                          Chocolate                                                                                      
    };                

    for (int i = 0; i < 6; i++)                
    {                
        faceFrameSources[i] = new FaceFrameSource(kinect, 0, faceFrameFeatures);                
        faceFrameReaders[i] = faceFrameSources[i].OpenReader();                
        faceFrameReaders[i].FrameArrived += Face_FrameArrived;                
    }                

    infraredBitmap = new WriteableBitmap(frameDescription.Width,
         frameDescription.Height,
         96.0, 96.0,
         PixelFormats.Gray32Float,
         null);
    infraredPixels = new ushort[frameDescription.LengthInPixels];
    drawingGroup = new DrawingGroup();                
    drawingImageSource = new DrawingImage(drawingGroup);                

    kinect.Open();

    DataContext = this;

    InitializeComponent();
}                                                                                      

With the Face API, we need to declare in advance which features we plan to extract from the FaceFrame. This is why we assign the desired features in the FaceFrameFeatures enum in Listing 5-22. In addition to the features listed, there are also FaceFrameFeatures_BoundingBoxInColorSpace and FaceFrameFeatures_PointsInColorSpace. We are using the infrared space equivalents, which is why we did not include them.

We create the arrays for the face frame sources, readers, and results for the number of faces that we are interested in tracking—in our case, all six. We then loop through the FaceFrameSource and FaceFrameReader arrays and create the FaceFrameSources and set up their readers. The constructor method for FaceFrameSource, new FaceFrameSource(kinect, 0, faceFrameFeatures), takes the Kinect sensor, initial tracking ID, and desired face frame features as inputs. The tracking ID is a property of the Body class, and we can use it to set the FaceFrameSource to track the face of a specific body.

For our sample app making use of face data, we will display the features on top of an infrared video stream. For this, we have to construct our ImageSource slightly differently. Instead of displaying the infrared stream’s WriteableBitmap directly, we create a DrawingGroup, which can be converted to an ImageSource by creating a DrawingImage out of it. As this is not a book on .NET or WPF, the peculiarities are not important. Just know that we can add multiple visual elements to a DrawingGroup and that DrawingImage is a type of ImageSource. Instead of returning the WriteableBitmap in the getter for our public ImageSource, we return the DrawingImage object, as follows:

public ImageSource ImageSource
{
    get
    {
        return drawingImageSource;
    }
}

Listing 5-23. FaceFrameReader Event Handler

private void Face_FrameArrived(object sender, FaceFrameArrivedEventArgs e)
{
    using (FaceFrame faceFrame = e.FrameReference.AcquireFrame())
    {
        if (faceFrame != null)
        {
            int index = GetFaceSourceIndex(faceFrame.FaceFrameSource);

            faceFrameResults[index] = faceFrame.FaceFrameResult;
        }
    }
}

In Listing 5-23, we save the FaceFrameResult, which is a property of FaceFrame, to our faceFrameResults array. We want to keep track of its index so that we can work with its respective frame source, frame reader, and body. Since we have a reference to its FaceFrameSource in the FaceFrame, we take the initiative of finding its index using GetFaceSoureIndex(FaceFrameSource faceFrameSource), as shown in Listing 5-24. The Face_FrameArrived event handler is applied to each reader.

Listing 5-24. Obtaining the Index of a FaceFrameSource

private int GetFaceSourceIndex(FaceFrameSource faceFrameSource)
{
    int index = -1;

    for (int i = 0; i < 6; i++)
    {
        if (faceFrameSources[i] == faceFrameSource)
        {
            index = i;
            break;
        }
    }

    return index;
}

Finding the index is a straightforward matter. We loop through our faceFrameSources array and check whether any of them match the FaceFrameSource of the obtained FaceFrame. If they do, we know what index to put our FaceFrameResult in.

Listing 5-25. MultiSourceFrameReader Event Handler for Face-Detector App

private void MultiSource_FrameArrived(object sender, MultiSourceFrameArrivedEventArgs e)
{
    MultiSourceFrame multiSourceFrame = e.FrameReference.AcquireFrame();

    using (InfraredFrame infraredFrame = multiSourceFrame.InfraredFrameReference.AcquireFrame())
    {
        if (infraredFrame != null)
        {
            using (KinectBuffer infraredBuffer = infraredFrame.LockImageBuffer())
            {
                ProcessInfraredFrameData(infraredBuffer.UnderlyingBuffer, infraredBuffer.Size);
            }
        }
    }

    using (BodyFrame bodyFrame = multiSourceFrame.BodyFrameReference.AcquireFrame())
    {
        if (bodyFrame != null)
        {
            bodyFrame.GetAndRefreshBodyData(bodies);

            using (DrawingContext dc = drawingGroup.Open())                
            {                
                dc.DrawImage(infraredBitmap, new Rect(0, 0,                
                    infraredBitmap.Width,                
                    infraredBitmap.Height));                

                for (int i = 0; i < 6; i++)                
                {                
                    if (faceFrameSources[i].IsTrackingIdValid)                
                    {                
                        if (faceFrameResults[i] != null)                
                        {                
                            DrawFace(i, faceFrameResults[i], dc);                
                        }                
                    }                
                    else                
                    {                
                        if (bodies[i].IsTracked)                
                        {                
                            faceFrameSources[i].TrackingId = bodies[i].TrackingId;                
                        }                                                                                                        
                    }                
                }                
            }                
        }
    }
}

private unsafe void ProcessInfraredFrameData(IntPtr infraredFrameData, uint infraredFrameDataSize)
{
    //infrared frame data is a 16-bit value
    ushort* frameData = (ushort*)infraredFrameData;

    //lock the target bitmap
    this.infraredBitmap.Lock();

    //get the pointer to the bitmap's back buffer
    float* backBuffer = (float*)this.infraredBitmap.BackBuffer;

    //process the infrared data
    for (int i = 0; i < (int)(infraredFrameDataSize / this.frameDescription.BytesPerPixel); ++i)
    {
        //since we are displaying the image as a normalized grayscale image, we need to convert from
        //the ushort data (as provided by the InfraredFrame) to a value from
        //[InfraredOutputValueMinimum, InfraredOutputValueMaximum]
        backBuffer[i] = Math.Min(InfraredOutputValueMaximum, (((float)frameData[i] / InfraredSourceValueMaximum * InfraredSourceScale) * (1.0f - InfraredOutputValueMinimum)) + InfraredOutputValueMinimum);
    }

    //mark the entire bitmap as needing to be drawn
    this.infraredBitmap.AddDirtyRect(new Int32Rect(0, 0, this.infraredBitmap.PixelWidth, this.infraredBitmap.PixelHeight));

    //unlock the bitmap
    this.infraredBitmap.Unlock();
}                                                                                      

In Listing 5-25, the event handler for the MultiSourceFrameReader collects infrared and body data. Seeing as this represents a single event that is fired repeatedly, as opposed to six different events as with the FaceFrameReader, it is better to include the drawing code in here. This way, we avoid having multiple unordered methods trying to access the DrawingGroup in a similar timeframe, all trying to overwrite it.

We obtain the infrared data in the same we did in Listing 3-14 in Chapter 3. The XAML front end is also the same as in Chapter 3. Inside of the using (BodyFrame bodyFrame = multiSourceFrame.BodyFrameReference.AcquireFrame()) { [...] } block we access the DrawingContext of drawingGroup. DrawingContext is a .NET construct that allows us to draw various types of media in a DrawingGroup (among other drawing-related objects). We start off by drawing our infrared image data as the background, and then we loop through the faceFrameResults and draw any facial data with the DrawFace(int index, FaceFrameResult faceFrameResult, DrawingContext drawingContext) method (which will be defined later). Keep in mind that we only do this if the faceFrameSources[i].IsTrackingIdValid property returns true. If not, we consider whether a body is being checked for index i and, if so, assign its TrackingId to its respective FaceFrameSource (faceFrameSources[i].TrackindId = bodies[i].TrackingId).

At this point, if you were to run the code, minus the DrawFace(...) method, it would obtain facial data and display an infrared feed to boot, but it would not actually indicate that it has obtained facial data, let alone do anything with it. We thus venture to the first Face API feature, detection, to inform the user that the face has been found by the Kinect.

Detection

In the Face API, detection refers to whether any faces belonging to bodies are visible in the Kinect’s field of view. In practice, the Face API achieves this by framing a face inside of a bounding box, as seen in Figure 5-13. This bounding box is a property of FaceFrameResult called either FaceBoundingBoxInInfraredSpace or FaceBoundingBoxInColorSpace depending on the image feed you are interested in. It is a RectI struct—a rectangle with integer coordinates. The RectI’s Top and Bottom fields each represent a pixel coordinate on the vertical axis, and the Right and Left fields represent the same on a horizontal axis. Thus, a RectI with the following fields would represent a rectangle whose top-left coordinate is (0, 10), and its size would be 13 pixels wide and 10 pixels in height.

Figure 5-13. Bounding box representation (FaceBoundingBoxInInfraredSpace) around a detected face. Note that the box drawn on screen is thicker than the actual bounding box, given that the pen used to draw it has a thickness.

RectI box = new RectI(); //has no constructor
Box.Top = 10;
Box.Bottom = 20;
Box.Left = 0;                                                                               
Box.Right = 13;

Before we can draw the bounding box on our infrared image, it would be prudent for us to ensure that the box fits within the image. Hence, when new FaceFrameResults are collected in Face_FrameArrived(...), we should reject results whose bounding boxes do not fit. This involves a small edit to Face_FrameArrived(...), whose initial version we saw in Listing 5-23.

Listing 5-26. Face Bounding Box Validation Method Call

if (faceFrame != null)
{
    int index = GetFaceSourceIndex(faceFrame.FaceFrameSource);

    if (ValidateFaceBoundingBox(faceFrame.FaceFrameResult))                
    {                
        faceFrameResults[index] = faceFrame.FaceFrameResult;                
    }                
    else                
    {                
        faceFrameResults[index] = null;                
    }                
}

In Listing 5-26, we check to see if the bounding box is within the confines of the image with our custom ValidateFaceBoundingBox (FaceFrameResult faceFrameResult) method and reject the FaceFrameResult if it is not.

Listing 5-27. Face Bounding Box Validation

private bool ValidateFaceBoundingBox(FaceFrameResult faceFrameResult)
{
    bool isFaceValid = faceFrameResult != null;

    if (isFaceValid)
    {
        RectI boundingBox = faceFrameResult.FaceBoundingBoxInInfraredSpace;
        if (boundingBox!= null)
        {
            isFaceValid = (boundingBox.Right - boundingBox.Left) > 0 &&
                          (boundingBox.Bottom - boundingBox.Top) > 0 &&
                          boundingBox.Right <= frameDescription.Width &&
                          boundingBox.Bottom <= frameDescription.Height;
        }
    }

    return isFaceValid;
}

In Listing 5-27, we immediately reject empty FaceFrameResults. We then check to see if the box has meaningful coordinates, e.g., the right side of the bounding box is to the right of the box’s left side ((boundingBox.Right – boundingBox.Left) > 0). If it does, we approve of the FaceFrameResult, and it can be used in the DrawFace(...) method to draw the bounding box.

Listing 5-28. Drawing a Bounding Box

private void DrawFace(int index, FaceFrameResult faceFrameResult, DrawingContext drawingContext)
{
    Brush drawingBrush = faceBrushes[0];
    if (index < 6)
    {
        drawingBrush = faceBrushes[index];
    }

    Pen drawingPen = new Pen(drawingBrush, 4);

    RectI faceBoxSource = faceFrameResult.FaceBoundingBoxInInfraredSpace;
    Rect faceBox = new Rect(faceBoxSource.Left, faceBoxSource.Top, faceBoxSource.Right - faceBoxSource.Left, faceBoxSource.Bottom - faceBoxSource.Top);
    drawingContext.DrawRectangle(null, drawingPen, faceBox);

}

Listing 5-28 features the DrawFace(...) method we called earlier in the MultiFrameSourceReader event handler in Listing 5-25. We start by picking a Brush, which contains our desired color for the boxes drawn around faces. The brushes are from a list initialized in the app constructor as such:

faceBrushes = List<Brush>()
{
    Brushes.Pink,
    Brushes.Orange,                                                                               
    [...] //Add six brushes in total with differing colors
};

The brush is then used to create a Pen that includes the desired thickness of the box’s outline as well. Finally, we create a rectangle based on the bounding box in FaceFrameResult and draw the rectangle on the DrawingContext, which will then display it to the front end. The compiled result will resemble the previously shown Figure 5-13.

Alignment

Alignment refers to Face API’s ability to easily detect five facial landmarks on a face. These landmarks, or points, consist of both eyes, the tip of the nose, and both corners of the mouth. They can be obtained from the FacePointsInInfraredSpace or FacePointsInColorSpace properties of FaceFrameResult (again, the difference between both is which image feed the landmarks map to). FacePointsInXSpace is an IReadOnlyDictionary<FacePointType, PointF>, where FacePointType refers to the title of the point (e.g., MouthCornerLeft) and PointF is a 2D coordinate with float members.

Generally, the points will be found within the bounding box, but it is better to ensure that they fit within the image confines anyway. The ValidateFaceBoundingBox(...) method from Listing 5-27 can be altered to accommodate this check.

Listing 5-29. Face Bounding Box and Points Validation

private bool ValidateFaceBoxAndPoints(FaceFrameResult faceFrameResult)                    
{
    bool isFaceValid = faceFrameResult != null;

    if (isFaceValid)
    {
        RectI boundingBox = faceFrameResult.FaceBoundingBoxInInfraredSpace;
        if (boundingBox != null)
        {
            [...] //Check if bounding box is valid

            if (isFaceValid)                
            {                
                var facePoints = faceFrameResult.FacePointsInInfraredSpace;                
                if (facePoints != null)                
                {                
                    foreach (PointF pointF in facePoints.Values)                
                    {                
                        bool isFacePointValid = pointF.X > 0.0f &&                
                        pointF.Y > 0.0f &&                
                        pointF.X < frameDescription.Width &&                
                        pointF.Y < frameDescription.Height;                

                        if (!isFacePointValid)                
                        {                
                            isFaceValid = false;                
                            break;                
                        }                
                    }                
                }                
            }                
        }
    }
    return isFaceValid;
}

In Listing 5-29, we modified the ValidateFaceBoundingBox(...) method to conduct this check. The check consists of looping through the dictionary of face points and confirming that their x and y values fit in the image. On a side note, the method signature was also altered to reflect its new duties. Make sure to update the method call in Face_FrameFrameArrived(...).

Listing 5-30. Indicating Facial Landmarks on an Image Feed

...
drawingContext.DrawRectangle(null, drawingPen, faceBox);
//rest of DrawFace(...) body

if (faceFrameResult.FaceBoundingBoxInInfraredSpace != null)
{
    foreach (PointF pointF in faceFrameResult.FacePointsInInfraredSpace.Values)
    {
        drawingContext.DrawEllipse(null, drawingPen, new Point(pointF.X, pointF.Y), 0.4, 0.4);
    }
}

In Listing 5-30, we loop through every face point in faceFrameResult and draw an ellipse on the location of the landmark on the image. The final result looks like Figure 5-14.

Figure 5-14. Facial landmark locations drawn on an infrared image

Orientation

The Face API gives us access to the face’s orientation, facing the Kinect, in 3D space. It can be accessed through FaceFrameResult’s FaceRotationQuaternion property. It is defined with a Vector4. The face orientation is given as a quaternion, similar to joint orientations. This was again done to avoid gimbal lock. Face orientation can be useful in many circumstances, but gaze tracking deserves special mention. It is not possible to determine where a user is looking with the Kinect API, but knowing in which direction their face is pointing is good enough for most use cases and is the solution that most unspecialized commercial systems employ.

The Kinect SDK Browser’s FaceBasics-WPF sample includes a convenient method by which to obtain a Euler angle in degrees from the face-rotation quaternion. If you are using a game engine or DirectX, you might choose to simply feed the quaternion into them directly, but if you are making a relatively simple app that does something like checking whether a head is tilting, using Euler angles with a heuristic system might be less work for you.

Listing 5-31. Method to Obtain Euler Angles from Face-Rotation Quaternion

/**usage:
int pitch, yaw, roll; ExtractFaceRotationInDegrees(faceFrameResult.FaceRotationQuaternion, out pitch, out yaw, out roll);
**/
private static void ExtractFaceRotationInDegrees(Vector4 rotQuaternion, out int pitch, out int yaw, out int roll)
{
    double x = rotQuaternion.X;
    double y = rotQuaternion.Y;
    double z = rotQuaternion.Z;
    double w = rotQuaternion.W;                                                                                       

    double yawD, pitchD, rollD;
    pitchD = Math.Atan2(2 * ((y * z) + (w * x)), (w * w) - (x * x) - (y * y) + (z * z)) / Math.PI * 180.0;
    yawD = Math.Asin(2 * ((w * y) - (x * z))) / Math.PI * 180.0;
    rollD = Math.Atan2(2 * ((x * y) + (w * z)), (w * w) + (x * x) - (y * y) - (z * z)) / Math.PI * 180.0;

    double increment = 5.0;
    pitch = (int)(Math.Floor((pitchD + ((increment / 2.0) * (pitchD > 0 ? 1.0 : -1.0))) / increment) * increment);
    yaw = (int)(Math.Floor((yawD + ((increment / 2.0) * (yawD > 0 ? 1.0 : -1.0))) / increment) * increment);
    roll = (int)(Math.Floor((rollD + ((increment / 2.0) * (rollD > 0 ? 1.0 : -1.0))) / increment) * increment);
}

The formulas in Listing 5-31 might look complex to the uninitiated, but you should not have to worry about their specifics. The first three formulas that calculate pitchD, yawD, and rollD are responsible for converting our quaternion into a Euler angle. The next three, used to calculate pitch, yaw, and roll, are clamping functions. They basically round the Euler angle values to the nearest multiple of the increment value. For example, if the pitch is actually 53.01666666 degrees and we set the increment to 5, it would appear as 55 degrees.

Expressions

Expressions constitute the final portion of the Face API’s capabilities. You might have noticed that the Body class has a handful of members, namely Activities, Appearance, Engaged, and Expressions, that are unusable. These were included for cross-compatibility with the Xbox Kinect API. However, we are able to access the same capabilities though the Face API. The Face API has a series of binary classifiers that can be used to determine whether a person is happy, wearing glasses, has their left or right eye closed, is looking away, is engaged with the application, if their mouth has moved, and if their mouth is open. In the case of looking away and engagement, the classifiers assume that the user is facing toward the Kinect. Thus, if you use these classifiers in your app, make sure that the Kinect is above or beneath the screen or location that the user is supposed to be looking at.

The expressions can be accessed from the FaceProperties property of FaceFrameResult. FaceProperties is an IReadOnlyDictionary<FaceProperty, DetectionResult>. FaceProperty is an enum that contains the title of the expression (e.g., Happy or Engaged), and DetectionResult is an enum that be Yes, No, Maybe, or Unknown to indicate whether the expression has been detected.

To demonstrate, let us display a smiley face on the screen when the user is happy.

Listing 5-32. Displaying a Smiley Face When a Face Indicates Happiness

if (faceFrameResult.FaceProperties != null)
{
    if (faceFrameResult.FaceProperties[FaceProperty.Happy] == DetectionResult.Yes)
    {
        Point nosePoint = new Point(faceFrameResult.FacePointsInInfraredSpace[FacePointType.Nose].X,
                        faceFrameResult.FacePointsInInfraredSpace[FacePointType.Nose].Y);
        drawingContext.DrawText(new FormattedText(
            "☺",
            System.Globalization.CultureInfo.GetCultureInfo("en-us"),
            FlowDirection.RightToLeft,
            new Typeface("Segoe UI"),
            68,
            drawingBrush),
            nosePoint);
    }
}

Listing 5-32 is a snippet of code that can be include in the DrawFace(...) method to draw a ☺ character on the screen when the DetectionResult of the Happy classifier holds a Yes value. We use the coordinates of the Nose face point to draw the character. The character’s top-right corner, which consists of empty space, is set at the location of the nose. DrawingContext’s DrawText(...) method requires a FormattedText and Point to output text onto the image. In practice, a TextBlock XAML control can offer more customization options than can those available through FormattedText, but it requires a bit more wiring. The completed work should look like Figure 5-15.

Figure 5-15. Smiley face character displayed when face appears to be happy. The bounding box and facial landmark visualizations were omitted in this picture.

Obtaining HD Face data

Data procurement for HD Face is akin to that for any other Data Source. As you could have expected, there are a reader and a Data Source for HD face data, HighDefinitionFaceFrameReader and HighDefinitionFaceFrameSource, respectively, from which we collect frames containing data. The resulting HighDefinitionFaceFrame is responsible for the two central pieces of HD Face data: FaceModel and FaceAlignment.

FaceModel contains the structural details of the face. Namely, this includes the FaceShapeDeformations property, which is an IReadOnlyDictionary<FaceShapeDeformations, float> describing the shape units applied to the face. It also includes a TriangleIndices property, an IReadOnlyList<uint> that lists the 7,890 indices of the triangles used to create the face mesh for modeling; SkinColor and HairColor, which are 32-bit unsigned integers that describe a 4-byte ARGB color describing the skin color or hair color; Scale, a float representing the scaling factor of the face; and CalculateVerticesForAlignment(FaceAlignment faceAlignment), a method that furnishes a list of CameraSpacePoints representing facial vertices calculated from the FaceAlignment.

uint color = faceModel.SkinColor;
Byte A = (byte) (color >> 24);
Byte B = (byte) (color >> 16);
Byte G = (byte) (color >> 8);                                                                                       
Byte R = (byte) (color >> 0);
Color skinColor = Color.FromArgb(A, R, G, B);

FaceAlignment contains orientation and animation unit data. Its members include AnimationUnits, which is an IReadOnlyDictionary<FaceShapeAnimations, float>, FaceShapeAnimations being an enum describing the animation unit FaceOrientation, a Vector4 representing a quaternion, and HeadPivotPoint, which is a CameraSpacePoint. Since the locations of vertices depend on how the face model is rotated, in practice it is more critical to our application than is FaceModel, for which we can just use the default average face model provided by the Kinect.

Listing 5-33. Private Variable Declaration to Obtain HD Face Data

private KinectSensor kinect;

private BodyFrameReader bodyFrameReader;

private HighDefinitionFaceFrameSource HDFaceFrameSource;
private HighDefinitionFaceFrameReader HDFaceFrameReader;
private FaceModel faceModel;
private FaceAlignment faceAlignment;

Listing 5-33 features the essential private variables for HD face tracking. We see that body data is to be tracked in addition to face data. While this was imperative for regular face tracking, it is downright necessary for HD face tracking. Since only one face can be tracked in HD, the TrackingId of the face must be manually picked from one of the available bodies.

Listing 5-34. Initializing HD Face Data Source and Reader

HDFaceFrameSource = new HighDefinitionFaceFrameSource(Kinect);
HDFaceFrameReader = HDFaceFrameSource.OpenReader();
HDFaceFrameReader.FrameArrived += HDFace_FrameArrived;

faceAlignment = new FaceAlignment();                                                                                      

In Listing 5-34, HD Face’s Data Source is shown to be perhaps the simplest no-frills Data Source to get running. Additionally, we create an instance of the FaceAlignment object to hold its data throughout the lifetime of the application. Like with an array of Bodies, FaceAlignment will be updated instead of being re-initialized on each HD Face frame.

Listing 5-35. BodyFrameReader Event Handler for HD Face Tracking

private void Body_FrameArrived(object sender, BodyFrameArrivedEventArgs e)
{
    using (BodyFrame bodyFrame = e.FrameReference.AcquireFrame())
    {
        if (bodyFrame != null)
        {
            Body[] bodies = new Body[bodyFrame.BodyCount];
            bodyFrame.GetAndRefreshBodyData(bodies);

            Body body = bodies.Where(b => b.IsTracked).FirstOrDefault();                

            if (!HDFaceFrameSource.IsTrackingIdValid)                
            {                
                if (body != null)                
                {                
                    HDFaceFrameSource.TrackingId = body.TrackingId;                
                }                
            }                
        }
    }
}

In Listing 5-35, our goal is to determine which body’s face to track. We otherwise do not need to touch the skeleton data. Using a bit of LINQ, we pick the first available body, and if the HighDefinitionFaceFrameSource is not currently tracking a body, we assign it a body to track.

Listing 5-36. HighDefinitionFaceFrameReader Event Handler

private void HDFAce_FrameArrived(object sender, HighDefinitionFaceFrameArrivedEventArgs e)
{
    using (HighDefinitionFaceFrame HDFrame = e.FrameReference.AcquireFrame())
    {
        if (HDFrame!= null && HDFrame.IsFaceTracked)
        {
                                                                                                    HDFrame.GetAndRefreshFaceAlignmentResult(faceAlignment);                
            if (faceModel == null) {
                faceModel = HDFrame.FaceModel;

            }
            ProcessHDFaceData();
        }
    }
}

In Listing 5-36, we see that FaceAlignment is collected through the GetAndRefereshFaceAlignmentResult(FaceAlignment faceAlignment) method of HighDefinitionFaceFrame. If we so choose, we can also grab the FaceModel, but this is not necessary. The standard face model provided by the Kinect could be initialized in the constructor and coupled with the FaceAlignment to provide a decent representation of a user’s face. After this point, we are free to process our data for whatever purpose.

It should be noted that HighDefinitionFaceFrame has an additional property, FaceAlignmentQuality, which is an enum with High and Low values to describe how accurate the obtained FaceAlignment is.

Listing 5-37. Processing HD Face Data

private void ProcessHDFaceData()
{
    if (faceModel != null)
    {
        var vertices = faceModel.CalculateVerticesForAlignment(faceAlignment);
        //Perform work with vertices, such as displaying on screen
    }
}

Other than the vertices, the rest of HD Face’s data can be gleaned from FaceModel and FaceAlignment directly. As mentioned earlier, the position of the vertices in a 3D space depends on how their constituent model is rotated in 3D space. Hence, in Listing 5-37 we must calculate them with the CalculateVerticesForAlignment(FaceAlignment faceAlignment) method. This is not a static method; it does depend on the data of a FaceModel instance. However, it can be used on the data in the default initialization of FaceModel without gathering anything from HighDefinitionFaceFrames. Despite this possibility, HD Face unsurprisingly shines brightest when it gets a chance to reconstruct a FaceModel from an analysis of a user’s face, which we will cover next .

Face Capture & Modeling

With the vertices provided from FaceModel and FaceAlignment, we can create a 3D mesh of the user’s face by connecting them into triangles. To get a more accurate representation, however, we have to use the FaceModelBuilder class to produce a FaceModel with shape unit deformations applied. FaceModelBuilder needs to capture 16 frames from all sides of the face. Thus, with the help of the API, you will have to inform users of your app when to move their faces to be properly captured. Your users should take off any accessories on or near their faces, such as glasses or hats.

To learn how to use FaceModelBuilder and create a mesh with or without it, we will turn to the HDFaceBasics-WPF sample in the SDK Browser. This app (shown in Figure 5-17) creates a 3D visualization of a user, and once face capture has been completed, this visualization is deformed to look more like the user’s face.

Figure 5-17. 3D visualization created by HD Face. User face has not yet been captured

In Figure 5-17, we see that HD Face is Ready To Start Capture. Pressing Start Capture will fire off FaceModelBuilder, which will instruct the user in which direction to tilt their face in the Builder Status Text Block.

In the InitializeHDFace() method of MainWindow.xaml.cs, the process of creating a 3D mesh is started off with the InitializeMesh() and UpdateMesh() methods. These methods take the vertices, create triangles from them, and then apply them to a MeshGeometry3D object that is responsible for the 3D visualization we see on the front end. MeshGeometry3D is a part of the System.Windows.Media.Media3D namespace that allows us to build simple 3D applications in WPF (the functionalities of this library are sometimes colloquially referred to as WPF 3D).

Listing 5-38. Initializing a 3D Mesh from HD Face Data

private void InitializeMesh()
{
    var vertices = this.currentFaceModel.CalculateVerticesForAlignment(this.currentFaceAlignment);

    var triangleIndices = this.currentFaceModel.TriangleIndices;

    var indices = new Int32Collection(triangleIndices.Count);

    for (int i = 0; i < triangleIndices.Count; i += 3)
    {
        uint index01 = triangleIndices[i];
        uint index02 = triangleIndices[i + 1];
        uint index03 = triangleIndices[i + 2];

        indices.Add((int)index03);
        indices.Add((int)index02);
        indices.Add((int)index01);
    }

    this.theGeometry.TriangleIndices = indices;
    this.theGeometry.Normals = null;
    this.theGeometry.Positions = new Point3DCollection();
    this.theGeometry.TextureCoordinates = new PointCollection();

    foreach (var vert in vertices)
    {
        this.theGeometry.Positions.Add(new Point3D(vert.X, vert.Y, -vert.Z));
        this.theGeometry.TextureCoordinates.Add(new Point());
    }
}

In Listing 5-38, we have all the code necessary to turn vertices into a 3D mesh. After grabbing the vertices and triangle indices of the standard model face, we add the triangle indices to an Int32Collection for use by a MeshGeometry3D instance. This will indicate which vertices create which triangles .

When adding the triangle indices to the Int32Collection, we add them in reverse order (i.e., index03, index02, index01). This is because WPF 3D creates front-facing mesh triangles in counterclockwise order. Setting them in clockwise would make the mesh triangles face backward.

After setting the indices to the MeshGeometry3D (known here as theGeometry and initialized in the XAML code), we add the vertices. The z coordinate of the vertices is made negative so that the face appears behind the screen, as the positive z-axis sticks out of the screen. TextureCoordinates are set as null values, as we do not intend to texture our face.

Listing 5-39. Updating a 3D Mesh from HD Face Data

private void UpdateMesh()
{
    var vertices = this.currentFaceModel.CalculateVerticesForAlignment(this.currentFaceAlignment);

    for (int i = 0; i < vertices.Count; i++)
    {
        var vert = vertices[i];
        this.theGeometry.Positions[i] = new Point3D(vert.X, vert.Y, -vert.Z);
    }
}

Since the triangle indices are already set into the MeshGeometry3D, we only need to update the position of the vertices over time. To accomplish this, UpdateMesh() (Listing 5-39) is called every time HighDefinitionFaceFrameReader provides a new frame.

Listing 5-40. Initializing FaceModelBuilder

private void StartCapture()
{
    this.StopFaceCapture();

    this.faceModelBuilder = null;

    this.faceModelBuilder = this.highDefinitionFaceFrameSource.OpenModelBuilder(FaceModelBuilderAttributes.None);

    this.faceModelBuilder.BeginFaceDataCollection();

    this.faceModelBuilder.CollectionCompleted += this.HdFaceBuilder_CollectionCompleted;
}

In Listing 5-40, the StartCapture() method encapsulates the initializing process of FaceModelBuilder. StopFaceCapture() disposes of any existing FaceModelBuilder so that we can start fresh. We assign the FaceModelBuilder by opening it from the HighDefinitionFaceFrameSource’s OpenModelBuilder(FaceModelBuilderAttribute enabledAttributes) method. The other parameters we can include are HairColor and SkinColor. If we wanted both, we could do OpenModelBuilder(FaceModelBuilderAttribute.HairColor | FaceModelBuilderAttribute.SkinColor).

The FaceModelBulder is started with its BeginFaceDataCollection() method, and we attach an event handler to its CollectionCompleted event so that we can tell it to deform the face after the new FaceModel has been created from it .

Listing 5-41. Creating a New FaceModel from FaceModelBuilder’s Analysis

private void HdFaceBuilder_CollectionCompleted(object sender, FaceModelBuilderCollectionCompletedEventArgs e)
{
    var modelData = e.ModelData;

    this.currentFaceModel = modelData.ProduceFaceModel();

    this.faceModelBuilder.Dispose();
    this.faceModelBuilder = null;
                                                                                          
    this.CurrentBuilderStatus = "Capture Complete";
}

Once the FaceModelBuilder has finished capturing all its requisite frames (Listing 5-41), it provides us with a FaceModelData object whose ProduceFaceModel() method creates the more tailored FaceModel that we desire. It will be used by UpdateMesh() automatically to alter the mesh.

Listing 5-42. Determining FaceModelBuilder’s Status

private void CheckOnBuilderStatus()
{
    if (this.faceModelBuilder == null)
    {
        return;
    }

    string newStatus = string.Empty;

    var captureStatus = this.faceModelBuilder.CaptureStatus;
    newStatus += captureStatus.ToString();

    var collectionStatus = this.faceModelBuilder.CollectionStatus;

    newStatus += ", " + GetCollectionStatusText(collectionStatus);

    this.CurrentBuilderStatus = newStatus;
}

Two final interesting bits of FaceModelBuilder functionality are its CaptureStatus and CollectionStatus properties (Listing 5-42). They provide capture-quality warnings and details as to what frames are left to capture, respectively. The FaceModelBuilderCaptureStatus enum contains the OtherViewsNeeded, LostFaceTrack, FaceTooFar, FaceTooNear, GoodFrameCapture, and SystemError values. The FaceModelBuilderCollectionStatus enum contains the MoreFramesNeeded, FrontViewFramesNeeded, LeftViewsNeeded, TiltedUpViewsNeeded, and Complete values. These can be used to provide tips to users on how they should position their faces to complete face capture .