Taking a Screenshot

Some of you have probably already thought of ways to grab a screenshot from the application. If not, take a couple of moments to think through how you could potentially do this. It seems simple enough, and there is more than one right way to achieve the desired result. The most common first guesses on how to do this are the following: grabbing and saving data from a frame, taking a snapshot of our UI, or saving our WriteableBitmap. Out of these options, the third one is the practical one and is how Microsoft does it in their samples.

Grabbing and saving from a frame would not be ideal, at least not with how our application is currently structured. When the screenshot button is clicked on the UI, an event is raised, and we manage to encode a picture file in its handler. We would have no convenient access to the ColorFrameSource in there, as all that logic and data is located in the ColorFrame event handler. Even if we did, there is no guarantee that there is a frame available for us to pick from at the instant the screenshot button is clicked. This is where polling could have been useful had we not gone with an event-driven model. We could poll for a frame to use as the basis for our screenshot. We could choose to cache a frame in the event handler and call on that, but seeing as our image already has a copy of the data, that would be redundant.

While it is possible with WPF to take a snapshot of our UI element, it is much more straightforward to simply save an image from its original source, the WriteableBitmap. It is already a bitmap, so we can encode it into a picture file with relative ease.

The code for adding screenshot functionality requires very few modifications to the application. Specifically, we will be adding a namespace reference, an event handler method , as shown in Listing 3-5, and a button on the UI, as shown in Listing 3-6. The namespace that we need to add is System.IO, which will provide us with methods with which to save an image file to the drive.

Listing 3-5. MainWindow.xaml.cs Screenshot Event Handler

//add to the Using section
using System.IO;                                                                                                                  

//Directly after the MainWindow_Closing method
private void ScreenshotButton_Click(object sender, RoutedEventArgs e)
{
    if (this.colorBitmap != null)
    {

        BitmapEncoder encoder = new PngBitmapEncoder();

        encoder.Frames.Add(BitmapFrame.Create(this.colorBitmap));

        string photoLoc = Environment.GetFolderPath(Environment.SpecialFolder.MyPictures);

        string time = System.DateTime.Now.ToString("d MMM yyyy hh-mm-ss");

        string path = Path.Combine(photoLoc, "Screenshot " + time + ".png");

        try
        {
            using (FileStream fs = new FileStream(path, FileMode.Create))
            {
                encoder.Save(fs);
            }
            //provide some confirmation to user that picture was taken
        }
        catch (IOException)
        {
            //inform user that picture failed to save for whatever reason
        }
    }
}

None of the code involved in capturing and saving an image is from the Kinect API. We are using code from the System.Windows.Media.Imaging and System.IO namespaces. After checking whether our WriteableBitmap has any data in it, we create a new PngBitmapEncoder, to which we add our WriteableBitmap as a BitmapFrame in encoder.Frames.Add(BitmapFrame.Create(this.colorBitmap));.

After grabbing the location of our My Pictures folder and a unique timestamp to discourage our screenshot from overwriting prior ones, our encoder saves the screenshot to our desired path using .NET’s FileStream class. As with any input/output operation, this is wrapped in a try/catch block to handle the casual error (e.g., lack of disk space, no write access, and so on).

In MainWindow.xaml we will add the XAML for the Button UI element directly after the Viewbox UI element but inside the Grid UI element with which we will take our screenshots.

Listing 3-6. MainWindow.xaml Screenshot Button Markup

<Grid Margin="10">
  ...
  </Viewbox>

  <Button Grid.Row="2" Content="Screenshot" Height="Auto" HorizontalAlignment="Left" VerticalAlignment="Center" Margin="10" Click="ScreenshotButton_Click" />

</Grid>

Most of the Button attributes are merely decorative. The important thing is to assign the event handler from our code-behind for when the button is clicked, which we accomplish with Click="ScreenshotButton_Click".

If you open our app now, you should be able to click the Screenshot button. There’s no picture-taken confirmation (though you can easily include this), so to find out if it worked, go visit your My Pictures folder. Now you can go share your beautiful Kinect picture on social media or the like. (Maybe on Snapchat. . . . oh wait, never mind; I forgot they took down 6snap.)

Taking Another Screenshot

Windows Store Apps also have a different set of APIs to handle files and image encoding, so the code for taking screenshots is different as well. We will be saving images to the My Pictures folder, so you have to enable the Pictures Library permission in the app manifest, as depicted in Figure 3-2. We need to include a few more namespaces, namely System, System.Runtime.InteropServices.WindowsRuntime, Windows.Graphics.Imaging, Windows.Storage, and Windows.Storage.Streams.

As we did with the WPF example, add a Button element to the XAML code and give it an event handler for when it is clicked. You might need to change the color of the button to make it appear, as sometimes it shares the same color as the background. The rest of the code alterations take place in the event-handler body, as shown in Listing 3-9.

Listing 3-9. MainPage.xaml.cs Screenshot Event Handler in a Windows Store App

private async void ScreenshotButton_Click(object sender, RoutedEventArgs e)
{

    StorageFolder picFolder = KnownFolders.PicturesLibrary;

    StorageFile picFile = await picFolder.CreateFileAsync("Kinect Screenshot.png",
        CreationCollisionOption.GenerateUniqueName);

    using (IRandomAccessStream stream = await picFile.OpenAsync(FileAccessMode.ReadWrite))
    {
        var encoder = await BitmapEncoder.CreateAsync(BitmapEncoder.PngEncoderId, stream);

        Stream pixelStream = colorBitmap.PixelBuffer.AsStream();
        byte[] pixels = new byte[pixelStream.Length];
        await pixelStream.ReadyAsync(pixels, 0, pixels.Length);

        encoder.SetPixelData(BitmapPixelFormat.Bgra8,
            BitmapAlphaMode.Straight,
            (uint)colorBitmap.PixelWidth,
            (uint)colorBitmap.PixelHeight,
            96.0, 96.0,
            pixels);

        await encoder.FlushAsync();
    }
}

We have made a small edit to the event handler method header. We have added the async keyword to modify the method and let the compiler know that there is asynchronous code in its body. Our input and output operations need to be done asynchronously so as to not lock up the UI thread while in operation. Within the method, we first find the My Pictures folder and create a PNG image within it. The CreateFileAsync method conveniently handles collisions as we desire, so we tell it to make unique files each time instead of having ourselves add unique timestamps programmatically.

We create a stream with the PNG file and a PNG BitmapEncoder to write to it. We create another stream called pixelStream from our WriteableBitmap’s data buffer and use it to write to an array of bytes. Then, the encoder sets the pixel data of our PNG using this array. The BitmapAlphaMode input in the SetPixelData method refers to how we want to save the alpha values to our PNG. Sometimes our final image is in a format that does not measure opacity, so we can choose to ignore or apply opacity to the color channels by multiplying the color values by the alphas. We will simply use the alphas that we already have in hand, thus the Straight option. The two 96.0 values are the horizontal and vertical dpis, respectively. We call the FlushAsync method of encoder to finalize the image operations. After it is called, we can make no more alterations to the image. Your Windows Store App should now be able to take pictures just as well as the WPF application did.

Displaying ColorFrameSource Data with a Byte Array

Starting with the Kinect camera-viewing application we just built in the previous section, we will make some adjustments to obtain the image data inside of a byte array instead of a buffer.

Listing 3-10. MainPage.xaml.cs Variable Initialization for Color Data with Use of Byte Array

public sealed partial class MainPage : Page
{
    private KinectSensor kinect = null;
    private ColorFrameReader colorFrameReader = null;
    private WriteableBitmap colorBitmap = null;

    private readonly uint bytesPerPixel;                
    private byte[] colorPixels = null;                
    ...

In Listing 3-10, we see the variable initialization stage of our application. KinectSensor, ColorFrameReader, and WriteableBitmap all make a return. The structure of the code is still the same. What will change is how WriteableBitmap reads the data from our color frames. Previously, we could copy our data directly to the WriteableBitmap’s back buffer/pixel buffer, with an IntPtr or IBuffer provided by the Kinect API. Instead, we will now provide an array of bytes. colorPixels will be this array. Earlier, I mentioned that RGBA consisted of 3 bytes of color data + 1 byte of alpha opacity per pixel, for a total of 4 bytes per pixel. bytesPerPixel will hold this value as garnered from the ColorFrameSource. We could have resorted to using the literal four instead of creating a whole variable, but since the bytes per pixel change based on the color format, we can set it up so that we do not have to manually switch it when we change the format.

Seeing as we introduced two new variables, the bootstrap process is different as well (Listing 3-11).

Listing 3-11. MainPage.xaml.cs Bootstrapping and Variable Assignment for Color Data with Use of Byte Array

public MainPage()
{
    this.kinect = KinectSensor.GetDefault();
    this.colorFrameReader = this.kinect.ColorFrameSource.OpenReader();

    this.colorFrameReader.FrameArrived += this.Reader_ColorFrameArrived;

    FrameDescription colorFrameDescription = this.kinect.ColorFrameSource.CreateFrameDescription(ColorImageFormat.Rgba);

    this.bytesPerPixel = colorFrameDescription.BytesPerPixel;                

    this.colorPixels = new byte[colorFrameDescription.Width * colorFrameDescription.Height * this.bytesPerPixel];                

    this.colorBitmap = new                                       WriteableBitmap(colorFrameDescription.Width, colorFrameDescription.Height);

    this.kinect.Open();

    this.DataContext = this;
    this.InitializeComponent();
}
...

In Listing 3-11, we added two new lines preparing the colorPixels data array. Since colorPixels will hold the bytes of every pixel in our image, we are declaring its total size as the number of bytes per pixel for the relevant format multiplied by the total number of pixels. We get the bytes per pixel directly from the ColorFrameDescription, which takes into account the image format that we chose (RGBA in our case).

The rest of the code modification is in the event handler for the color frames (Listing 3-12).

Listing 3-12. MainPage.xaml.cs ColorFrameReader Event Handler for Direct Access to Pixel Data

private void Reader_ColorFrameArrived(object sender, ColorFrameArrivedEventArgs e)
{
    using (ColorFrame colorFrame = e.FrameReference.AcquireFrame())
    {
        if (colorFrame != null)
        {
            FrameDescription colorFrameDescription = colorFrame.FrameDescription;

            if ((colorFrameDescription.Width == this.colorBitmap.PixelWidth) && (colorFrameDescription.Height == this.colorBitmap.PixelHeight))
            {
               if (colorFrame.RawColorImageFormat == ColorImageFormat.Bgra)
                {
                    colorFrame.CopyRawFrameDataToArray(                
                    this.colorPixels);                    
                }
                else
                {
                                                                                                            colorFrame.CopyConvertedFrameDataToArray(                
                    this.colorPixels, ColorImageFormat.Bgra);                    
                }

                Stream pixelStream = colorBitmap.PixelBuffer.AsStream();                
                pixelStream.Seek(0, SeekOrigin.Begin);                
                pixelStream.Write(colorPixels, 0, colorPixels.Length);                

                colorFrameProcessed = true;
            }
        }
    }

    if (colorFrameProcessed)
    {
        this.colorBitmap.Invalidate();
    }
}

Instead of the CopyRawFrameDataToBuffer and CopyConvertedFrameDataToBuffer methods we used earlier, we are using their byte-array equivalents, CopyRawFrameDataToArray and CopyConvertedFrameDataToArray, respectively. Once the pixel data is copied to the arrays, we write to the WriteableBitmap in way a similar to that used when we read from it in the screenshot code: we create a stream from its pixel buffer. The Seek method of our Stream pixelStream is used to set the position of where we will write in the stream. The SeekOrigin.Begin input sets it to the start of the stream, and the first input, 0, is the offset from the beginning position where we want to start. In the Write method of Stream, we specify that we want to write with the contents of the byte array colorPixels for the entire length of the array. Again, the 0 here is also the offset.

For now, the rest of the code is the same. Make sure that the System.IO namespace is included so that we can use the Stream class. Your program should run just as if you had used a buffer to write to WriteableBitmap.

Manipulating Image Pixel Data

We only really benefit from having the image data available as a byte array if we plan to manipulate it. Let us make a very simple alteration. As mentioned earlier, each pixel has three color channels—Blue, Green and Red–with 8 bits of intensity, or 256 different values. We will be making a filter that hides the green and blue channels of the image, meaning we will set their respective intensities to 0.

In our Reader_ColorFrameArrived event handler, our code will find itself in the lines right after ColorFrame data has been copied to an array and right before that array is used by a stream to write to the WriteableBitmap (Listing 3-13).

Listing 3-13. MainPage.xaml.cs ColorFrameReader Event Handler Manipulating Color Data Byte Array Values

...
colorFrame.CopyConvertedFrameDataToArray(this.colorPixels, ColorImageFormat.Bgra);
}

for (uint i = 0; i < colorPixels.Length; i += bytesPerPixel)                    
{                    
    colorPixels[i] = 0x00;                
    colorPixels[i + 1] = 0x00;                
}                    

Stream pixelStream = colorBitmap.PixelBuffer.AsStream();
...

In Listing 3-13, the for loop cycles through each pixel in the image and sets the first two channels, Blue and Green, to 0 intensity. We are iterating by the value of bytesPerPixel, which is 4, because the array’s indices contain bytes, 4 for each pixel. When we go through the first 4 bytes, we have covered the first pixel and applied the desired operations to its color channels. Since we are manipulating bytes, we set the pixel color channel intensity values using hexadecimal notation. Hence, when we say colorPixels[i + 1] = 0x00, we are saying the green channel (the second channel in BGRA) has an intensity of 0.

Compile the program, and you should see nothing but red throughout the image (Figure 3-3).

Figure 3-3. Redmond bear shows his true colors

On my i5 Surface Pro 2, tinting the picture red resulted in a 7 to 10 point higher reading for CPU usage in Task Manager. Generally, we will want to avoid doing heavy image processing in WPF without C++ interop because of performance issues. After all, we are looping through and applying operations to over 2 million pixels, 30 times a second! You can compile and run your program in a Release configuration to improve performance.

Color Gradient Depth Feed

We are not obliged to use a mundane gray to display our depth feed. On the Internet or in demo apps such as Kinect Evolution you will often see the depth feed shaded with various colors. We can choose to do that too.

A common depth feed coloring technique is the color gradient. Right now, our depth feed has a gradient going from black to white. We can change this so that it goes from two or more different colors to white. The first step to achieving this is to change our color format from Gray8 to a more appropriate format. We can choose anything, but Bgra32 is a good bet for any basic color work.

We start by changing the format of our WriteableBitmap and the size of the byte array that will render data to it (Listing 3-17).

Listing 3-17. Variable Declaration for Colorized Depth Feed

//Inside the MainWindow() Constructor

public MainWindow() {
// ...

this.depthPixels = new byte[this.depthFrameDescription.Width * this.depthFrameDescription.Height * 4];
this.depthBitmap = new WriteableBitmap(this.depthFrameDesription.Width, this.depthFrameDescription.Height, 96.0, 96.0, PixelFormats.Bgra32, null);

// ...

The depthPixels array’s size must be increased to accommodate the bytes from the additional color channels it will now hold for each pixel. We multiply the total number of pixels by 4, as we have 4 bytes of color data per pixel. Remember, Bgra32 is 32 bits of color data, which is equivalent to 4 bytes.

Inside our ProcessDepthFrameData method from earlier, we will have to change how we put data into our depthPixels array, as it is now four times the size (Listing 3-18).

Listing 3-18. Processing Depth Data for Bgra32 WriteableBitmaps

private unsafe void ProcessDepthFrameData(IntPtr depthFrameData, uint depthFrameDataSize, ushort minDepth, ushort maxDepth)
{

    ushort* frameData = (ushort*)depthFrameData;

    int colorByteIndex = 0;                

    for (int i = 0; i < (int)(depthFrameDataSize / this.depthFrameDescription.BytesPerPixel); ++i)
    {

        ushort depth = frameData[i];

          // Set Blue Channel
        this.depthPixels[colorByteIndex++] = (byte)(depth >= minDepth && depth <= maxDepth ? depth : 0);                

          // Set Green Channel
        this.depthPixels[colorByteIndex++] = (byte)(depth >= minDepth && depth <= maxDepth ? depth : 0);                

          // Set Red Channel
        this.depthPixels[colorByteIndex++] = 0;                

          // Set Alpha /transparent Channel
        this.depthPixels[colorByteIndex++] = 255;                

    }
}                                                                                                                 

The same line is repeated twice—is that a printing error? Not quite. Since we have three color channels, we have to set the color value for each of them every time we iterate a single pixel. colorByteIndex tracks which channel we are inspecting in the depthPixels array. The fourth channel, the one for alpha, is always set to 255, the highest 8-bit value, because we always want full opacity. Before we try to get a color gradient between multiple colors for our image, let us get it working for one color. We set the red channel to 0 so that we can get different shades of cyan as the color of our depth feed.

The one last change needed in order to get our depth feed displaying data through Bgra32 must be made in the method so as to render the data to the WriteableBitmap (Listing 3-19).

Listing 3-19. WriteableBitmap’s WritePixel Method for Bgra32

// Inside the Reader_DepthFrameArrived() {
// ...

this.depthBitmap.WritePixels(
    New Int32Rect(0, 0, this.depthBitmap.PixelWidth, this.depthBitmap.PixelHeight),
        this.depthPixels,
        this.depthBitmap.PixelWidth * 4,                
        0);

// ...

The penultimate input for this particular configuration of the WritePixels method is the image’s stride. We must multiply by 4 since we now have 4 bytes per pixel worth of data. Note that we use the depthBitmap’s PixelWidth property and not its Width property, as the latter does not give us the width of the picture in pixels.

Compile and run the program when you have finished, and you should get something like Figure 3-7, but in cyan instead of magenta.

Figure 3-7. The author’s living room in a colorized depth feed. Magenta was obtained instead of cyan by setting the first and third color channels (Blue and Red, respectively) of each pixel with the depth value and setting the second color channel (Green) as 0.

Wait, what is this? What are the black bands repeated throughout the image? Keen readers will have noticed that I omitted the MapDepthToByte divisor from the depth-intensity calculation. Since our ushort depth value is not normalized, the color intensity goes from black to magenta every 0mm to 255mm and then restarts. This is because when the ushort value is converted to a byte value, we truncate it and take the last 8 bits of its 16-bit value. While this can make clustered objects harder to see, when compared to our 8-bit grayscale image in Figure 3-6, it definitely makes the shape of the room easier to visualize. The back wall is clearer, along with the curtains and the rear sofa. Additionally, we can make out the contours of some of the stickers on the HP printer box at the bottom left of the image. Unfortunately, it does give a bit more of a damned vibe to my living room.

Let us now apply the algorithm to get our depth feed mapped to a color gradient (Listing 3-20).

Listing 3-20. Depth Data Processed with a Color Gradient

private unsafe void ProcessDepthFrameData(IntPtr depthFrameData, uint depthFrameDataSize, ushort minDepth, ushort maxDepth)
{

    ushort* frameData = (ushort*)depthFrameData;

    int colorByteIndex = 0;                

    for (int i = 0; i < (int)(depthFrameDataSize / this.depthFrameDescription.BytesPerPixel); ++i)
    {

        ushort depth = frameData[i];                                                                                                              
        float depthPercentage = (depth / 8000f);                

        this.depthPixels[colorByteIndex++] = 0;                
        this.depthPixels[colorByteIndex++] = (byte)(depth >= minDepth && depth <= maxDepth ? (depthPercentage) * 255 + (1 – depthPercentage) * 0 : 0);                
        this.depthPixels[colorByteIndex++] = (byte)(depth >= minDepth && depth <= maxDepth ? (depthPercentage) * 0 + (1 – depthPercentage) * 255 : 0);                
        this.depthPixels[colorByteIndex++] = 255;

    }
}

The algorithm to apply a color gradient to our depth feed is not particularly difficult. We are trying to go from one color to another color, so as the depth increases, we use more of the color for the maximum depth and a smaller percentage of the color we started from. Since our colors are measured with three color channels, we do this for the intensity of each channel using the RGB component of the desired colors. This can be described by the following formula (using the Green channel as an example):

• Green = G component of Color1 × Depth Percentage + G component of Color2 × (1– Depth Percentage)

In Listing 3-20, the colors I went for were Green and Red, whose BGR components are 0, 255, 0 and 0, 0, 255, respectively. The Blue channel is not used, so I set it to 0 from the get-go. When depthPercentage, which is a measure of how close we are to 8m (the official limit of the Kinect’s depth-sensing abilities), reaches 1, we want the depth to be colored green. Looking at the formula for the Green channel, we thus have 1 x 255 + (1 – 1) x 0 = 255. If we were at 4m, we would expect 50 percent green (and 50 percent red), so the formula for green would result in 0.5 x 255 + (1 – 0.5) x 0 = 127.5, which is rounded to 127. The resulting color gradient depth feed should look like that seen in Figure 3-8.

Figure 3-8. The depth feed under a Red-Green color gradient

We use colors in other ways to help us visualize the depth of a scene. Instead of having a linear gradient where we go from one color to another gradually, we can choose to have a handful of colors represent different ranges of depth. We could choose yellow for anything between 1 and 2 meters, green for anything between 2 and 3 meters, and so on. This would provide a legend as to where objects are positioned in a scene, rather than saying “it is kind of green and kind of red.” Fortunately, this is not difficult at all. It is simply a matter of making a handful of if statements for each color region (Listing 3-21).

Listing 3-21. Visualizing Depth Data with Color Regions

// Inside the ProcessDepthFrameData() {
    ...
// Replace all the logic inside the method with the following:
    ushort* frameData = (ushort*)depthFrameData;

    int colorByteIndex = 0;

    Color color = new Color();                

    for (int i = 0; i < (int)(depthFrameDataSize / this.depthFrameDescription.BytesPerPixel); ++i)
    {

        ushort depth = frameData[i];
        float depthPercentage = (depth / 8000f);

        if (depthPercentage <= 0) {
            color.B = 0;
            color.G = 0;
            color.R = 0;
        }
        if (depthPercentage > 0f & depthPercentage <= 0.2f) {
            color.B = 255;
            color.G = 0;
            color.R = 0;
        }
        if (depthPercentage > 0.2f & depthPercentage <= 0.4f) {
            color.B = 0;
            color.G = 255;
            color.R = 0;                                                                                                              
        }
        if (depthPercentage > 0.4f & depthPercentage <= 0.6f) {
            color.B = 0;
            color.G = 0;
            color.R = 255;
        }
        if (depthPercentage >= 0.6f) {
            color.B = 0;
            color.G = 255;
            color.R = 255;
        }

        this.depthPixels[colorByteIndex++] = color.B;
        this.depthPixels[colorByteIndex++] = color.G;
        this.depthPixels[colorByteIndex++] = color.R;
        this.depthPixels[colorByteIndex++] = 255;

    }
// ...

In Listing 3-21, we made use of WPF’s Color structure to keep track of our desired colors for the various regions. Each Color object has properties for Blue, Green, Red, and Alpha values that we can set. Our series of conditional statements simply determines which depth region a pixel falls in and then assigns its color. For example, in Figure 3-9, if our depth value for a pixel is between 20 percent and 40 percent of the maximum possible value of 8000mm, we set it as green. Looking at the image, we can instantly determine that any area colored green is between 1.6m and 3.2m. Of course, we could have picked smaller ranges and added more colors to make it even more practical. If you are measuring anything with this method, remember to take into account the fact that distances are measured from the camera’s plane. This means measuring by holding a ruler or tape perpendicular to the Kinect’s face. In Figure 3-10, you can see this imaginary plane, which is created by the x and y axes.

Figure 3-9. The author’s living room in bands of 1.6m (yellow is anything farther than 4.8m)

Figure 3-10. The camera measures depth perpendicular from the x–y plane, on the z-axis

Determining the Depth of a Pixel

You might be wondering at this point why, if we can get numerical depth measurements, are we wasting our time with colors? Fair enough. I do have to meet a page quota to have this book published, so bear with me. Let us make a tool that will allow us to determine the depth in millimeters of a particular pixel in an image.

Before I give you the code, let us consider the fact that the Kinect camera’s field of view, like most other cameras, is shaped like a pyramid, as in Figure 3-11. This means that if the Kinect sees a surface near it, the depth points represented in 3D space by the pixels are more packed together than if the Kinect sees a surface farther away. The x and y coordinates of our image therefore do not correspond to heights and widths in the real world.

Figure 3-11. The Kinect’s field of view. The blue 4 × 4 pixel grid closer to the Kinect is more densely packed than the 4 × 4 pixel grid farther from the Kinect. In reality, the Kinect has way more pixels, but a 4 × 4 grid was arbitrarily picked for demonstration purposes.

Since we want to access depth data outside of the DepthFrameReader event handler on a mouse click, we will have to store the depth data in some type of data structure. We could take the data from the byte array depthPixels that we used to write to our WriteableBitmap, but depth data is 16-bits, so we need something larger. We could use a Gray16 image format, or hold it in 2 of the 4 bytes in each RGBA color channel, but we will not bother with the hassle and instead will create another array to hold the depth data (Listing 3-22).

Listing 3-22. Getting Depth Distance for a Pixel on Mouse Click

...
//among other private variables such as WriteableBitmap depthBitmap
private int[] depthData = null;                    
...

public MainWindow()
{
    ...
    //other variables assigned here
    this.depthData = new int[this.depthFrameDescription.Width * this.depthFrameDescription.Height];                
    ...
}
private unsafe void ProcessDepthFrameData(IntPtr depthFrameData, uint depthFrameDataSize, ushort minDepth, ushort maxDepth)
{
    ushort* frameData = (ushort*)depthFrameData;

    int index = 0;

    for (int i = 0; i <(int)(depthFrameDataSize / this.depthFrameDescription.BytesPerPixel); ++i)
    {
        ushort depth = frameData[i];

        this.depthData[i] = depth;                

        this.depthPixels[index++] = (byte)(depth >= minDepth && depth <= maxDepth ? depth : 0);
        this.depthPixels[index++] = (byte)(depth >= minDepth && depth <= maxDepth ? depth / MapDepthToByte : 0);
        this.depthPixels[index++] = (byte)(depth >= minDepth && depth <= maxDepth ? depth / MapDepthToByte : 0);

        this.depthPixels[index++] = 255;
    }
}
                                                                                          
private void Image_MouseDown(object sender, System.Windows.Input.MouseEventArgs e)                    
{                    
    Point p = e.GetPosition(DepthImage);                

    int index = ((int)p.X + ((int)p.Y * depthBitmap.PixelWidth));                
    int depth = depthData[index];                

    ToolTip toolTip = new ToolTip { Content = depth + "mm", FontSize = 36, IsOpen = true, StaysOpen = false };                
    DepthImage.ToolTip = toolTip;                

}                    

We added an array of integers, depthData, to cache the depth for each pixel in the image every time we have a new frame. The depth data processing code was barely modified to accommodate this. We simply added a line, this.depthData[i] = depth;, to have it collect the depth every time we process a new pixel. An event handler called Image_MouseDown was added to be called whenever a pixel in our image is clicked. It gets the x and y coordinates of the pixel that was clicked, and we use it to find the corresponding pixel in our integer array. Our integer array is one-dimensional, whereas the picture coordinates are two-dimensional. The integer array stores each row of the image one after another, so we can multiply the y coordinate by the width of the image in pixels to find out how far into the integer array to go to get the desired depth data. To better imagine this, pretend our image is a 10-pixel by 10-pixel bitmap and we want to find the depth of the pixel at column 7 and row 5. The size of our integer array is 100. Counting from the top left and starting at 0, we get the pixel number as 5 x 10 + 6, or 56. The corresponding depth value in our integer array would be at index 56 as well. We start at 0, of course, as C# arrays start their indexes at 0.

To display the data, we create a tooltip UI element that pops up wherever on the image we clicked.

On the front end, we need to modify our Image tag to support the mouse-clicked event (Listing 3-23).

Listing 3-23. Image XAML for Getting Depth Distance of a Pixel on Mouse Click

<Image Name="DepthImage" Source="{Binding ImageSource}" Stretch="UniformToFill" MouseDown="Image_MouseDown" />

In addition to adding the MouseDown event-handler tag for when the mouse is clicked, we also added the name DepthImage so that our event handler knows to which image to add its tooltip. After compiling the program, click on any pixel on the image to find out how far it is from the Kinect’s plane, as demonstrated in Figure 3-12.

Figure 3-12. Distance from the Kinect to the couch. The pixel that was clicked is located at the top-right corner of the tooltip.

Keen observers will again have noted why the image gets progressively yellower further in. In the code to set the intensity for each color channel, we divided the depth values for the Green and Red channels by the MapDepthToByte value, which we did not do for the Blue channel. Thus, the Blue channel has the wrapping effect throughout the image, whereas the Green and Red do not. If you have memorized some RGB values by now, you will know that Red + Green = Yellow.