Chapter 10. Controls for Kinect

As you saw in Chapter 9 it is important to adapt the user interface (UI) for your application to Kinect. You can use a Kinect sensor to control a standard mouse-oriented application, but it is far better to create an original Kinect-oriented application to take advantage of the unique characteristics of the sensor.

In this chapter, you will write a basic application designed to be controlled with a mouse or a Kinect sensor. As we clearly are in a transition period, it is important to support mouse-based UIs when possible, because this provides the ability to work around some of the difficulties of using Kinect (precision, user acceptance and understanding, etc.).

But if you want to create a complete Kinect user interface, do not hesitate to do so. Just be sure to take time to think about the usability and the discoverability of your user experience. Make sure that your user interface is as simple to use as a mouse. For now, users are accustomed to clicking a button or scrolling up and down using the mouse wheel. Because they are not currently aware of the conventions for using a Kinect sensor, you must make sure the controls are intuitive and easily understandable by users.

Your user interface must be natural. With a Kinect sensor, you remove the user’s physical contact with the computer, so the experience must follow the natural habits of those users. For instance, if you want to control horizontal scrolling, it is most natural for the user to grab the current page and move it to the right or to the left with his hands to achieve that movement. Think about the most intuitive ways to accomplish an action with the user’s body movements and gestures when you are designing the user interface.

Adapting the size of the elements

In the applications you create for Kinect, the best idea is to start by tracking the position of the hands of the user. Humans always use their hands to control the environment they are in or to support what they are saying, so it is natural for a user to choose to control the application with her hands.

This choice implies that the user wants to control a kind of mouse pointer with his or her dominant hand, for instance. This cursor will benefit from the same smoothing algorithm introduced in Chapter 9 to filter out the jitter—the Holt Double Exponential Smoothing filter. But you must also consider that even using the best smoothing algorithm, the precision of the movements of the cursor when controlled by the Kinect sensor is not the same as when using a true mouse. So you have to create a user interface with larger controls, as shown in Figure 10-1.

A sample user interface adapted for Kinect.

Figure 10-1. A sample user interface adapted for Kinect.

As you can see, the Kinect cursor (the arrow pointing up near Option 3) is bigger than your standard mouse pointer and the controls are larger to give the user more room to click them.

Providing specific feedback control

Even before you begin working on your user experience with Kinect, you must provide a visual feedback of the view of the user captured by the sensor.

This graphic feedback has been the subject of code created in previous chapters (such as Chapter 3), but for this application, you will write a reusable control based on the depth stream and the position of the user’s hands. Using this control in your application will provide an immediate visual feedback for the user to view what the Kinect sensor is seeing and to determine if it is able to track the user’s hands properly.

Figure 10-1 includes the visual feedback from this control as a frame inside the user interface for the sample application.

To create the visual feedback, first you must create a user control called PresenceControl with the following XAML:

<UserControl x:Class="Kinect.Toolbox.PresenceControl"
             xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
             xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
             xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
             xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
             mc:Ignorable="d"
             d:DesignHeight="300" d:DesignWidth="300">
    <Grid>
        <Grid.Resources>
            <RadialGradientBrush x:Key="radial" GradientOrigin="0.5, 0.5">
                <RadialGradientBrush.GradientStops>
                    <GradientStop Color="White" Offset="0"/>
                    <GradientStop Color="White" Offset="0.5"/>
                    <GradientStop Color="Transparent" Offset="1.0"/>
                </RadialGradientBrush.GradientStops>
            </RadialGradientBrush>
        </Grid.Resources>
        <Image x:Name="image"/>
        <Ellipse x:Name="leftEllipse" Opacity="0.6" Width="50" Height="50" 
VerticalAlignment="Top" HorizontalAlignment="Left" Margin="-25,-25,0,0"
Fill="{StaticResource radial}">
            <Ellipse.RenderTransform>
                <TranslateTransform x:Name="leftTransform"/>
            </Ellipse.RenderTransform>
        </Ellipse>
        <Ellipse x:Name="rightEllipse" Opacity="0.6" Width="50" Height="50"
VerticalAlignment="Top" HorizontalAlignment="Left" Margin="-25,-25,0,0"
Fill="{StaticResource radial}">
            <Ellipse.RenderTransform>
                <TranslateTransform x:Name="rightTransform"/>
            </Ellipse.RenderTransform>
        </Ellipse>
    </Grid>
</UserControl>

This control is based mainly on an image that is updated with the depth stream and two ellipses that move to indicate where the hands of the user should be positioned (you can see the ellipses as two light areas near the hands in the image shown at the bottom of Figure 10-1). The ellipses are filled with a radial gradient brush to simulate small glowing spots where the hands belong.

To update the image, you have to handle the DepthFrameReady event, and to update the position of the ellipses, you have to handle the SkeletonFrameReady event (as you did in the Chapter 3):

using System.Linq;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using Microsoft.Kinect;

namespace Kinect.Toolbox
{
    public partial class PresenceControl : UserControl
    {
        KinectSensor kinectSensor;
        byte[] depthFrame32;
        short[] pixelData;
        WriteableBitmap bitmap;

        public PresenceControl()
        {
            InitializeComponent();
        }

        public void SetKinectSensor(KinectSensor sensor)
        {
            kinectSensor = sensor;

            kinectSensor.DepthFrameReady += kinectSensor_DepthFrameReady;
            kinectSensor.DepthStream.Enable(DepthImageFormat.Resolution640x480Fps30);

            kinectSensor.SkeletonFrameReady += kinectSensor_SkeletonFrameReady;
        }
}
}

There is nothing new here—as mentioned, you have already used this kind of code.

The depth stream part of the code now should be obvious and even simpler than what was used in previous chapters, because you don’t need to include the player index color:

void kinectSensor_DepthFrameReady(object sender, DepthImageFrameReadyEventArgs e)
        {
            using (var frame = e.OpenDepthImageFrame())
            {
                if (frame == null)
                    return;

                if (depthFrame32 == null)
                {
                    pixelData = new short[frame.PixelDataLength];
                    depthFrame32 = new byte[frame.Width * frame.Height * sizeof(int)];
                }

                frame.CopyPixelDataTo(pixelData);

                if (bitmap == null)
                {
                    bitmap = new WriteableBitmap(frame.Width, frame.Height, 96, 96,
PixelFormats.Bgra32, null);
                    image.Source = bitmap;
                }

                ConvertDepthFrame(pixelData);

                int stride = bitmap.PixelWidth * sizeof(int);
                Int32Rect dirtyRect =
new Int32Rect(0, 0, bitmap.PixelWidth, bitmap.PixelHeight);
                bitmap.WritePixels(dirtyRect, depthFrame32, stride, 0);
            }
        }

        void ConvertDepthFrame(short[] depthFrame16)
        {
            int i32 = 0;
            for (int i16 = 0; i16 < depthFrame16.Length; i16++)
            {
                int user = depthFrame16[i16] & 0x07;
                int realDepth = (depthFrame16[i16] >> DepthImageFrame.PlayerIndexBitmaskWidth);

                byte intensity = (byte)(255 - (255 * realDepth / 0x1fff));

                depthFrame32[i32] = 0;
                depthFrame32[i32 + 1] = 0;
                depthFrame32[i32 + 2] = 0;
                depthFrame32[i32 + 3] = 255;

                if (user > 0)
                {
                    depthFrame32[i32] = intensity;
                }
                else
                {
                    depthFrame32[i32] = (byte)(intensity / 2);
                    depthFrame32[i32 + 1] = (byte)(intensity / 2);
                    depthFrame32[i32 + 2] = (byte)(intensity / 2);
                }

                i32 += 4;
            }
        }

The skeleton part of the code will look for the position of the user’s hands and will update the position of the ellipses in response, using the MapSkeletonPointToDepth method to compute screen coordinates:

void kinectSensor_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
    Skeleton[] skeletons = null;

    leftEllipse.Visibility = System.Windows.Visibility.Collapsed;
    rightEllipse.Visibility = System.Windows.Visibility.Collapsed;

    using (SkeletonFrame frame = e.OpenSkeletonFrame())
    {
        if (frame == null)
            return;

        Tools.GetSkeletons(frame, ref skeletons); // See Chapter 3, "Displaying Kinect data"

        if (skeletons.All(s => s.TrackingState == SkeletonTrackingState.NotTracked))
            return;
        foreach (var skeleton in skeletons)
        {
            if (skeleton.TrackingState != SkeletonTrackingState.Tracked)
                continue;

            foreach (Joint joint in skeleton.Joints)
            {
                if (joint.TrackingState != JointTrackingState.Tracked)
                    continue;

                if (joint.JointType == JointType.HandRight)
                {
                    rightEllipse.Visibility = System.Windows.Visibility.Visible;
                    var handRightDepthPosition =
kinectSensor.MapSkeletonPointToDepth(joint.Position, DepthImageFormat.Resolution640x480Fps30);

                    rightTransform.X = (handRightDepthPosition.X / 640.0f) * Width;
                    rightTransform.Y = (handRightDepthPosition.Y / 480.0f) * Height;
                }
                else if (joint.JointType == JointType.HandLeft)
                {
                    leftEllipse.Visibility = System.Windows.Visibility.Visible;
                    var handLeftDepthPosition =
kinectSensor.MapSkeletonPointToDepth(joint.Position, DepthImageFormat.Resolution640x480Fps30);

                    leftTransform.X = (handLeftDepthPosition.X / 640.0f) * Width;
                    leftTransform.Y = (handLeftDepthPosition.Y / 480.0f) * Height;
                }
            }
        }
    }
}

With this control, you can now show your user whether or not he or she is currently being tracked by the Kinect sensor.

Replacing the mouse

The next step is to change the MouseController class you wrote in Chapter 9. You will draw a large custom pointer to replace it.

First you will create a user control called MouseImpostor to display the cursor. This control integrates a big arrow next to a hidden progress bar. (The progress bar will be used later to detect the click.) Indeed, instead of trying to simulate a click with a gesture (which is too complex for now), it is better to generate a click when the cursor is static during a few seconds (as in Xbox 360). To represent the time interval, the progress bar will fill to indicate to the user the remaining time before the click will be raised, as shown in Figure 10-2.

You can simulate a click by keeping the pointer on top of a control during a given duration.

Figure 10-2. You can simulate a click by keeping the pointer on top of a control during a given duration.

The XAML of the user control is defined with the following code:

<UserControl x:Class="Kinect.Toolbox.MouseImpostor"
             xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
             xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
             xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
             xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
             mc:Ignorable="d"
             Width="100"
             Height="100"
             d:DesignHeight="300" d:DesignWidth="300">
    <Grid>
        <Image Source="pack://application:,,,/Kinect.Toolbox;component/Resources/Arrow.png"
Margin="10"/>
        <ProgressBar x:Name="progressBar" Height="15" VerticalAlignment="Bottom" Margin="2"
Minimum="0" Maximum="100" Visibility="Collapsed"/>
    </Grid>
</UserControl>

This control is simple: it consists of an image (which you must add to the assembly in a folder called Resources and define as Embedded Resource) and a progress bar.

Note

If your project is not named Kinect.Toolbox, you will have to change the pack reference for the image. You can find a sample image in the companion content.

The code is also simple, with a Progression property used to fill and show (or hide) the progress bar:

using System;
using System.Windows.Controls;

namespace Kinect.Toolbox
{
    /// <summary>
    /// Interaction logic for MouseImpostor.xaml
    /// </summary>
    public partial class MouseImpostor : UserControl
    {
        public event Action OnProgressionCompleted;

        public MouseImpostor()
        {
            InitializeComponent();
        }

        public int Progression
        {
            set
            {
                if (value == 0 || value > 100)
                {
                    progressBar.Visibility = System.Windows.Visibility.Collapsed;
                    if (value > 100 && OnProgressionCompleted != null)
                        OnProgressionCompleted();
                }
                else
                {
                    progressBar.Visibility = System.Windows.Visibility.Visible;
                    progressBar.Value = value;

                }
            }
        }
    }
}

Note

The OnProgressionCompleted event is raised when the progress bar is completely filled.

To use the new cursor you have created, you must update the MouseController class with the following new members:

// Impostors
using System;
using System.Windows.Media;
using System.Windows.Controls;

Canvas impostorCanvas;
Visual rootVisual;
MouseImpostor impostor;

The impostorCanvas is placed over all of the content on the client page to allow the cursor to display in the topmost layer. Thus, the code for a typical client page will be as follows:

<Window x:Class="KinectUI.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:local="clr-namespace:Kinect.Toolbox;assembly=Kinect.Toolbox"
        Loaded="MainWindow_Loaded_1"
        Height="600"
        Title="MainWindow" Width="800">
    <Grid>
        <Grid.RowDefinitions>
            <RowDefinition/>
            <RowDefinition Height="184"/>
        </Grid.RowDefinitions>
        <StackPanel  HorizontalAlignment="Center" VerticalAlignment="Center">
            <CheckBox Content="Option 1">
                <CheckBox.LayoutTransform>
                    <ScaleTransform ScaleX="4" ScaleY="4"/>
                </CheckBox.LayoutTransform>
            </CheckBox>
            <CheckBox Content="Option 2">
                <CheckBox.LayoutTransform>
                    <ScaleTransform ScaleX="4" ScaleY="4"/>
                </CheckBox.LayoutTransform>
            </CheckBox>
            <CheckBox Content="Option 3">
                <CheckBox.LayoutTransform>
                    <ScaleTransform ScaleX="4" ScaleY="4"/>
                </CheckBox.LayoutTransform>
            </CheckBox>
        </StackPanel>
        <Button Content="Close" FontSize="38" HorizontalAlignment="Right" Width="150"
Height="80" VerticalAlignment="Top" Margin="10" Click="Button_Click_1"/>
        <local:PresenceControl Width="256" Height="184" VerticalAlignment="Bottom"
HorizontalAlignment="Center" x:Name="presenceControl" Grid.Row="1"/>
        <Canvas Grid.RowSpan="2" x:Name="mouseCanvas"/>
    </Grid>
</Window>

Notice that the mouseCanvas canvas overlaps all of the content of the root grid. The rootVisual visual is detected when the impostorCanvas is set. It contains the parent of the canvas (and the parent of all controls on the page).

After adding the previous members, you have to add the following property in the MouseController class to set all the required data:

public Canvas ImpostorCanvas
{
    set
    {
        if (value == null)
        {
            if (impostorCanvas != null)
                impostorCanvas.Children.Remove(impostor);

            impostor = null;
            rootVisual = null;
            return;
        }

        impostorCanvas = value;
        rootVisual = impostorCanvas.Parent as Visual;
        impostor = new MouseImpostor();

        value.Children.Add(impostor);
    }
}

To simplify the use of the MouseController class, you can add a simple singleton (a class that only allows one instance of itself to be created) to it:

static MouseController current;
public static MouseController Current
{
    get
    {
        if (current == null)
        {
            current = new MouseController();
        }

        return current;
    }
}

You create the connection between the client page and the MouseController class by adding the following code in the code behind the client page:

MouseController.Current.ImpostorCanvas = mouseCanvas;

Finally, you can update the SetHandPosition of the MouseController class to take in account the impostor:

public void SetHandPosition(KinectSensor sensor, Joint joint, Skeleton skeleton)
        {
            Vector2 vector2 = FilterJointPosition(sensor, joint);

            if (!lastKnownPosition.HasValue)
            {
                lastKnownPosition = vector2;
                previousDepth = joint.Position.Z;
                return;
            }

            var impostorPosition =
new Vector2((float)(vector2.X * impostorCanvas.ActualWidth), (float)(vector2.Y *
impostorCanvas.ActualHeight));

            bool isClicked = false;

                if (ClickGestureDetector == null)
                    isClicked = Math.Abs(joint.Position.Z - previousDepth) > 0.05f;
                else
                    isClicked = clickGestureDetected;


            if (impostor != null)
            {
                    Canvas.SetLeft(impostor, impostorPosition.X - impostor.ActualWidth / 2);
                    Canvas.SetTop(impostor, impostorPosition.Y);
            }
            else
            {
                MouseInterop.ControlMouse((int)(
(vector2.X - lastKnownPosition.Value.X) * Screen.PrimaryScreen.Bounds.Width * GlobalSmooth),
(int)((vector2.Y - lastKnownPosition.Value.Y) *
Screen.PrimaryScreen.Bounds.Height * GlobalSmooth), isClicked);
                lastKnownPosition = vector2;
            }

            previousDepth = joint.Position.Z;

            clickGestureDetected = false;
        }

According to the impostor member, the mouse pointer is updated directly or the MouseController class will use the cursor impostor control.

Magnetization!

The final concept you need to understand to complete your application is magnetization. Magnetization helps the user by automatically attracting the cursor when it is near a specific control, which helps offset the imprecision of the Kinect user interface.

The magnetized controls

First, you can add a simple collection of magnetized controls in the MouseController:

using System.Windows;

public List<FrameworkElement> MagneticsControl
{
    get;
    private set;
}

The magnetized range (the minimal distance required to attract the cursor) is also defined with a new property:

public float MagneticRange
{
    get;
    set;
}

Now you need to add some more class members:

bool isMagnetized;
DateTime magnetizationStartDate;
FrameworkElement previousMagnetizedElement;

Use the isMagnetized member to define whether a control is currently magnetizing the cursor. Use magnetizationStartDate to define when the cursor began to be static. The previousMagnetizedElement member defines the previous magnetized control. The constructor becomes:

MouseController()
{
    TrendSmoothingFactor = 0.25f;
    JitterRadius = 0.05f;
    DataSmoothingFactor = 0.5f;
    PredictionFactor = 0.5f;

    GlobalSmooth = 0.9f;

    MagneticsControl = new List<FrameworkElement>();

    MagneticRange = 25.0f;
}

The SetHandPosition has to evolve into a more complex method:

using System.Drawing;

public void SetHandPosition(KinectSensor sensor, Joint joint, Skeleton skeleton)
{
    Vector2 vector2 = FilterJointPosition(sensor, joint);

    if (!lastKnownPosition.HasValue)
    {
        lastKnownPosition = vector2;
        previousDepth = joint.Position.Z;
        return;
    }

    bool isClicked = false;

        if (ClickGestureDetector == null)
            isClicked = Math.Abs(joint.Position.Z - previousDepth) > 0.05f;
        else
            isClicked = clickGestureDetected;

    if (impostor != null)
    {
        // Still magnetized ?
        if ((vector2 - lastKnownPosition.Value).Length > 0.1f)
        {
            impostor.Progression = 0;
            isMagnetized = false;
            previousMagnetizedElement = null;
        }

        // Looking for nearest magnetic control
        float minDistance = float.MaxValue;
        FrameworkElement nearestElement = null;
        var impostorPosition =
new Vector2((float)(vector2.X * impostorCanvas.ActualWidth),
(float)(vector2.Y * impostorCanvas.ActualHeight));

        foreach (FrameworkElement element in MagneticsControl)
        {
            // Getting the four corners
            var position = element.TransformToAncestor(rootVisual).Transform(new Point(0, 0));
            var p1 = new Vector2((float)position.X, (float)position.Y);
            var p2 = new Vector2((float)(position.X + element.ActualWidth), (float)position.Y);
            var p3 = new Vector2((float)(position.X + element.ActualWidth), (float)(position.Y +
element.ActualHeight));
            var p4 = new Vector2((float)position.X, (float)(position.Y + element.ActualHeight));

            // Minimal distance
            float previousMinDistance = minDistance;
            minDistance = Math.Min(minDistance, (impostorPosition - p1).Length);
            minDistance = Math.Min(minDistance, (impostorPosition - p2).Length);
            minDistance = Math.Min(minDistance, (impostorPosition - p3).Length);
            minDistance = Math.Min(minDistance, (impostorPosition - p4).Length);

            if (minDistance != previousMinDistance)
            {
                nearestElement = element;
            }
        }

        // If a control is at a sufficient distance
        if (minDistance < MagneticRange || isMagnetized)
        {
            // Magnetic control found
            var position =
nearestElement.TransformToAncestor(rootVisual).Transform(new Point(0, 0));

            Canvas.SetLeft(impostor, position.X + nearestElement.ActualWidth / 2 -
impostor.ActualWidth / 2);
            Canvas.SetTop(impostor, position.Y + nearestElement.ActualHeight / 2);
            lastKnownPosition = vector2;

            if (!isMagnetized || previousMagnetizedElement != nearestElement)
            {
                isMagnetized = true;
                magnetizationStartDate = DateTime.Now;
            }
            else
            {
                impostor.Progression =
(int)(((DateTime.Now - magnetizationStartDate).TotalMilliseconds * 100) / 2000.0);
            }
        }
        else
        {
            Canvas.SetLeft(impostor, impostorPosition.X - impostor.ActualWidth / 2);
            Canvas.SetTop(impostor, impostorPosition.Y);
        }

        if (!isMagnetized)
            lastKnownPosition = vector2;

        previousMagnetizedElement = nearestElement;
    }
    else
    {
        MouseInterop.ControlMouse((int)((vector2.X - lastKnownPosition.Value.X) * Screen.
PrimaryScreen.Bounds.Width * GlobalSmooth), (int)((vector2.Y - lastKnownPosition.Value.Y) * Screen.
PrimaryScreen.Bounds.Height * GlobalSmooth), isClicked);
        lastKnownPosition = vector2;
    }
    previousDepth = joint.Position.Z;
    clickGestureDetected = false;
}

The intent here is to track the position of each corner and the relative distance of the corners to the cursor. If a control is near the cursor (meaning at a distance lesser than the MagneticRange property), the cursor is moved to the center of the control and the progression value of the impostor is updated, indicating that a click is in progress.

Simulating a click

You also need to change the ImpostorCanvas property to handle the OnProgressionCompleted of the impostor (to raise a click):

using System.Windows.Automation.Peers;
using System.Windows.Automation.Provider;

public Canvas ImpostorCanvas
{
    set
    {
        if (value == null)
        {
            if (impostorCanvas != null)
                impostorCanvas.Children.Remove(impostor);

            impostor.OnProgressionCompleted -= impostor_OnProgressionCompleted;

            impostor = null;
            rootVisual = null;
            return;
        }

        impostorCanvas = value;
        rootVisual = impostorCanvas.Parent as Visual;
        impostor = new MouseImpostor();

        impostor.OnProgressionCompleted += impostor_OnProgressionCompleted;

        value.Children.Add(impostor);
    }
}

void impostor_OnProgressionCompleted()
{
    if (previousMagnetizedElement != null)
    {
        var peer = UIElementAutomationPeer.CreatePeerForElement(previousMagnetizedElement);

        IInvokeProvider invokeProv =
peer.GetPattern(PatternInterface.Invoke) as IInvokeProvider;

        if (invokeProv == null)
        {
            var toggleProv = peer.GetPattern(PatternInterface.Toggle) as IToggleProvider;

            toggleProv.Toggle();
        }
        else
            invokeProv.Invoke();

        previousMagnetizedElement = null;
        isMagnetized = false;
    }
}

When the impostor raises the OnProgressionCompleted event, you can use the magic of the Automation assembly (reference the UIAutomationProvider.dll in your project). This namespace contains a class that helps you simulate the use of standard controls. For instance, the IToggleProvider interface allows you toggle a check box, and so on.

Adding a behavior to integrate easily with XAML

The last step you need to be able to use magnetization in the UI is to register controls as magnetizers. To do this, you can use an attached property to tag standard controls like this:

<CheckBox Content="Option 3" local:MagneticPropertyHolder.IsMagnetic="True">
    <CheckBox.LayoutTransform>
        <ScaleTransform ScaleX="4" ScaleY="4"/>
    </CheckBox.LayoutTransform>
</CheckBox>

The attached property is fairly simple, because it only adds the source control in the magnetizers list:

using System;
using System.Windows;

namespace Kinect.Toolbox
{
    public class MagneticPropertyHolder
    {
        public static readonly DependencyProperty IsMagneticProperty =
DependencyProperty.RegisterAttached("IsMagnetic", typeof(bool), typeof(MagneticPropertyHolder),
            new FrameworkPropertyMetadata(false, FrameworkPropertyMetadataOptions.AffectsRender,
OnIsMagneticChanged));

        static void OnIsMagneticChanged(DependencyObject d,
DependencyPropertyChangedEventArgs e)
        {
            FrameworkElement element = d as FrameworkElement;
            if ((bool)e.NewValue)
            {
                if (!MouseController.Current.MagneticsControl.Contains(element))
                    MouseController.Current.MagneticsControl.Add(element);
            }
            else
            {
                if (MouseController.Current.MagneticsControl.Contains(element))
                    MouseController.Current.MagneticsControl.Remove(element);
            }
        }

        public static void SetIsMagnetic(UIElement element, Boolean value)
        {
            element.SetValue(IsMagneticProperty, value);
        }

        public static bool GetIsMagnetic(UIElement element)
        {
            return (bool)element.GetValue(IsMagneticProperty);
        }
    }
}

To make it as easy as possible to understand, here is the final version of the MouseController class:

using System;
using System.Collections.Generic;
using System.Windows;
using System.Windows.Automation.Peers;
using System.Windows.Automation.Provider;
using System.Windows.Controls;
using System.Windows.Forms;
using System.Windows.Media;
using Microsoft.Kinect;

namespace Kinect.Toolbox
{
    public class MouseController
    {
        static MouseController current;
        public static MouseController Current
        {
            get
            {
                if (current == null)
                {
                    current = new MouseController();
                }

                return current;
            }
        }

        Vector2? lastKnownPosition;
        float previousDepth;

        // Filters
        Vector2 savedFilteredJointPosition;
        Vector2 savedTrend;
        Vector2 savedBasePosition;
        int frameCount;

        // Impostors
        Canvas impostorCanvas;
        Visual rootVisual;
        MouseImpostor impostor;

        bool isMagnetized;
        DateTime magnetizationStartDate;
        FrameworkElement previousMagnetizedElement;

        // Gesture detector for click
        GestureDetector clickGestureDetector;
        bool clickGestureDetected;
        public GestureDetector ClickGestureDetector
        {
            get
            {
                return clickGestureDetector;
            }
            set
            {
                if (value != null)
                {
                    value.OnGestureDetected += (obj) =>
                        {
                            clickGestureDetected = true;
                        };
                }

                clickGestureDetector = value;
            }
        }

        public Canvas ImpostorCanvas
        {
            set
            {
                if (value == null)
                {
                    if (impostorCanvas != null)
                        impostorCanvas.Children.Remove(impostor);

                    impostor.OnProgressionCompleted -= impostor_OnProgressionCompleted;

                    impostor = null;
                    rootVisual = null;
                    return;
                }

                impostorCanvas = value;
                rootVisual = impostorCanvas.Parent as Visual;
                impostor = new MouseImpostor();

                impostor.OnProgressionCompleted += impostor_OnProgressionCompleted;

                value.Children.Add(impostor);
            }
        }

        void impostor_OnProgressionCompleted()
        {
            if (previousMagnetizedElement != null)
            {
                var peer =
UIElementAutomationPeer.CreatePeerForElement(previousMagnetizedElement);

                IInvokeProvider invokeProv =
peer.GetPattern(PatternInterface.Invoke) as IInvokeProvider;

                if (invokeProv == null)
                {
                    var toggleProv =
peer.GetPattern(PatternInterface.Toggle) as IToggleProvider;

                    toggleProv.Toggle();
                }
                else
                    invokeProv.Invoke();

                previousMagnetizedElement = null;
                isMagnetized = false;
            }
        }

        public float MagneticRange
        {
            get;
            set;
        }

        public List<FrameworkElement> MagneticsControl
        {
            get;
            private set;
        }

        // Filter parameters
        public float TrendSmoothingFactor
        {
            get;
            set;
        }

        public float JitterRadius
        {
            get;
            set;
        }

        public float DataSmoothingFactor
        {
            get;
            set;
        }

        public float PredictionFactor
        {
            get;
            set;
        }

        public float GlobalSmooth
        {
            get;
            set;
        }

        MouseController()
        {
            TrendSmoothingFactor = 0.25f;
            JitterRadius = 0.05f;
            DataSmoothingFactor = 0.5f;
            PredictionFactor = 0.5f;

            GlobalSmooth = 0.9f;

            MagneticsControl = new List<FrameworkElement>();

            MagneticRange = 25.0f;
        }

        Vector2 FilterJointPosition(KinectSensor sensor, Joint joint)
        {
            Vector2 filteredJointPosition;
            Vector2 differenceVector;
            Vector2 currentTrend;
            float distance;

            Vector2 baseJointPosition = Tools.Convert(sensor, joint.Position);
            Vector2 prevFilteredJointPosition = savedFilteredJointPosition;
            Vector2 previousTrend = savedTrend;
            Vector2 previousBaseJointPosition = savedBasePosition;

            // Checking frames count
            switch (frameCount)
            {
                case 0:
                    filteredJointPosition = baseJointPosition;
                    currentTrend = Vector2.Zero;
                    break;
                case 1:
                    filteredJointPosition =
(baseJointPosition + previousBaseJointPosition) * 0.5f;
                    differenceVector = filteredJointPosition - prevFilteredJointPosition;
                    currentTrend = differenceVector * TrendSmoothingFactor + previousTrend *
(1.0f - TrendSmoothingFactor);
                    break;
                default:
                    // Jitter filter
                    differenceVector = baseJointPosition - prevFilteredJointPosition;
                    distance = Math.Abs(differenceVector.Length);

                    if (distance <= JitterRadius)
                    {
                        filteredJointPosition = baseJointPosition * (distance / JitterRadius) +
prevFilteredJointPosition * (1.0f - (distance / JitterRadius));
                    }
                    else
                    {
                        filteredJointPosition = baseJointPosition;
                    }

                    // Double exponential smoothing filter
                    filteredJointPosition = filteredJointPosition * (1.0f - DataSmoothingFactor)
+ (prevFilteredJointPosition + previousTrend) * DataSmoothingFactor;

                    differenceVector = filteredJointPosition - prevFilteredJointPosition;
                    currentTrend = differenceVector * TrendSmoothingFactor + previousTrend *
(1.0f - TrendSmoothingFactor);
                    break;
            }

            // Compute potential new position
            Vector2 potentialNewPosition = filteredJointPosition + currentTrend *
PredictionFactor;

            // Cache current value
            savedBasePosition = baseJointPosition;
            savedFilteredJointPosition = filteredJointPosition;
            savedTrend = currentTrend;
            frameCount++;

            return potentialNewPosition;
        }

        public void SetHandPosition(KinectSensor sensor, Joint joint, Skeleton skeleton)
        {
            Vector2 vector2 = FilterJointPosition(sensor, joint);

            if (!lastKnownPosition.HasValue)
            {
                lastKnownPosition = vector2;
                previousDepth = joint.Position.Z;
                return;
            }

            bool isClicked = false;


            if (ClickGestureDetector == null)
                isClicked = Math.Abs(joint.Position.Z - previousDepth) > 0.05f;
            else
               isClicked = clickGestureDetected;

            if (impostor != null)
            {
                // Still magnetized ?
                if ((vector2 - lastKnownPosition.Value).Length > 0.1f)
                {
                    impostor.Progression = 0;
                    isMagnetized = false;
                    previousMagnetizedElement = null;
                }

                // Looking for nearest magnetic control
                float minDistance = float.MaxValue;
                FrameworkElement nearestElement = null;
                var impostorPosition = new Vector2(
(float)(vector2.X * impostorCanvas.ActualWidth),
(float)(vector2.Y * impostorCanvas.ActualHeight));

                foreach (FrameworkElement element in MagneticsControl)
                {
                    // Getting the four corners
                    var position =
element.TransformToAncestor(rootVisual).Transform(new Point(0, 0));
                    var p1 = new Vector2((float)position.X, (float)position.Y);
                    var p2 = new Vector2((float)(position.X + element.ActualWidth),
(float)position.Y);
                    var p3 = new Vector2((float)(position.X + element.ActualWidth),
(float)(position.Y + element.ActualHeight));
                    var p4 = new Vector2((float)position.X,
(float)(position.Y + element.ActualHeight));

                    // Minimal distance
                    float previousMinDistance = minDistance;
                    minDistance = Math.Min(minDistance, (impostorPosition - p1).Length);
                    minDistance = Math.Min(minDistance, (impostorPosition - p2).Length);
                    minDistance = Math.Min(minDistance, (impostorPosition - p3).Length);
                    minDistance = Math.Min(minDistance, (impostorPosition - p4).Length);

                    if (minDistance != previousMinDistance)
                    {
                        nearestElement = element;
                    }
                }

                // If a control is at a sufficient distance
                if (minDistance < MagneticRange || isMagnetized)
                {
                    // Magnetic control found
                    var position =
nearestElement.TransformToAncestor(rootVisual).Transform(new Point(0, 0));

                    Canvas.SetLeft(impostor, position.X +
nearestElement.ActualWidth / 2 - impostor.ActualWidth / 2);
                    Canvas.SetTop(impostor, position.Y + nearestElement.ActualHeight / 2);
                    lastKnownPosition = vector2;

                    if (!isMagnetized || previousMagnetizedElement != nearestElement)
                    {
                        isMagnetized = true;
                        magnetizationStartDate = DateTime.Now;
                    }
                    else
                    {
                        impostor.Progression =
(int)(((DateTime.Now - magnetizationStartDate).TotalMilliseconds * 100) / 2000.0);
                    }
                }
                else
                {
                    Canvas.SetLeft(impostor, impostorPosition.X - impostor.ActualWidth / 2);
                    Canvas.SetTop(impostor, impostorPosition.Y);
                }

                if (!isMagnetized)
                    lastKnownPosition = vector2;

                previousMagnetizedElement = nearestElement;
            }
            else
            {
                MouseInterop.ControlMouse((int)
((vector2.X - lastKnownPosition.Value.X) * Screen.PrimaryScreen.Bounds.Width * GlobalSmooth),
(int)((vector2.Y - lastKnownPosition.Value.Y) *
Screen.PrimaryScreen.Bounds.Height * GlobalSmooth), isClicked);
                lastKnownPosition = vector2;
            }


            previousDepth = joint.Position.Z;

            clickGestureDetected = false;
        }
    }
}

You now have everything you need to create a Kinect-ready application. Using these new controls and tools, you can introduce a more adapted user interface that takes into account the precision available when working with the Kinect sensor.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.219.22.107