Recognizing holding and swiping gestures

We are going to implement two of the gestures from the previous table in this section: holding and swiping. They are actually not "multi" gestures because they can be finished with only one finger on the surface, or one hand cursor from Kinect, but both are very useful for developing Kinect-based applications. The holding gesture can be used to trigger a button on the screen, and the swiping gesture can be used to select the menu item, by scrolling the item list and finding the required one.

Drawing cursors using two hands

Let's start now.

  1. Declare variables for gesture recognizing. We can increase the related counters when prerequisites of a specific gesture are fitted. And when the counter reaches a certain value, we will mark the gesture as "recognized".
    GLfloat cursors[6];  // Left hand: 0, 1, 2; Right: 3, 4, 5
GLfloat lastCursors[6];
GLfloat cursorColors[8];  // Left hand: 0-3; Right: 4-7
unsigned int holdGestureCount[2] = {0};
unsigned int swipeGestureCount[2] = {0};
  2. At the end of the updateSkeletonData() function, we declare and call a new function named guessGesture(). It will check possible gestures of both cursors. The locations will then be recorded to the variable lastCursors for the next frame use.
    guessGesture( 0, (yMin<cursors[1] && cursors[2]<zMax) );
guessGesture( 1, (yMin<cursors[4] && cursors[5]<zMax) );
for ( int i=0; i<6; ++i ) lastCursors[i] = cursors[i];
  3. The guessGesture() function has two parameters: the cursor index (0 is the left hand, and 1 is the right hand), and a Boolean value to tell if the cursor is in the available range.
    void guessGesture( unsigned int index, bool inRange )
{
...
}
  4. In the function body, we will determine if the current state fits the conditions of either holding or swiping a gesture. Because lastCursors records the cursor locations of the previous frame, we can obtain the velocities of both cursors between two frames and use them for instantaneous judgment.
    if ( !inRange )
{
    // If the hand is not in range, reset all counters and
    // the cursor color (turn to translucence)
    holdGestureCount[index] = 0;
    swipeGestureCount[index] = 0;
    cursorColors[3 + index*4] = 0.2f;
}
else
{
    // Compute the distance of this and last cursor, which
    // is actually the instantaneous velocity of the cursor
    float distance = sqrt(
      powf(cursors[index*3]-lastCursors[index*3], 2.0f) +
      powf(cursors[1+index*3]-lastCursors[1+index*3], 2.0f));
    if ( distance<0.02 )
    {
        // If the cursor is nearly unmoved, increase holding
        holdGestureCount[index]++;
        swipeGestureCount[index] = 0;
    }
    else if ( distance>0.05 )
    {
        // If the cursor changes fast, increase swiping
        holdGestureCount[index] = 0;
        swipeGestureCount[index]++;
    }
    else
    {
        // Otherwise, reset the counters
        holdGestureCount[index] = 0;
        swipeGestureCount[index] = 0;
    }
    cursorColors[3 + index*4] = 1.0f;
}
  5. We will print the gesture names on screen in the render() function. If the counters for holding gesture are increased to a large enough value (in this case, to 30), it means we are holding the cursor for a long time, so "hold" is triggered. And if the swiping counters are set, it means we have already swiped the hands and the "swipe" gesture is triggered.
    std::string text = "Gestures (L/R): ";
for ( int i=0; i<2; ++i )
{
    if ( holdGestureCount[i]>30 ) text += "Hold;";
    else if ( swipeGestureCount[i]>1 ) text += "Swipe;";
    else text += "None;";
}
glRasterPos2f( 0.01f, 0.01f );
glColor4f( 1.0f, 1.0f, 1.0f, 1.0f );
glutBitmapString( GLUT_BITMAP_TIMES_ROMAN_24, (const unsigned char*)text.c_str() );
  6. Now run the application and try to achieve these gestures by using your hands. The gesture names will be displayed at the bottom-left corner of the screen. Now lift one of your arms and stop for some time, and flick the other hand quickly in front of you. Pay attention to the text displayed on the screen and see if the recognized gestures are correct and stable.
    Drawing cursors using two hands

    Display hand gestures (at the bottom-left)

  7. Please note that the swiping gesture may not be easily noticed from the text. That's because it will happen in a very short time. The next example of this chapter will demonstrate the use of the swiping gesture more clearly.

Understanding the code

Determining gestures always requires some mathematics, as well as some user-engineering knowledge. An instantaneous state may not be used to decide if a gesture happens or not. For example, the holding gesture requires the user to keep his/her cursor at a specified place for a while, and double tapping means we must detect the "tap" gesture at least twice in a short period of time. Thus, we have to keep a historical list of the cursors, which is similar to the implementation of a linetrail effect in the previous chapter.

Here are some hints for implementing different gestures, including the two gestures we have already done:

  • Tap: Not good for Kinect-based use, as you can hardly decide the time of "pushing" and "releasing" motions.
  • Double tap: Again, not good for Kinect-based use, as you can hardly decide the time of "pushing" and "releasing" motions.
  • Hold: Checks the distance between the current cursor and the previous cursor for every frame to see if they are near enough. The holding gesture is triggered if the cursor is still for a significant amount of time.
  • Swipe: Checks the distance between the current cursor and the previous cursor. If the distance is large enough, the user must exert himself to fling the arms and thus make a "swiping" gesture. Note that you must exclude the jitters.
  • Drag: Checks the distance between the current cursor and the previous cursor for every frame to determine if the cursor is moving all the time, while neither exceeding the holding and swiping threshold.
  • Zoom: Checks the distance of both the cursors. If the average velocities are opposite and the historical cursors of both are in a line, it produces a "zooming" gesture.
  • Rotate: Checks the distance of both the cursors. Make sure that the historical cursors of both are not in a line. If one is still and the other is moving a lot, or the average velocities are opposite, it can be considered as a "rotating" gesture.

Maybe you will have some other ideas and solutions, so don't hesitate to replace any in the previous list with your own, and see if it can make your customers feel better.

Additional information

Try to implement some more gestures on your own, especially the dragging and zooming gestures. They are very useful in your future projects for easy user interactions.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.225.11.98