In the last example of this chapter, we are going to make some use of the cursors we obtained from the skeleton data. As there are many other applications developed with only the mouse and keyboard as main input devices, it is sometimes meaningful to synthesize keyboard inputs, mouse motions, and button clicks, and send them to these applications for the purpose of providing more interaction methods.
A cool example, which we will be implementing here, is to convert the cursor data from Kinect to Windows mouse information so that we can use motion-sense techniques to control common operation systems. Other useful ideas include converting the cursors to the TUIO protocol (a unified framework for tangible multitouch surfaces) and use them for remote controls, or communicating with some famous multimedia software such as VVVV and Max/MSP.
This recipe will be slightly different from the previous ones. It won't render anything in the OpenGL window, but will send mouse events to Windows, to control the real mouse cursor. This requires us to make use of the Windows function
SendInput().
- We don't have to make it fullscreen. Just comment the following line:
//glutFullScreen();
- At the end of
updateSkeletonData(), after guessing the possible gestures, we convert the left hand's location and gesture into mouse press and wheel events, with the help of the Windows system APISendInput().if ( cursors[2]<zMax ) { // Set values of the INPUT structure INPUT input = {}; input.type = INPUT_MOUSE; input.mi.dx = (LONG)(65535 * cursors[0]); input.mi.dy = (LONG)(65535 * (1.0 - cursors[1])); input.mi.dwExtraInfo = GetMessageExtraInfo(); if ( holdGestureCount[0]>30 ) { // Send mouse push and release events when the // holding gesture is detected if ( !isMouseDown ) { input.mi.dwFlags = MOUSEEVENTF_ABSOLUTE | MOUSEEVENTF_LEFTDOWN; } else { input.mi.dwFlags = MOUSEEVENTF_ABSOLUTE | MOUSEEVENTF_LEFTUP; } isMouseDown = !isMouseDown; // Reset the counter so we won't receive the same // gesture continuously holdGestureCount[0] = 0; } else if ( swipeGestureCount[0]>1 ) { // If we encounter the swiping gesture, use it // to emulate the mouse wheel to scroll pages input.mi.dwFlags = MOUSEEVENTF_WHEEL; input.mi.mouseData = WHEEL_DELTA; } else { // For all other cases, simply move the mouse input.mi.dwFlags = MOUSEEVENTF_ABSOLUTE | MOUSEEVENTF_MOVE; } SendInput( 1, &input, sizeof(INPUT) ); } - Start the program and stand in front of the Kinect device. Now lift the left hand and we can see the Windows mouse cursor also move along with us. Hold and stay on an icon on the desktop. This will be recognized as "selecting and dragging", and a quick swipe in any direction will result in scrolling the content we are currently viewing.
There is nothing special in this example. We just used the Windows native function SendInput() to send cursors and gestures inferred from Kinect data to Windows mouse events. Windows uses the top-left corner as the original point, so we have to alter the coordinates before sending them. The gestures are parsed as left clicks and wheel events here, but of course you can consider them as different operations.
It is more valuable to change your Kinect cursors to TUIO, which can then be sent to local or remote clients that listen to the current TUIO server. There are several pieces of multimedia software that regard TUIO as a regular input, so you can easily connect them with Kinect devices.
TUIO API for C/C++ and other languages can be found at:
Simply call the corresponding TUIO commands when you receive a new cursor position or detect a new gesture, and any clients that listen to your PC using the same TUIO protocol will receive it and can parse it for their own uses.