The Kinect device consists of a microphone array that supports a multitude of audio features. The Kinect device has four separate downward-facing microphones that are placed at the bottom of the Kinect device in a linear fashion.
The microphone array allows the following:
- Capturing better quality sound and providing inbuilt signal processing, including noise suppression and echo cancellation
- Identifying the source direction of the incoming sound
- Based on the sound to each microphone in the array, it can automatically find out the direction that the sound is coming from and listen to a specific microphone by suppressing the other noises
Once the direction of the sound source is set, the Kinect sensor is intelligent enough to change the direction as and when the source moves. One of the common examples of such a scenario is playing a game using voice commands. If the player moves, the sound source's direction moves automatically.
The Kinect sensor has an inbuilt audio-processing pipeline that takes care of the complete audio-processing capabilities. Another important aspect of the Kinect microphone array is speech recognition. The microphone array helps to recognize human speech very clearly by focusing only in a particular direction and canceling noises in the environment.
This chapter digs deep into the understanding of Kinect audio data processing using the microphone array and playing around with the Kinect SDK Audio APIs. The major areas we will cover in this chapter are as follows:
- Verifying the Kinect audio configuration
- The Kinect SDK architecture for audio
- How Kinect processes audio signals
- Inside Kinect's microphone array
- Capturing and playing audio
- Processing audio data by suppressing and canceling noise
- Understanding sound source localization and beam formation.
Before starting development with the Kinect Audio API, the very first thing you must check is whether your system is recognizing the Kinect microphone array as an audio device and can listen to it.
To make sure the audio devices are set up properly, navigate to Control Panel | Device Manager, look for the Kinect for Windows node, and there you will find Kinect for Windows Audio Array Control (refer to the following screenshot), which indicates that the Kinect microphone array is installed and recognized by your system properly:
In addition to the Kinect for Windows node, the SDK installs audio driver components under the Sound, video and game controllers node:
The Kinect for Windows USB Audio component has the sound drivers for the microphone array. If you change the Device Manager view from Device by type to Device by Connection, you will find that Kinect for Windows Audio Array Control and Kinect for Windows USB Audio are part of a USB composite device (refer to the following screenshot), which means on combining, these two will work together under a single USB root hub and do more than one job.
If there is a problem during the installation of the SDK, you may encounter a problem with the audio drivers, and the Kinect USB Audio device displays a warning symbol in the Device Manager as shown in the following screenshot:
To overcome such scenarios, follow these steps:
- While installing the SDK make sure the Kinect device is unplugged
- Before using Kinect, restart your system once the installation is done