In this chapter, we saw how to obtain color and depth images from the Kinect sensors and display them in the OpenGL context. A special type of depth image, which packs both depth value and player index in one pixel, was also introduced and used for background subtraction. It is more powerful than the traditional green screen technique because it doesn't need a single colored background (but the precision for civil-level use is not too high).

The main problem we encountered here was that the depth and color pixels were not aligned. The Kinect SDK also provides some functions for quickly mapping depth space coordinates to color space ones so that we can combine these two streams smoothly for different uses.