For the purposes of Cameo, we are interested in disparity maps and valid depth masks. They can help us refine our estimates of facial regions.

Using the FaceTracker function and a normal color image, we can obtain rectangular estimates of facial regions. By analyzing such a rectangular region in the corresponding disparity map, we can tell that some pixels within the rectangle are outliers—too near or too far to really be a part of the face. We can refine the facial region to exclude these outliers. However, we should only apply this test where the data is valid, as indicated by the valid depth mask.

Let's write a function to generate a mask whose values are 0 for the rejected regions of the facial rectangle and 1 for the accepted regions. This function should take a disparity map, valid depth mask, and a rectangle as arguments. We can implement it in depth.py as follows:

def createMedianMask(disparityMap, validDepthMask, rect = None):
    """Return a mask selecting the median layer, plus shadows."""
    if rect is not None:
        x, y, w, h = rect
        disparityMap = disparityMap[y:y+h, x:x+w]
        validDepthMask = validDepthMask[y:y+h, x:x+w]
    median = numpy.median(disparityMap)
    return numpy.where((validDepthMask == 0) | 
                       (abs(disparityMap - median) < 12),
                       1.0, 0.0)

To identify outliers in the disparity map, we first find the median using numpy.median(), which takes an array as an argument. If the array is of an odd length, median() returns the value that would lie in the middle of the array if the array were sorted. If the array is of even length, median() returns the average of the two values that would be sorted nearest to the middle of the array.

To generate a mask based on per-pixel Boolean operations, we use numpy.where() with three arguments. In the first argument, where() takes an array whose elements are evaluated for truth or falsity. An output array of like dimensions is returned. Wherever an element in the input array is true, the where() function's second argument is assigned to the corresponding element in the output array. Conversely, wherever an element in the input array is false, the where() function's third argument is assigned to the corresponding element in the output array.

Our implementation treats a pixel as an outlier when it has a valid disparity value that deviates from the median disparity value by 12 or more. I've chosen the value of 12 just by experimentation. Feel free to tweak this value later based on the results you encounter when running Cameo with your particular camera setup.