Faces can then be detected using the detect method. Here, we ensure that we operate on a downscaled grayscale image:

def detect(self, frame):
    frameCasc = cv2.cvtColor(cv2.resize(frame, (0, 0), fx=1.0 / self.scale_factor, fy=1.0 / self.scale_factor), cv2.COLOR_RGB2GRAY)
    faces = self.face_casc.detectMultiScale(frameCasc, 
        scaleFactor=1.1, minNeighbors=3,  
        flags=cv2.cv.CV_HAAR_FIND_BIGGEST_OBJECT) * self.scale_factor

If a face is found, we continue to extract the head region from the bounding box information and store the result in head:

for (x, y, w, h) in faces:
    head = cv2.cvtColor(frame[y:y + h, x:x + w], cv2.COLOR_RGB2GRAY)

We also draw the bounding box onto the input image:

cv2.rectangle(frame, (x, y), (x + w, y + h), (100, 255, 0), 2)

In case of success, the method should return a Boolean indicating success (True), the annotated input image (frame), and the extracted head region (head):

return True, frame, head

Otherwise, if no faces were detected, the method indicates failure with a Boolean (False) and returns the unchanged input image (frame) and None for the head region:

return False, frame, None

After a face has been detected, we might want to preprocess the extracted head region before applying classification on it. Although the face cascade is fairly accurate, for recognition, it is important that all the faces are upright and centered on the image. This idea is best illustrated with an image. Consider a sad programmer under a tree:

Because of his emotional state, the programmer tends to keep his head slightly tilted to the side while looking down. The facial region as extracted by the face cascade is shown as the leftmost grayscale thumbnail on the right. In order to compensate for the head orientation, we aim to rotate and scale the face so that all data samples will be perfectly aligned. This is the job of the align_head method in the FaceDetector class:

def align_head(self, head):
    height, width = head.shape[:2]

Fortunately, OpenCV comes with a few eye cascades that can detect both open and closed eyes, such as haarcascade_lefteye_2splits.xml and haarcascade_righteye_2splits.xml. This allows us to calculate the angle between the line that connects the center of the two eyes and the horizon so that we can rotate the face accordingly. In addition, adding eye detectors will reduce the risk of having false positives in our dataset, allowing us to add a data sample only if both the head and the eyes have been successfully detected.

After loading these eye cascades from file in the FaceDetector constructor, they are applied to the input image (head):

    left_eye_region = head[0.2*height:0.5*height, 0.1*width:0.5*width]
    left_eye = self.left_eye_casc.detectMultiScale(left_eye_region, scaleFactor=1.1, minNeighbors=3, flags=cv2.cv.CV_HAAR_FIND_BIGGEST_OBJECT)

Here, it is important that we pass only a small, relevant region (left_eye_region; compare small thumbnails in the top-right corner of the preceding figure) to the eye cascades. For simplicity, we use hardcoded values that focus on the top half of the facial region and assume the left eye to be in the left half.

If an eye is detected, we extract the coordinates of its center point:

left_eye_center = None
for (xl, yl, wl, hl) in left_eye:
    # find the center of the detected eye region
    left_eye_center = np.array([0.1 * width + xl + wl / 2, 0.2 * height + yl + hl / 2])
    break # need only look at first, largest eye

Then, we proceed to do the same for the right eye:

right_eye_region = head[0.2*height:0.5*height, 0.5*width:0.9*width]
right_eye = self.right_eye_casc.detectMultiScale(right_eye_region, scaleFactor=1.1, minNeighbors=3,flags=cv2.cv.CV_HAAR_FIND_BIGGEST_OBJECT)
right_eye_center = None
for (xr, yr, wr, hr) in right_eye:
    # find the center of the detected eye region
    right_eye_center = np.array([0.5 * width + xr + wr / 2, 0.2 * height + yr + hr / 2])
    break  # need only look at first, largest eye

As mentioned earlier, if we do not detect both the eyes, we discard the sample as a false positive:

if left_eye_center is None or right_eye_center is None:
    return False, head

Now, this is where the magic happens. No matter how crooked the face that we detected is, before we add it to the training set, we want the eyes to be exactly at 25 percent and 75 percent of the image width (so that the face is in the center) and at 20 percent of the image height:

desired_eye_x = 0.25
desired_eye_y = 0.2
desired_img_width = 200
desired_img_height = desired_img_width

This can be achieved by warping the image using cv2.warpAffine (remember Chapter 3, Finding Objects via Feature Matching and Perspective Transforms?). First, we calculate the angle (in degrees) between the line that connects the two eyes and a horizontal line:

eye_center = (left_eye_center + right_eye_center) / 2
eye_angle_deg = np.arctan2(
    right_eye_center[1] – left_eye_center[1],
    right_eye_center[0] – left_eye_center[0]) * 
    180.0 / cv2.cv.CV_PI

Then, we derive a scaling factor that will scale the distance between the two eyes to be exactly 50 percent of the image width:

eye_size_scale = (1.0 - desired_eye_x * 2) * desired_img_width / np.linalg.norm(right_eye_center – left_eye_center)

With these two parameters (eye_angle_deg and eye_size_scale) in hand, we can now come up with a suitable rotation matrix that will transform our image:

rot_mat = cv2.getRotationMatrix2D(tuple(eye_center), eye_angle_deg, eye_size_scale)

We make sure that the center of the eyes will be centered in the image:

rot_mat[0,2] += desired_img_width*0.5 – eye_center[0]
rot_mat[1,2] += desired_eye_y*desired_img_height – eye_center[1]

Finally, we arrive at an upright scaled version of the facial region that looks like the lower-right thumbnail of the preceding image:

res = cv2.warpAffine(head, rot_mat, (desired_img_width, desired_img_height))
return True, res