So let's go ahead and write a script that will generate those images for us. A few images containing different expressions are all that we need, but we have to make sure the sample images adhere to certain criteria:

Here's the script itself:

import cv2

def generate():
  face_cascade = cv2.CascadeClassifier('./cascades/haarcascade_frontalface_default.xml')
  eye_cascade = cv2.CascadeClassifier('./cascades/haarcascade_eye.xml')
  camera = cv2.VideoCapture(0)
  count = 0
  while (True):
    ret, frame = camera.read()
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    faces = face_cascade.detectMultiScale(gray, 1.3, 5)
    
    for (x,y,w,h) in faces:
        img = cv2.rectangle(frame,(x,y),(x+w,y+h),(255,0,0),2)
        
        f = cv2.resize(gray[y:y+h, x:x+w], (200, 200))

        cv2.imwrite('./data/at/jm/%s.pgm' % str(count), f)
        count += 1

    cv2.imshow("camera", frame)
    if cv2.waitKey(1000 / 12) & 0xff == ord("q"):
      break

  camera.release()
  cv2.destroyAllWindows()

if __name__ == "__main__":
  generate()

What is quite interesting about this exercise is that we are going to generate sample images building on our newfound knowledge of how to detect a face in a video feed. Effectively, what we are doing is detecting a face, cropping that region of the gray-scaled frame, resizing it to be 200x200 pixels, and saving it with a name in a particular folder (in my case, jm; you can use your initials) in the .pgm format.

I inserted a variable, count, because we needed progressive names for the images. Run the script for a few seconds, change expressions a few times, and check the destination folder you specified in the script. You will find a number of images of your face, grayed, resized, and named with the format, <count>.pgm.

Let's now move on to try and recognize our face in a video feed. This should be fun!

OpenCV 3 comes with three main methods for recognizing faces, based on three different algorithms: Eigenfaces, Fisherfaces, and Local Binary Pattern Histograms (LBPH). It is beyond the scope of this book to get into the nitty-gritty of the theoretical differences between these methods, but we can give a high-level overview of the concepts.

I will refer you to the following links for a detailed description of the algorithms:

First and foremost, all methods follow a similar process; they all take a set of classified observations (our face database, containing numerous samples per individual), get "trained" on it, perform an analysis of faces detected in an image or video, and determine two elements: whether the subject is identified, and a measure of the confidence of the subject really being identified, which is commonly known as the confidence score.

Eigenfaces performs a so called PCA, which—of all the mathematical concepts you will hear mentioned in relation to computer vision—is possibly the most descriptive. It basically identifies principal components of a certain set of observations (again, your face database), calculates the divergence of the current observation (the faces being detected in an image or frame) compared to the dataset, and it produces a value. The smaller the value, the smaller the difference between face database and detected face; hence, a value of 0 is an exact match.

Fisherfaces derives from PCA and evolves the concept, applying more complex logic. While computationally more intensive, it tends to yield more accurate results than Eigenfaces.

LBPH instead roughly (again, from a very high level) divides a detected face into small cells and compares each cell to the corresponding cell in the model, producing a histogram of matching values for each area. Because of this flexible approach, LBPH is the only face recognition algorithm that allows the model sample faces and the detected faces to be of different shape and size. I personally found this to be the most accurate algorithm generally speaking, but each algorithm has its strengths and weaknesses.

Now that we have our data, we need to load these sample pictures into our face recognition algorithms. All face recognition algorithms take two parameters in their train() method: an array of images and an array of labels. What do these labels represent? They are the IDs of a certain individual/face so that when face recognition is performed, we not only know the person was recognized but also who—among the many people available in our database—the person is.

To do that, we need to create a comma-separated value (CSV) file, which will contain the path to a sample picture followed by the ID of that person. In my case, I have 20 pictures generated with the previous script, in the subfolder, jm/, of the folder, data/at/, which contains all the pictures of all the individuals.

My CSV file therefore looks like this:

jm/1.pgm;0
jm/2.pgm;0
jm/3.pgm;0
...
jm/20.pgm;0

OK, at this stage, we have everything we need to instruct OpenCV to recognize our face.

Next up, we need to load these two resources (the array of images and CSV file) into the face recognition algorithm, so it can be trained to recognize our face. To do this, we build a function that reads the CSV file and—for each line of the file—loads the image at the corresponding path into the images array and the ID into the labels array.

def read_images(path, sz=None):
    
    c = 0
    X,y = [], []
    for dirname, dirnames, filenames in os.walk(path):
        for subdirname in dirnames:
            subject_path = os.path.join(dirname, subdirname)
            for filename in os.listdir(subject_path):
                try:
                    if (filename == ".directory"):
                        continue
                    filepath = os.path.join(subject_path, filename)
                    im = cv2.imread(os.path.join(subject_path, filename), cv2.IMREAD_GRAYSCALE)

                    # resize to given size (if given)
                    if (sz is not None):
                        im = cv2.resize(im, (200, 200))

                    X.append(np.asarray(im, dtype=np.uint8))
                    y.append(c)
                except IOError, (errno, strerror):
                    print "I/O error({0}): {1}".format(errno, strerror)
                except:
                    print "Unexpected error:", sys.exc_info()[0]
                    raise
            c = c+1
            

    return [X,y]

We're ready to test the face recognition algorithm. Here's the script to perform it:

def face_rec():
    names = ['Joe', 'Jane', 'Jack']
    if len(sys.argv) < 2:
        print "USAGE: facerec_demo.py </path/to/images> [</path/to/store/images/at>]"
        sys.exit()

    [X,y] = read_images(sys.argv[1])
    y = np.asarray(y, dtype=np.int32)
    
    if len(sys.argv) == 3:
        out_dir = sys.argv[2]
    
    model = cv2.face.createEigenFaceRecognizer()
    model.train(np.asarray(X), np.asarray(y))
    camera = cv2.VideoCapture(0)
    face_cascade = cv2.CascadeClassifier('./cascades/haarcascade_frontalface_default.xml')
    while (True):
      read, img = camera.read()
      faces = face_cascade.detectMultiScale(img, 1.3, 5)
      for (x, y, w, h) in faces:
        img = cv2.rectangle(img,(x,y),(x+w,y+h),(255,0,0),2)
        gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
        roi = gray[x:x+w, y:y+h]
        try:
            roi = cv2.resize(roi, (200, 200), interpolation=cv2.INTER_LINEAR)
            params = model.predict(roi)
            print "Label: %s, Confidence: %.2f" % (params[0], params[1])
            cv2.putText(img, names[params[0]], (x, y - 20), cv2.FONT_HERSHEY_SIMPLEX, 1, 255, 2)
        except:
            continue
      cv2.imshow("camera", img)
      if cv2.waitKey(1000 / 12) & 0xff == ord("q"):
        break
    cv2.destroyAllWindows()

There are a few lines that may look a bit mysterious, so let's analyze the script. First of all, there's an array of names declared; those are the actual names of the individual people I stored in my database of faces. It's great to identify a person as ID 0, but printing 'Joe' on top of a face that's been correctly detected and recognized is much more dramatic.

So whenever the script recognizes an ID, we will print the corresponding name in the names array instead of an ID.

After this, we load the images as described in the previous function, create the face recognition model with cv2.createEigenFaceRecognizer(), and train it by passing the two arrays of images and labels (IDs). Note that the Eigenface recognizer takes two important parameters that you can specify: the first one is the number of principal components you want to keep and the second is a float value specifying a confidence threshold.

Next up, we repeat a similar process to the face detection operation. This time, though, we extend the processing of the frames by also operating face recognition on any face that's been detected.

This happens in two steps: firstly, we resize the detected face to the expected size (in my case, samples were 200x200 pixels), and then we call the predict() function on the resized region.

Lastly, we obtain the results of the recognition and, just for effect, we draw it in the frame:

What about Fisherfaces? The process doesn't change much; we simply need to instantiate a different algorithm. So, the declaration of our model variable would look like so:

model = cv2.face.createFisherFaceRecognizer()

Fisherface takes the same two arguments as Eigenfaces: the Fisherfaces to keep and the confidence threshold. Faces with confidence above this threshold will be discarded.

Finally, let's take a quick look at the LBPH algorithm. Again, the process is very similar. However, the parameters taken by the algorithm factory are a bit more complex as they indicate in order: radius, neighbors, grid_x, grid_y, and the confidence threshold. If you don't specify these values, they will automatically be set to 1, 8, 8, 8, and 123.0. The model declaration will look like so:

  model = cv2.face.createLBPHFaceRecognizer()

The predict() method returns a two-element array: the first element is the label of the recognized individual and the second is the confidence score. All algorithms come with the option of setting a confidence score threshold, which measures the distance of the recognized face from the original model, therefore a score of 0 signifies an exact match.

There may be cases in which you would rather retain all recognitions, and then apply further processing, so you can come up with your own algorithms to estimate the confidence score of a recognition; for example, if you are trying to identify people in a video, you may want to analyze the confidence score in subsequent frames to establish whether the recognition was successful or not. In this case, you can inspect the confidence score obtained by the algorithm and draw your own conclusions.

LBPH works similarly; however, the reference value for a good recognition is below 50, and any value above 80 is considered as a low confidence score.

A normal custom approach would be to hold-off drawing a rectangle around a recognized face until we have a number of frames with a satisfying arbitrary confidence score, but you have total freedom to use OpenCV's face recognition module to tailor your application to your needs.