Features are extracted, matched, and tracked by the FeatureMatching class, especially by its public match method. However, before we can begin analyzing the incoming video stream, we have some homework to do. It might not be clear right away what some of these things mean (especially for SURF and FLANN), but we will discuss these steps in detail in the following sections. For now, we only have to worry about initialization:
class FeatureMatching:
def __init__(self, train_image='salinger.jpg'):- This sets up a SURF detector (see the next section for details) with a Hessian threshold between 300 and 500:
self.min_hessian = 400 self.SURF = cv2.SURF(self.min_hessian)
- We load a template of our object of interest (
self.img_obj), or print an error if it cannot be found:self.img_obj = cv2.imread(train_image, cv2.CV_8UC1) if self.img_obj is None: print "Could not find train image " + train_imageraise SystemExit - Also, store the shape of the image (
self.sh_train) for convenience:self.sh_train = self.img_train.shape[:2] # rows, cols
For reasons that will soon be evidently clear, we will call the template image the train image and every incoming frame a query image. The train image has a size of 480 x 270 pixels and looks like this:
- Apply SURF to the object of interest. This can be done with a convenient function call that returns both a list of keypoints and the descriptor (see the next section for details):
self.key_train, self.desc_train = self.SURF.detectAndCompute(self.img_obj, None)
We will do the same with each incoming frame and compare lists of features across images.
- Set up a FLANN object (see the next section for details). This requires the specification of some additional parameters via dictionaries, such as which algorithm to use and how many trees to run in parallel:
FLANN_INDEX_KDTREE = 0 index_params = dict(algorithm = FLANN_INDEX_KDTREE,trees = 5) search_params = dict(checks=50) self.flann = cv2.FlannBasedMatcher(index_params, search_params)
- Finally, initialize some additional bookkeeping variables. These will come in handy when we want to make our feature tracking both quicker and more accurate. For example, we will keep track of the latest computed homography matrix and of the number of frames we have spent without locating our object of interest (see the next section for details):
self.last_hinv = np.zeros((3,3)) self.num_frames_no_success = 0 self.max_frames_no_success = 5 self.max_error_hinv = 50.
Then, the bulk of the work is done by the FeatureMatching method match. This method follows the procedure elaborated here:
- It extracts interesting image features from each incoming video frame. This is done in
FeatureMatching._extract_features. - It matches features between the template image and the video frame. This is done in
FeatureMatching._match_features. If no such match is found, it skips to the next frame. - It finds the corner points of the template image in the video frame. This is done in
FeatureMatching._detect_corner_points. If any of the corners lies (significantly) outside the frame, it skips to the next frame. - It calculates the area of the quadrilateral that the four corner points span. If the area is either too small or too large, it skips to the next frame.
- It outlines the corner points of the template image in the current frame.
- It finds the perspective transform that is necessary to bring the located object from the current frame to the
frontoparallelplane. This is done inFeatureMatching._warp_keypoints. If the result is significantly different from the result we got recently for an earlier frame, it skips to the next frame. - It warps the perspective of the current frame to make the object of interest appear centered and upright.
In the following sections, we will discuss these steps in detail.