In this recipe, we will learn how to track specific planar objects in our webcam using OpenCV and it's corresponding CinderBlock.
You will need an image depiction of the physical object you wish to track in the camera. For this recipe place that image in the assets
folder and name it object.jpg
.
We will use the OpenCV CinderBlock in this recipe, so please refer to the Integrating with OpenCV recipe from Chapter 3, Using Image Processing Techniques and add OpenCV and it's CinderBlock to your project.
If you are using a Mac, you will need to compile the OpenCV static libraries yourself, because the OpenCV CinderBlock is missing some needed libraries on OSX (it will work fine on Windows). You can download the correct version from the following link: http://sourceforge.net/projects/opencvlibrary/files/opencv-unix/2.3/.
You will need to compile the static libraries yourself using the provided CMake
files. Once your libraries are correctly added to your project, include the following files:
#include "cinder/Capture.h" #include "cinder/gl/Texture.h" #include "cinder/ImageIo.h"
Add the following using
statements:
using namespace ci; using namespace ci::app; using namespace std;
We will track an object in the camera frames based on an image depicting the object
struct
method to store the necessary objects for feature tracking and matching. Add the following code before your application class declaration:struct DetectionInfo{ vector<cv::Point2f> goodPoints; vector<cv::KeyPoint> keyPoints; cv::Mat image, descriptor; gl::Texture texture; };
DetectionInfo mObjectInfo, mCameraInfo; cv::Mat mHomography; cv::SurfFeatureDetector mFeatureDetector; cv::SurfDescriptorExtractor mDescriptorExtractor; cv::FlannBasedMatcher mMatcher; vector<cv::Point2f> mCorners;
setup
method let's start by initializing the camera:try{ mCamera = Capture( 640, 480 ); mCamera.start(); } catch( ... ){ console() << "could not initialize capture" << endl; }
mCorners
, load our object image, and calculate its image
, keyPoints
, texture
, and descriptor
:mCorners.resize( 4 ); Surface objectSurface = loadImage( loadAsset( "object.jpg" ) ); mObjectInfo.texture = gl::Texture( objectSurface ); mObjectInfo.image = toOcv( Channel( objectSurface ) ); mFeatureDetector.detect( mObjectInfo.image, mObjectInfo.keyPoints ); mDescriptorExtractor.compute( mObjectInfo.image, mObjectInfo.keyPoints, mObjectInfo.descriptor );
update
method, we will check if mCamera
has been initialized and whether we have a new frame to process:if( mCamera ){ if( mCamera.checkNewFrame() ){
mCamera
and initialize texture
and image
objects of mCameraInfo
. We will create a ci::Channel
object from cameraSurface
that converts color surfaces to gray channel surfaces:Surface cameraSurface = mCamera.getSurface(); mCameraInfo.texture = gl::Texture( cameraSurface ); mCameraInfo.image = toOcv( Channel( cameraSurface ) );
features
and descriptor
values of mCameraInfo
:mFeatureDetector.detect( mCameraInfo.image, mCameraInfo.keyPoints); mDescriptorExtractor.compute( mCameraInfo.image, mCameraInfo.keyPoints, mCameraInfo.descriptor );
mMatcher
to calculate the matches between mObjectInfo
and mCameraInfo
:vector<cv::DMatch> matches; mMatcher.match( mObjectInfo.descriptor, mCameraInfo.descriptor, matches );
double minDist = 640.0; for( int i=0; i<mObjectInfo.descriptor.rows; i++ ){ double dist = matches[i].distance; if( dist < minDist ){ minDist = dist; } }
minDist*3.0
to mObjectInfo.goodPoints.clear();
mCameraInfo.goodPoints.clear(); for( vector<cv::DMatch>::iterator it = matches.begin(); it != matches.end(); ++it ){ if( it->distance < minDist*3.0 ){ mObjectInfo.goodPoints.push_back( mObjectInfo.keyPoints[ it->queryIdx ].pt ); mCameraInfo.goodPoints.push_back( mCameraInfo.keyPoints[ it->trainIdx ].pt ); }
}
With all our points calculated and matched, we need to calculate the homography between the points of mObjectInfo
and mCameraInfo
:mHomography = cv::findHomography( mObjectInfo.goodPoints, mCameraInfo.goodPoints, CV_RANSAC );
vector<cv::Point2f>
with the corners of our object and perform a perspective transform to calculate the corners of our object in the camera image:vector<cv::Point2f> objCorners( 4 ); objCorners[0] = cvPoint( 0.0f, 0.0f ); objCorners[1] = cvPoint( mObjectInfo.image.cols, 0.0f); objCorners[2] = cvPoint( mObjectInfo.image.cols, mObjectInfo.image.rows ); objCorners[3] = cvPoint( 0.0f, mObjectInfo.image.rows); mCorners = vector< cv::Point2f >( 4 ); cv::perspectiveTransform( objCorners, mCorners, mHomography ); } }
draw
method and begin by clearing the background and drawing the camera and object textures:gl::clear( Color( 0, 0, 0 ) ); gl::color( Color::white() ); if( mCameraInfo.texture ){ gl::draw( mCameraInfo.texture, getWindowBounds() ); } if( mObjectInfo.texture ){ gl::draw( mObjectInfo.texture ); }
goodPoints
values in both mObjectInfo
and mCameraInfo
and draw them:for( int i=0; i<mObjectInfo.goodPoints.size(); i++ ){ gl::drawStrokedCircle( fromOcv( mObjectInfo.goodPoints[ i ] ), 5.0f ); gl::drawStrokedCircle( fromOcv( mCameraInfo.goodPoints[ i ] ), 5.0f ); gl::drawLine( fromOcv( mObjectInfo.goodPoints[ i ] ), fromOcv( mCameraInfo.goodPoints[ i ] ) ); }
mCorners
and draw the corners of the found object:gl::color( Color( 1.0f, 0.0f, 0.0f ) ); gl::begin( GL_LINE_LOOP ); for( vector<cv::Point2f>::iterator it = mCorners.begin(); it != mCorners.end(); ++it ){ gl::vertex( it->x, it->y ); } gl::end();
object.jpg
image and put it in front of the image. The program will try to track that object in the camera image and draw it's corners in the image.We are using a Speeded Up Robust Features (SURF) feature detector and descriptor to identify features. In the step 4, we are calculating the features and descriptor. We use a cv::SurfFeatureDetect
object or
that calculates good features to track on our object. The cv::SurfDescriptorExtractor
object then uses these features to create a description of our object. In the step 7, we do the same for the camera image.
In the step 8, we then use a Fast Library for Approximate Nearest Neighbor
(FLANN) called cv::FlannBasedMatcher
. This matcher takes the description from both the camera frame and our object, and calculates matches between them.
In steps 9 and 10, we use the minimum distance between matches to eliminate the possible false matches. The result is passed into mObjectInfo.goodPoints
and mCameraInfo.goodPoints
.
In the step 11, we calculate the homography between image and camera. A homography is a projection transformation from one space to another using projective geometry. We use it in the step 12 to apply a perspective transformation to mCorners
to identify the object corners in the camera image.
To learn more about what SURF is and how it works, please refer to the following web page: http://en.wikipedia.org/wiki/SURF.
To learn more about FLANN, please refer to the web page http://en.wikipedia.org/wiki/Nearest_neighbor_search.
To learn more about homography please refer to the following web page:
3.144.37.12