11.1. Overview
The Open Source Computer Vision Library (OpenCV for short [1, 14] is a collection of C and C++ source code and executables that span a wide range of computer vision algorithms. The code is optimized and intended for real-time vision applications. Popular application areas supported by OpenCV algorithms in decreasing order of coverage are human-computer interaction; video security; robotics and image retrieval; and factory inspection/machine vision. Look for a full handbook on OpenCV due from Springer in 2004 [12]. OpenCV supports Windows and Linux, but the code is well behaved and has been ported to many other platforms. Tested compilers include Intel's compiler version 6.0 or higher, MSVC++ 6.0 or higher, Borland C++ 5.5 or higher, and GNU C/C++ 2.95.3 or higher.
OpenCV is distributed under a BSD-style license (see Section 11.6). This license allows for royalty-free commercial or research use with no requirement that the user's code be free or open. Intel Labs (http://www.intel.com/research/mrl/index.htm) media group (http://www.intel.com/research/mrl/research/media.htm) undertook development of this library for the following reasons:
- Computer vision algorithms are being studied for their implications on future computer architecture. Optimized code is needed for these studies.
- We wanted to advance external computer vision research by creating an optimized infrastructure that would avoid the need for others to endlessly reinvent and reoptimize common algorithms.
- We wanted to hasten the adoption of computer vision applications out in the market by lowering the algorithmic "barrier to entry" for firms seeking to add computer vision functionality to their products.
OpenCV's main information Web site is located at http://www.intel.com/research/mrl/research/opencv. This site contains pointers to the user group and the download site.
The user group is at http://groups.yahoo.com/group/OpenCV/ [4]. To join the user group, you must first register (free) for Yahoogroups (http://groups.yahoo.com/) and then sign up for OpenCV at http://groups.yahoo.com/group/OpenCV/join. The user group is the forum for reporting bugs, discussing problems, technical issues, posting/reading calls for papers or jobs, and so on. This is an active group with thousands of members. You control at any time whether you want to read it on the web only (no emails) or get daily summaries or every posting. We do not distribute names or email addresses from the list to outside parties.
The download site is on SourceForge at http://sourceforge.net/projects/opencvlibrary/ [6]. At this site, you can obtain the various release versions of OpenCV for Windows and Linux, documentation files, and the foil set [5] from which many of the images in this chapter were taken.
 | 11.1.1. Installation |
Installation[1] under Windows is straightforward. Download the windows executable installation from the SourceForge site to a directory on your local disk and run it. The script installs OpenCV, registers DirectShow filters, and does other post-installation procedures. After it is finished, you may start using OpenCV; you do not need to compile OpenCV into binaries unless you want to debug files.
[1] Adapted with permission from the INSTALL file that comes with the download [6].
It is also possible to build core OpenCV binaries manually from the source-code distribution for Linux (though the executable installation includes sources as well). To build manually from the Linux source, do the following.
+ Download and unpack the OpenCV-*.tar.gz package somewhere, e.g., C:MySoft(the rootfolder is referred to further as <opencv_root>). The tree should look like this: <opencv_root> _dsw cv include src make cvaux ... + Add <opencv_root>in to the system path. Under Windows 9x/ME, this is done by
How to test built OpenCV binaries in Windows
Run algorithmic tests: <opencv_root>incvtest. exe. This will produce cvtest. sum and cvtest.1st. cvtest.sum should contain all OKs. Or, run samples at <opencv_root>samplesc. (Note that some of the demos need AVI or Camera, e.g., motempl.c.)
 | How to add support for another compiler |
Look at <opencv_root>utilsgen_make.py. It looks at .dsp files in the specified folders and generates makefiles for all compilers it knows. GCC handling now gets a little bit ugly because the linker cannot handle long lists of files and it wasn't known if it was possible to use temporary inline files instead.
There are no prebuilt binaries for the Linux version (because of different C++-incompatible versions of GCC in different distributions), so you have to build it from sources. The following has been tested on RedHat 8.0 (GCC 3.2) and SuSE 8.0 (GCC 2.95.3). To build fully functional libraries and demos in Linux, use
+ motif (LessTif or OpenMotif) with development files. configure script assumes it is at /usr/X11R6/lib & /usr/X11R6/include/Xm. + libpng, libjpeg and libtiff with development files. + libavcodec from ffmpeg 0.4.6(pre) and headers. Earlier version does not fit because of changed interface and because of the GPL license (newer version is LGPL). However, static linking is still prohibited for non-GPL software (such as OpenCV), so: Get CVS snapshot of ffmpeg from ffmpeg.sourceforge.net ./configure --enable-shared make make install You will then have /usr/local/lib/libavcodec. so and /usr/local/include/ffmpeg/*.h. + For building demo applications only: fltk 1.1.x (1.1.1 is currently the preferred one) with development files. If you do not have it, get it from www.fltk.org. In case of RPM-based distribution it is possible to build fltk RPMs by rpmbuild -ta fItk-x.y.z-source.tar.gz (for RH 8.x) or rpm -ta fItk-x.y.z-source.tar.gz (for others). + For demo applications only: TCL/TK 8.3.x with development files and BWidgets >=1.3.x. Download BWidgets from http://sourceforge.net/projects/tcllib/ Now build OpenCV: ====== 1. If your distribution uses RPM, you may build RPMs via rpmbuild -ta OpenCV-x.y.z.tar.gz" (for RH 8.x) or rpm -ta OpenCV-x.y.z.tar.gz" (for others) where OpenCV-x.y.z.tar.gz should be put into /usr/src/redhat/SOURCES/ or a similar folder. The command will build OpenCV-x.y.z.*.rpm (there is no OpenCV-devel; everything is in one package). Then, install it by rpm -i --nodeps OpenCV-x.y.z.*.rpm nodeps is needed in this version, because it cannot find libavcodec.so, even if it is in the path (a weird bug somewhere). 2. If your distribution does not support RPM, build and install it in the *nix traditional
How to test OpenCV under Linux:
- Run /usr/local/bin/cvtest.
- Or, compile and run simple C examples at /usr/local/share/opencv/ samples, for example, g++ 'opencv-config -cxxfiags' -o morphology morphology.c 'opencv-config -libs'. Plain gee won't work because of unresolved C++ specific symbols (located in highgui).
- Or, run /usr/local/bin/{cvlkdemocvcsdemocvenvvmdemotk}.
11.1.2. Organization
When you have set up OpenCV, the directory organization will be something like the following (slightly different for Linux):
C:Program FilesOpenCV -- Location
_dsw -- Build files under Windows
apps -- Demo application files
bin -- Binaries, dlls, filters ...
cv -- Source Code
cvaux -- "Experimental" Source Code
docs -- Manuals
filters -- DirectShow filters
interfaces -- Ch interpretive C, MATLAB
lib -- Static link libraries
otherlibs -- highgui interface and camera support
samples -- Simple code usage examples
tests -- Source and data for test code
utils -- cvinfo code and other utilities
apps: The apps directory contains the following:
C:Program FilesOpenCVapps -- Location
CamShiftDemo -- Track color probability distributions
Common -- Camera routines
cvcsdemo -- Color tracker application
cvlkdemo -- Optical flow application
HaarFaceDetect -- Viola-Jones Boosted face detection
HaarTraining -- Training code for Boosted face detection
Hawk -- EiC Interpretive C interface
HMMDemo -- Face recognition using HMMs
StereoDemo -- Depth detection from stereo correspondence
Tracker3dDemo -- 3D tracking with multiple cameras
VMDemo -- Interpolated view (morph) between 2 images
vmdemotk -- Interpolated view application
Of special interest here is the manual in the Hawk directory, which details some of the functions in HighGUI (see below) that make it easy to display an image or video in a window. One-line routines can be used to attach a menu and/or sliders to the window, which can be used to interactively control OpenCV routines operating in the window. The hidden Markov model (HMM) demo does face recognition using horizontal HMMs across a face feeding into a vertical HMM down a face to do 2D face recognition.
The StereoDemo is a console-based application that uses cvcam for video capture and HighGUI for visualization (both in otherlibs directory). To run this demo, you need two USB cameras compatible with DirectShow that can run simultaneously. Creative WebCam is an example of such a camera. Two different cameras types might be also work (not tested). You will need to register ProxyTrans.ax and SyncFilter.ax DirectShow filters from the opencvin folder using regsvr32. exe utility or opencvinRegisterAll. bat batch command file. The demo lets you calibrate the intrinsic and extrinsic parameters for the two cameras by automatically tracking a checkerboard pattern of known size and then finds stereo correspondence between the two image streams and converts that to a disparity or depth map.
Tracker3dDemo demonstrates the 3D calibration of multiple cameras (two or more) into one coordinate frame and then the 3D tracking of multiple objects from multiple views.
docs: The docs directory contains the HTML-based manual (index.htm, faq.htm), the license file (see Section 11.6), and directories describing some of the applications and tutorial papers.
filters: The filters directory contains code to build the following computer vision filters:
C:Program FilesOpenCVfilters -- Location
CalibFilter -- Calibrate camera lens via checkerboard
CamShift -- Meanshift robust object tracker
Condens -- Condensation particle filter tracker
Kalman -- Kalman filter
ProxyTrans -- Automatic filter calls your code
SyncFilter -- For use with multiple cameras
Tracker3dFilter -- 3D tracking with multiple cameras
otherlibs: The otherlibs directory contains the camera connection code (cvcam) and HighGUI which can be used to put up a window with still or video images and attach menus and controls to control video processing within the window.
samples: The samples directory is a good place to look for simple code examples of many of the functions in OpenCV.
11.1.3. Optimizations
OpenCV has two defining characteristics: (1) It spans a very wide range of functions, and (2) the code is fairly well optimized for speed. Figure 11.1 shows what code is open, what is not, and how things relate to one another. As with Intel's other performance libraries [3], when you build your application, you link against a dynamic library file, for example, cv.lib. At runtime, an optimized C dynamic-link library stub, cv.dll is loaded. This stub tries to determine what processor it is running on, and if the processor is identified, it will look for a DLL file that is specifically optimized for that processor. For example, on a Pentium 4, the specially optimized code is OptCVw7.dll. If such a file is found, cv.dll swaps itself out for this file. If the processor type cannot be determined or the corresponding optimized file cannot be found, OpenCV runs optimized C by default.
OpenCV's C code is algorithmically optimized. The processor-specific code is further optimized with inline assembly, memory alignment, and cache considerations, and it takes advantage when possible of the internal SIMD arithmetic MMX and SSE processors. OpenCV no longer depends on Intel's Image Processing Library, but some demos still require it. Again, as long as you have the set of DLLs on your machine, you can just compile/run OpenCV applications and the switcher code inside the cv. dll takes care of loading the correct one and takes advantage of the performance-primitive ippCV code described below.
ippCV [2]
OpenCV comes from Intel's research labs and is fully open and free to use. Intel has many other software tools that come from other internal teams such as the Software System's Group. This other software is released commercially. Such products include the VTune code profiler, the Intel Compiler, and the multithreading preprocessor macros. Of interest here are Intel's integrated performance primitives (IPP) and specifically the 60 or so computer vision functions included under ippCV. You can get information about IPP at http://developer.intel.com/software/products/ipp/ippvm20/.
The primitives part of the integrated performance primitives refers to functions that operate mostly on vectors, matrices, and simple structures. These functions are re-entrant and stateless. Thus, the IPP functions tend to be easy to interface to and work well embedded in streaming-data systems.
ippCV contains many fewer functions than OpenCV, but the functions are optimized on a wider variety of platforms. The platform support includes
IA32 Intel Itanium Architecture Intel StrongARM Microarchitecture Intel XScale Microarchitecture
Intel IPP supports various operating systems:
32-bit versions of Microsoft Windows 64-bit versions of Microsoft Windows Microsoft Windows CE Linux32 Linux64 ARM Linux
Inside ippCV, the following computer vision functions are supported:
Image pyramid (resample and blur) Filters: Laplace, Sobel, Scharr, Erode, Dilate Motion gradient Flood fill Canny edge detection Snakes Optical flow
Of course, there are many other non-computer vision functions supported in 2500 functions that make up IPP. IPP includes support for image processing, compression, signal processing, audio and video codecs, matrix operations, and speech recognition.
Other performance libraries
Two other libraries that serve as higher level wrappers to IPP are of interest: the signal processing library and the image processing library [3]. OpenCV was designed to work well with both IPP and these libraries. The functions covered by the signal processing and image processing libraries follow.
Signal Processing Functions:
Data manipulation
Windowing
Conversion
Arithmetic and logical operations
Filters and transforms Statistics
Audio
Signal generation
Image Processing Functions:
Data manipulation
Arithmetic and logical functions
Copy and conversions
Geometric operations (scale, rotate, warp)
Color conversion
Alpha composite
Gamma correction
General and specific filters: FFT, DCT, wavelet transform
A final separate library to be aware of is the Math Kernel Library (MKL), which contains the full set of linear algebra and matrix manipulation routines.